Making Pravda Blush: The CDC and Misinformation

The CDC says that an unvaccinated person with natural immunity is five times more likely to be infected with COVID than a vaccinated person. Is that even close to the truth?

When there’s a will, there’s a way. In its barely disguised quest for data that can be used to suggest that COVID vaccinations provide better protection than natural immunity, the CDC is straining its limited credibility by heralding a new paper published by CDC. In so doing, the CDC unwittingly implies (if the paper is to be believed) that not only is natural immunity worse than vaccine immunity, but natural immunity is actually worse than having no immunity at all!

Obviously, no one would believe that claim, and no one is making that case explicitly. However, the CDC is actively making the argument that based on this study, people who are vaccinated are five times less likely to become infected with COVID than unvaccinated people who have natural immunity by virtue of a previous COVID infection. Note that this figure is significantly higher than the current level of relative protection provided by vaccines when compared to people with no immunity whatsoever!

The Pfizer and Moderna mRNA vaccines have demonstrated a consensus 50% to 75% effectiveness against the Delta variant, meaning that people who are vaccinated are between two and four times less likely to be infected than people who have neither vaccine immunity nor natural immunity. Thus, the ceiling for effectiveness of the vaccines right now is around 4x, and probably lower. Yet the CDC proclaims 5x compared with natural immunity. The paper lists over 50 authors. You’d think one of them might have said “Atlanta, we’ve got a problem. Let’s re-run these numbers, and maybe review some of the assumptions.”

Either this paper is terribly wrong, or every other paper comparing natural immunity and vaccine immunity is terribly wrong. I know which one I'm betting on. This paper uses an indirect model to predict the likelihood of COVID infection, rather than the direct approach used by the many credible studies which looked at the total number of people in a given group, then identified the number that tested positive for COVID. Those numerous studies from the US, Israel, Israel, Denmark, Qatar, and the UK among others, used direct methods on large population groups to determine that the effectiveness of natural immunity in preventing COVID infection (or symptomatic COVID, or hospitalization), ranged from roughly 74% to 100% (at different times and places during the pandemic when different variants were predominant), and generally provided protection as good or better than the mRNA vaccines.

Alternatively, if you’re CDC Director Rochelle Walensky, you can instead choose to believe (or at least publicly suggest you believe) this new paper, which uses an indirect model for assessing infection percentage, contorts that data through a mathematical weighting model, and ultimately suggests that the vaccines are so much better than natural immunity that, combined with well-established, real-world studies of vaccine effectiveness, natural immunity would not only provide no protection whatsoever, but actually makes people more vulnerable to COVID. This is indeed remarkable.

The main data point in the study was that people with natural immunity who were hospitalized with COVID-like symptoms during 2021 were 1.8 times more likely to test positive for COVID than fully vaccinated people hospitalized with similar symptoms. That is solid (and interesting) data. From there, things go awry.

Through a footnoted, undetailed weighting process, the authors convert that 1.8x figure into 5.5x, concluding that people with natural immunity were 5.5 times more likely to test positive for COVID. Based on that data, the authors conclude that: “In this U.S.-based epidemiologic analysis of patients hospitalized with COVID-19–like illness whose previous infection or vaccination occurred 90–179 days earlier, vaccine-induced immunity was more protective than infection-induced immunity against laboratory-confirmed COVID-19, including during a period of Delta variant predominance.”

We won’t discuss the statistical weighting that turns 1.8x into 5.5x, as it is not explained in the paper, but instead is just footnoted with an acknowledgement in the paper that this weighting process might be flawed. I’m not inclined to evaluate the weighting model other than to say that the output numbers show that model was either wrong or used improperly.

Let’s consider the 1.8x data point. Vaccine effectiveness is routinely determined by comparing two groups given different treatments and comparing the percentage of each group who are infected. For example, we could study 10,000 people in each group (matched by age, health, sex, etc.). If 25 people in the vaccinated group develop the disease, whereas 50 people in the unvaccinated group develop the disease, that would correspond to a vaccine with 50% effectiveness. The process is simple, and all one needs is the proper numerator and denominator in order to make conclusions applicable to the groups.

The new CDC study has an appropriate numerator (i.e., the number of hospitalized people in the vaccinated and natural immunity groups who tested positive for COVID), but unfortunately, it lacks the appropriate denominator (i.e., the actual number of people in the vaccinated and natural immunity groups) in order to assess vaccine effectiveness. So how do the authors attempt to determine rate of infection? By substituting an artificial denominator; that is, the number of people in each group who are hospitalized with COVID-like symptoms.

In other words, rather than the correct denominator (the number of people in a given group), the authors have substituted a subgroup; that is, the number of people in the group who were hospitalized with COVID-like symptoms. I’m struggling to understand how this is an accurate model for the total number of people in that particular group (the variables are NOT independent of each other). The authors do not address this issue.

One implicit assumption appears to be that the hospitalization rate for non-COVID respiratory illnesses during the COVID pandemic is a constant for both the vaccinated population and the unvaccinated, naturally immune population. I can think of many reasons why this is unlikely to be the case. For example, perhaps people with natural immunity may be less frightened of COVID having already survived it, and therefore are less likely to seek medical care unless highly confident that they are indeed infected with COVID and need help (meaning a likely higher positive testing rate). Perhaps some of the people with natural immunity had long COVID which went undetected in an intermediate test, noting the high rate of false negatives in COVID testing. Perhaps after being masked and socially distanced for a year, individuals in the vaccinated group were more likely to pick up non-COVID infections which led to pneumonia and hospitalization once they returned to freely swapping germs with the masses.

Consider the following thought experiment. Imagine natural immunity against COVID was amazingly protective not only against COVID but also against other coronaviruses and respiratory infections, and thus dropped the total number of people with natural immunity who were hospitalized in the study down to a grand total of two (rather than roughly 1,000). Imagine one of those two people tested positive for COVID, meaning a 50% rate of positive COVID tests (rather than the 9% observed in the study). By this model, that would seemingly be sufficient for the authors and CDC to conclude that natural immunity was even worse relative to vaccine immunity, since the positive test rate for COVID among the hospitalized patients would have been so high. Quite clearly, this indirect approach to assessing relative effectiveness is a poor substitute for a direct assessment of populations.

Truthfully, I am surprised that the rate of positive tests in this study was significantly higher (9% to 5%) in the group with natural immunity. I would have expected something closer to parity. It would be useful to examine the numerator in each of the subgroups (age, hospital, month of infection, etc.) for more insights. While the appropriate denominator is unknown to the authors, as discussed previously, the authors clearly possess the numerators (numbers of positive tests). However, in all of the data tables, the numerators are only provided one time as the aggregated number of positives among the total hospitalizations in each group. Yet the model denominators (i.e., the numbers of hospitalizations) are broken down by age, sex, race, hospital, and most importantly, month of infection. Why not also provide the numerator (number of positive tests) for each of these groups? If the authors are going to make health recommendations for the entire population based on this study, as they have done in asserting that “all eligible persons should be vaccinated against COVID-19 as soon as possible, including unvaccinated persons previously infected with SARS-CoV-2”, they ought to reveal what is behind the curtain.

The failure to provide this crucial data is strange and problematic, as those numbers might be particularly helpful in revealing genuinely useful data from the study. I assume that the percentage of positive tests was relatively constant across the various subgroups such as age, hospital, and month of infection. Otherwise, failure to disclose the numerators would be misleading. For example, if the positive test rate for the group with natural immunity was much higher in April than July, or among elderly people than young people, or at one hospital relative to another, these differences would constitute significant information that could put the general conclusions of the paper into question.

I find it very interesting that the relative percentage of total hospitalizations was so much higher (by a factor of more than three) among young people with natural immunity than among the younger cohort of the vaccinated group (and relatively lower among the older groups). Note as well that the authors suggest that the protection afforded by vaccination was higher among the elderly (“In this study, the protective effect of vaccination also trended higher for adults aged ≥65 years than for those aged 18–64 years”), which implies that the percentage of positive COVID infections in the younger group with natural immunity might have been less than 9%, while it might have been higher than 9% among people over 65. There may be meaningful information in these data points, and it is unfortunate that the clarifying data is not provided.

The authors chose to look only at immunity conferred (by previous infection or natural immunity) three to six months prior to the hospitalization event characterized by COVID-like symptoms. Therefore, it is not surprising (given the rise and fall of daily infections over the course of the pandemic) that the number of hospitalizations in the natural immunity group dropped precipitously over the course of the study, while the number of hospitalizations in the vaccinated group rose roughly to a plateau. Nevertheless, at the start of the study, there were more hospitalizations among the previously infected group than among the vaccinated group. By the last month of the study in August 2021, the number of hospitalizations was almost 70 times higher among the vaccinated group (2,043 to 31). Assuming that the numbers were reasonably consistent over the course of the study, this would suggest that perhaps only 3 of the 31 people with natural immunity hospitalized during August tested positive for COVID, while presumably over 100 of the 2,049 hospitalized vaccinated individuals tested positive. Oh yeah, it’s a pandemic of the unvaccinated.

Given that the prevalence of false negatives in rapid COVID testing can be around 50%, depending on the timing of dose and the manufacturer, I’m particularly curious about the testing procedures. We do not know if each hospital used the same tests, and whether their COVID testing protocols changed over the course of the study. For example, consider how much it could skew the data if tests used during the month of August provided more false negatives.

This paper is premised on the notion that the percentage of positive COVID testing among vaccinated or previously infected people who are hospitalized for COVID-like symptoms accurately predicts the percentage of the general population of vaccinated or previously infected people who will test positive for COVID (or will be hospitalized for COVID). Compare this approach with, for example, the Gazit et al. study in Israel, which directly compared groups of people with natural immunity vs. vaccine-derived immunity, and determined that natural immunity was substantially more protective. Gazit et al. provided the total numbers of people in each group (vaccinated, natural immunity, and neither), along with the numbers (and percentages) who were infected with COVID, had symptomatic cases of COVID, and were hospitalized with COVID. Why would policy-makers rely on a study that uses arbitrary and unproven models for population groups, and chooses not to publish the relevant raw data, rather than rely on a far larger, direct study that provides raw data? Or many other studies like it?

As mentioned at the outset, this paper is also mathematically suspect, as should have been apparent to the authors and the CDC Director. Note that the authors don’t directly make the case that unvaccinated people with natural immunity are 5.5 times more likely to be infected with COVID than those fully vaccinated with the Pfizer or Moderna vaccines, but that conclusion is affirmatively stated by the CDC and its Director (and repeated across the world). Rochelle Walensky tweeted: “Vaccination offers higher protection against severe disease than prior #COVID19 infection. Those unvaccinated & had a recent infection were 5X more likely to have COVID-19 than those recently fully vaccinated & w/o prior infection.” Again, given that unvaccinated people without natural immunity are only 2 to 4 times more likely to be infected with the predominant COVID variant based on recent studies (e.g., from the Mayo Clinic and the UK), the CDC Director is unwittingly implying that natural immunity actually is worse than no immunity at all.

Obviously, no one believes that to be the case, let alone the CDC Director. For her to be so blind to the fundamental mathematics of vaccine epidemiology, or alternatively, so willing to deceive the American people to further an agenda, is disturbing.

Moreover, all the other “experts” touting this paper as "proof” that vaccine immunity is better than natural immunity ought to reconsider whether they really want to put so much faith in a paper that uses an indirect model rather than direct studies and comes to a conclusion that is spectacularly out of line with both (i) numerous large studies from several countries which have determined that natural immunity is as good or better than vaccine-derived immunity, and (2) the simple mathematics of COVID vaccine effectiveness relative to controls, which puts a ceiling on the the best-case possibility for vaccine immunity relative to natural immunity.

The only appropriate conclusions to be drawn from this paper as presented are that the models and metrics used in this paper probably ought not be used in the future, as they suggest conclusions divorced from scientific reality. It would be better to re-examine the data, especially the omitted data, and there might be some genuinely useful conclusions or insights available.

I don’t know whether natural immunity provides superior protection against COVID than vaccines. Based on the weight of the current evidence, that looks to be the case, particularly for people under the age of 65. Maybe that will change, or be solidified, with the emergence of the next COVID variant, or next generation vaccines. Maybe a couple rounds of booster shots will ultimately give the edge to vaccines. Typical series of vaccines often require more than two shots. In any case, at this point in time, with the current vaccines and the current COVID variants, and in light of the abundant data, there is simply no way that vaccines (matched for date of immunity) make someone 2x, let alone 5.5x, less likely to be infected than having natural immunity.

As for CDC, if you’re looking to cherry-pick some data to “prove” that vaccines are more protective than prior COVID infection, I suggest next time you look for some data (or massage some data through one of many models, just keep trying until you get the desired output number!) that shows vaccines have about 20% greater effectiveness than natural immunity. Given enough small studies, in enough populations, over enough time and COVID variants, it is almost a statistical certainly that this opportunity will present itself. This 20% figure would be far more believable, and less likely to be picked apart, even by people like me, a happily vaccinated scientist who believes that COVID vaccines are good, and COVID vaccine mandates are bad. You’ll still have plenty of advocates and acolytes willing to spread the misinformation that CDC has confirmed the inferiority of natural immunity and therefore everyone must get vaccinated.

Or better yet, would it be too much to ask for CDC to simply objective, unbiased, apolitical information?