THERE was much rejoicing in Global Health NOW when it reported that JAMA Network Open published a study ‘blocked from publication in a CDC journal by the Trump administration’, claiming that the latest covid vaccine was about 55 per cent effective against covid-associated hospitalisation.
The vaccine also cut the risk of urgent care and visits to the emergency department by 50 per cent. Except that is not actually what the study showed. It seems that covid vaccine effectiveness can mean anything that its proponents want it to mean.
The study reported in the JAMA article is not a randomised controlled trial comparing vaccinated and unvaccinated people in the population and asking how many in each group become ill. Instead, it uses a test-negative design, now one of the principal methods used to estimate vaccine effectiveness.
This design begins with people who are sufficiently ill to seek medical care and who have symptoms broadly compatible with covid. Everyone is tested. Those who test positive become the ‘cases’; those who test negative become the ‘controls’. The researchers analyse whether those testing positive were less likely to have been vaccinated than those testing negative. Vaccine effectiveness is then calculated.
A reported vaccine effectiveness of 55 per cent does not mean that vaccination more than halves your absolute risk of being hospitalised. It means that, among a selected group of symptomatic people who sought care and underwent testing, the odds of testing positive were lower among the vaccinated.
Using the crude numbers presented in the paper, the absolute differences are modest. Among adults attending emergency departments or urgent care, around 4.9 per cent of unvaccinated attendees tested positive compared with about 2.1 per cent of vaccinated attendees. For hospitalisations the figures were about 4.2 per cent versus 1.9 per cent.
Those are differences of only two or three percentage points, not the dramatic reductions implied. Even these figures are not true absolute risk reductions because the denominators include only people who were already ill enough to present with what the authors call ‘covid-like illness’ which in itself raises significant issues.
Most people probably imagine this means fever, cough and pneumonia. The supplementary material tells a different story. The diagnostic net is cast astonishingly wide. Alongside respiratory illnesses are headache, diarrhoea, abdominal pain, fatigue, altered mental status, elevated inflammatory markers, sepsis, shock and even pregnancy-related viral conditions. A patient admitted with abdominal pain who happens to test positive for SARS-CoV-2 can potentially contribute to estimates of ‘covid-associated’ hospitalisation.
That inevitably raises the chestnut of whether the study is measuring hospital admissions because of covid or simply admissions with covid. The authors themselves concede that patients may have sought care for reasons unrelated to covid. An acknowledgement should attenuate any confident claims that vaccination prevents severe covid requiring hospital admission.
Remarkably, despite the population having lived through several years of widespread SARS-CoV-2 circulation, the analysis did not adjust for previous infection or previous covid vaccination. Again, the authors explicitly acknowledge this limitation while simultaneously claiming that vaccination provides protection ‘beyond existing immunity’. However, if people declining the latest booster had recently recovered from covid infection, their naturally acquired immunity could easily influence the apparent effectiveness attributed to vaccination.
Behavioural differences present another challenge. Vaccinated and unvaccinated people are rarely identical. They may differ in healthcare-seeking behaviour, willingness to be tested, exposure risk and underlying frailty. Statistical adjustment can account only for factors that are measured. It cannot eliminate unknown or unmeasured differences. Once again, the authors acknowledge the possibility of residual confounding.
The study included only 60 vaccinated hospital cases, yet a sweeping public health message is generated. Moreover, the median follow-up after vaccination was only around seven weeks, giving little indication of durability beyond the immediate post-vaccination period.
The test-negative design is popular because it is inexpensive. But it rests on assumptions including similar healthcare-seeking behaviour among vaccinated and unvaccinated people, unbiased testing practices, representative controls and adequate adjustment for prior immunity.
Beyond problems with the methodology is the way its results are communicated. Journal abstracts and media reports routinely translate odds-based associations into language suggesting direct reductions in real-world risk. Readers could understandably conclude that vaccination literally halves their chance of hospitalisation. That is not what this study showed.
During the covid years we became accustomed to relative risk reductions presented without context and observational findings promoted as if they were the result of rigorous trials. In the case of the latest study a more accurate conclusion would have been rather less dramatic: among adults already ill enough to seek medical care with broadly defined covid-like illness, recent vaccination was associated with lower odds of testing positive for SARS-CoV-2.
That is a perfectly respectable scientific observation. It just doesn’t make quite such a compelling headline.
