MEDICAL journals have become adept at producing studies that generate headlines while concealing their weaknesses in the supplementary material.
Few examples illustrate this better than a recent paper in JAMA Internal Medicine claiming that the 2024-2025 covid vaccine reduced major adverse cardiovascular events (Mace) among more than one million US veterans. As usual, this was flagged up by our old friend Global Health NOW of June 18.
The headlines were predictable, as exemplified by the Washington Post of June 16. The vaccine was said to reduce cardiovascular events by almost 38 per cent. For those aged over 75, the benefits appeared even greater.
The authors concluded that vaccination reduced the risk of covid-associated cardiovascular complications and suggested that the benefits might extend even further, protecting against the ‘hidden burden’ of undetected covid infections.
It sounds impressive. Unfortunately, there is a problem, and we have the intrepid Vinay Passad to thank as explained in his substack of June 16. The study was not a randomised controlled trial. It was an observational study based on electronic health records from the US Veterans Affairs healthcare system.
Observational studies are not inherently worthless, but they carry a fundamental weakness: the groups being compared may differ in important ways which have nothing to do with the intervention being studied.
Consequently, the first question any competent reader should ask is simple: were the vaccinated and unvaccinated groups genuinely comparable? The answer appears to be no.
The clue lies not in the paper itself but in the supplementary material. There we find the cumulative mortality curves for the vaccinated and unvaccinated groups. Remarkably, these begin to separate within days or weeks of vaccination. This is biologically implausible.
The study’s principal outcome was major cardiovascular events associated with covid infection. Yet the all-cause mortality curves diverge almost immediately. The vaccinated group experiences fewer deaths from the outset.
According to Vinay Prasad, the mortality curves separate by around two weeks and the cardiovascular outcome curves by around five weeks. That is far too soon for the effect claimed.
The most obvious explanation is not that the vaccine has suddenly acquired miraculous life-preserving properties. Rather, the people choosing vaccination were healthier than those declining it. This phenomenon has a name: the healthy vaccinee effect.
Those who attend appointments, follow medical advice, engage with healthcare services and accept recommended vaccines tend also to have better underlying health and health-seeking behaviours. Conversely, those who decline vaccination are often older, frailer, sicker or less engaged with healthcare. Even sophisticated statistical adjustments frequently fail to eliminate these differences entirely.
The authors were clearly aware of this problem. Indeed, they attempted to address it by comparing people who all received influenza vaccination, with one group additionally receiving the covid vaccine. They also employed elaborate weighting procedures involving dozens of variables.
Yet despite all this statistical machinery, the mortality curves still separate almost immediately. That should have rung alarm bells. Instead, it appears to have reassured the reviewers.
The irony is that this problem is not new. Prasad and colleagues previously described precisely this phenomenon in observational studies, warning that when outcome curves separate implausibly quickly, selection bias is often the most likely explanation.
The principle is straightforward. If a treatment appears to prevent outcomes before it has had sufficient time to exert its effect, then the groups were probably never comparable in the first place.
What makes matters worse is that the absolute benefits reported are tiny. The primary outcome showed a risk reduction of only about two events per 10,000 people over eight months. Even accepting the findings at face value, we are discussing extremely small absolute differences.
Yet the paper goes further. The authors point to much larger reductions in all-cause cardiovascular outcomes and suggest these reflect protection against undiagnosed covid infections. This is speculative in the extreme.
When a study has already demonstrated evidence of residual confounding, invoking hidden infections to explain unexpectedly favourable findings looks less like science and more like storytelling. The broader lesson extends beyond this particular paper.
During the covid regime, observational studies repeatedly produced dramatic estimates of vaccine effectiveness, only for many of those estimates to shrink or disappear under closer scrutiny. The healthy vaccinee effect became a recurring feature of the literature.
Those determined to find benefits found them. Those willing to inspect the supplementary material often found something rather different.
The real scandal is not that flawed papers are published. Every scientific field produces flawed papers. The scandal is that prestigious journals continue to publish studies whose weaknesses are apparent from their own data while the media dutifully report the conclusions as established fact.
As ever, the most important findings are often hidden in the figures that journalists never read, and reviewers apparently never examine closely enough. In this case, the supplementary material may tell a more convincing story than the paper itself.
