The Contrast Between Opinion and Misinformation
Did the New York Times behave irresponsibly to publish an op-ed about masks?
On February 21 the New York Times published an article by one of its regular opinion columnists, Bret Stephens, titled “The mask mandates did nothing. Will any lessons be learned?” In it, Stephens declares: “the verdict is in: Mask mandates were a bust.” He bases his opinion on an analysis of studies published in the Cochrane Database of Systematic Reviews, which Stephens has decided is “The most rigorous and comprehensive analysis of scientific studies conducted on the efficacy of masks for reducing the spread
of respiratory illnesses—including Covid-19…”
Because this New York Times columnist has decided that Cochrane reports are the “gold standard for its reviews of health care data” he feels entitled to write that: “The Cochrane report ought to be the final nail in this particular coffin.”
Stephens, of course, is entitled to his opinion and the piece in the New York Times is clearly labeled as opinion. Still, the Cochrane meta-analysis on face masks has been widely criticized by experts and is far from the last word on the issue of whether masks work to reduce the spread of the virus that causes Covid-19. Many people will read Stephens’ article and an unknown number may be convinced that his position is definitive and will therefore not wear masks in public places. Given that Stephens is not right about his conclusions, this raises the question of whether the New York Times had a responsibility to add context and nuance to Stephens’ article in order to protect the public’s health.
The Cochrane Review is Controversial
Let’s first review why the Cochrane report is so controversial. It is indeed the case that Cochrane, a British non-profit, has an enormous library of health studies and its authors perform sometimes valuable reviews and meta-analyses of studies on particular topics. Cochrane reviews favor a particular type of study, called the randomized controlled trial (RCT) in their analyses. To answer some medical questions, RCTs are indeed the best approach. In an RCT, study participants are randomized to different interventions or conditions, minimizing the chance that extraneous variables will influence the outcome. For instance, if we want to know if drug A is superior to drug B for treating a specific illness, the most rigorous approach is to perform an RCT in which patients with the illness are randomized to receive either drug A or drug B. This way, the only thing that differs between the two groups is which drug they receive and an accurate assessment of whether one drug is better than the other can be made.
But RCTs have limitations that make them less ideal for addressing certain questions. For one thing, they are very expensive, involving months of planning, lots of personnel, and many patient visits for research assessments. That means that the number of people who can be included in an RCT is limited and therefore the results of the trial often do not reflect how a larger group of people might fare. Hence, it is frequently noted that RCTs may not reflect “real world” conditions. Sometimes, the number of participants in an RCT is too small to show a statistically significant difference in outcome between groups, even when one exists. In addition, RCTs only work when it is clear that patients enrolled in the trial are actually adhering to the research interventions. Adherence is easier to assess when it is a matter of whether pill A or pill B is being taken and harder with more complicated interventions. Finally, RCTs can raise ethical issues when a treatment or intervention is already known to work.
For all of these reasons, a reliance on RCTs alone, as is often the case with Cochrane reports, may not give an accurate picture of how well an intervention works. In the case of masks, the authors of the Cochrane report themselves note that “relatively low adherence with the interventions during the studies hampers drawing firm conclusions.” In other words, it is very difficult to know whether people enrolled in trials in which one group is randomized to wear masks and another to not wear masks actually follow study directions. What RCTs of masks to prevent the spread of respiratory viruses really show is whether advice to wear masks works, not whether actually wearing them works. Also, because some of the RCTs in the Cochrane report involved small numbers of study participants, they may not be able to demonstrate statistically significant benefits of masks. On a population basis, even small benefits of face masks could have important effects on limiting serious consequences of Covid-19 or other respiratory illnesses caused by airborne viruses. There's also the fact that most of the RCTs were of masks for other viruses - like flu, which are much less transmissible and so it might look like masks don't really work but it was just because not many people were getting infected anyway.
Other Types of Evidence Show Masks Work
Hence, while it is true that RCTs of masks have generally been equivocal about whether they work, other types of evidence give us more confidence that they do. There are mechanical studies, for example, in which scientists use mannequins to test whether masks block the spread of airborne viruses. These have generally shown that masks prevent spread of the virus that causes Covid-19.
Equally important are observational studies. Unlike RCTs, observational studies do not involve randomization but rather rely mainly on natural experiments in which some people are exposed to one condition and others to another condition. In the case of masks, for example, there were a number of instances in which one county mandated them during the Covid-19 pandemic but another nearby country did not, giving researchers the opportunity to see whether the county with the mask mandate had lower rates of infection or complications like hospitalization and death. Many such observational studies indeed demonstrate a positive effect for masks. The advantage of such observational studies is that they involve much larger numbers of people than RCTs and can give “real-world” evidence about whether an exposure has consequences. Remember that no one has ever done an RCT to see whether cigarette smoking causes lung cancer (that would of course be highly unethical). The evidence that cigarettes cause cancer comes from animal studies and human observational studies. Hence, it is wrong to discount observational studies when we are trying to answer some important health questions.
Many other flaws in the Cochrane report onmasks have been identified by experts. These include:
Most of the 78 studies used in the analysis did not involve Covid-19 but rather other viruses, including influenza. Some of the studies were done during seasons of high influenza prevalence and some during low prevalence, which impacts whether masks will work. Lumping all these studies together can dilute out real effects of face masks.
The studies analyzed by the Cochrane report mainly looked at whether masks prevent acquisition of respiratory viral infection, not whether they prevent spread. In an article in The Conversation, four experts in the field note that “A previous systematic review found face masks worn by sick people during an influenza epidemic reduced the risk of them transmitting the infection to family members or other carers.”
The authors of The Conversation article reacting to the Cochrane report concluded that “There is strong and consistent evidence for the effectiveness of masks and (even more so) respirators in protecting against respiratory infections. Masks are an important protection against serious infections.”
Given this profound difference of opinion on the quality of the Cochrane report, it is important to ask whether Bret Stephens and the New York Times acted responsibly in publishing a piece with a one-sided view that failed to alert readers to the contrary opinions of so many other experts. The Washington Post handled things very differently on its opinion page, noting in its story on the Cochrane report that it “has been criticized for several big flaws.” It goes on to explain to readers exactly what those flaws in the analysis are, giving us the opportunity to see the Cochrane report in context and to understand where the science is uncertain, rather than settled as Brett Stephens would have us believe. Even the authors of the Cochrane report themselves were less certain about what they had found than Stephens, concluding that “There is uncertainty about the effects of face masks. The low to moderate certainty of evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect.”
The issue here is two-fold. First is whether masks do in fact reduce the risk of acquisition and spread of airborne respiratory viruses like the one that causes Covid-19. Here, our read of the available data is that they do, although some types of masks, like N95 respirators, are better than others, like surgical masks, and they need to be worn consistently during high-risk periods and in high-risk places. Second, we believe that even when expressing their opinions, columnists and journalists have special responsibilities when the topic involves people’s health. This is not an area for which every “side” is necessarily valid—scientific research attempts to tease apart what things are more likely to be true than others. This allows us to make recommendations about what people can do to prevent and treat diseases. Bret Stephens had a responsibility to acknowledge the preponderance of scientific opinion that the Cochrane review on face masks is flawed and not definitive and the New York Times editors should have enforced that responsibility. Failing to do so represents a failure of journalistic ethics and a threat to the public’s health.