The Cochrane Controversy
And the existential crisis for evidence-based medicine
Evidence-based medicine faces a smoldering, slow-motion crisis as every few months new headlines pop up about a controversy between Peter Gøtzsche and the Cochrane Collaboration. There are both personal and political sides to the story, with magazine stories focusing on palace drama and intrigue, but what may seem like an arcane debate is likely to have significant ramifications on how data are analyzed and what makes a scientific fact a fact.
For over 25 years, patients and experts have come to trust Cochrane Collaboration’s evidence-based systematic reviews to provide authoritative guidance on which medical treatments work and which ones don’t. Cochrane believes so strongly in this type of analysis, their logo is literally a figure from one of their scientific reviews that concluded premature babies were safer if their mothers received a medication prior to birth. Their work has reinforced important treatments we now take for granted, like folic acid can prevent spina bifida. Busy doctors use their summaries routinely, if not daily, and the reviews are widely respected, having “a well‐deserved reputation of excellence,” according to John Ioannidis, an evidence-based medicine expert at Stanford. Of course, despite the best efforts of the reviewers, uncertainties remain, particularly related to the availability of high-quality evidence, but these reviews are, arguably, the closest thing medicine has to a gold standard.
The crisis has a number of angles that border on insider gossip. Briefly, Cochrane has behavior guidelines for members to ensure they do not compromise the perceived neutrality of the collaboration. Gøtzsche, a founding member but well known “firebrand” for his rigid views on evidence quality, bias and transparency, has been accused of repeatedly violating those guidelines and was voted off the board. Gøtzsche and his defenders argue his efforts are in defense of science, pushing for the highest standards of evidence and trying to ensure Cochrane reviews lack bias through transparency. His methods may be unorthodox or impolite, but they should not be silenced.
Juxtaposed on other internal debates, like whether Cochrane’s business model should be top-down or grassroots, the disagreement has unfortunately become personal and acrimonious with lawsuits and factions. (See here, here, or here for more on the controversy.) With the media’s focus on individual narratives, Gøtzsche is portrayed either as heroic rogue scientist, fighting against subtle biases that inevitably arise when a growing proportion of our medical data come from capitalistic enterprise, or as an out-of-touch champion of a halcyon era of evidence-based medicine where rigorously examined numbers and data could tell the whole story, regardless of context.
These simplistic narratives of the idealist vs more practical realists only touch on the larger philosophical debates at issue (that I’ll address in more detail below). But, most importantly, there is also a key question that touches the core of science and epistemology, the answer to which is likely to affect how science should be assessed and adjudicated: What constitutes scientific evidence and how do you weigh it?
The crisis at Cochrane is inseparable from increasing questions about the problems with evidence-based medicine (EBM) in practice. Clearly, EBM has seen a meteoric rise. Resistant to “eminence-based medicine” – a tongue-in-cheek way to describe medicine practiced by experienced physicians with little evidence behind their decisions – the movement to rely primarily on evidence for clinical decision-making is not even 30 years old.
But, in what could be argued is a mid-life crisis, in 2014, prominent evangelists of EBM noted its unintended foibles and suggested reform. The debate, then, can be framed as one of diagnosing the problem and deciding what to do about it. Cochrane relies on what is widely agreed is the highest quality evidence, the randomized controlled trial (RCT), published in peer-reviewed journals. But, some argue, that data are often biased, both in individual instances of RCTs, and in the fact that most RCT data comes from industry-funded sources.
Specifically, Gøtzsche’s critics contend that his views about the safety of certain medications like the HPV vaccine or antidepressants: “consistently expresses the most extreme views in the most dramatic and misleading way.” As he told Undark, Gøtzsche argues that industry interests dominate the available evidence, and Cochrane’s reliance on it makes it “a servant to the industry.” With the cost of RCTs inexorably rising, Gøtzsche’s concern seems well placed, as only industry has the financial means to undertake the large, expensive trials that Cochrane uses for their analyses.
Yet critics also argue that RCTs are stilted: their experimental conditions are so controlled as to be artificial, bearing little resemblance to actual clinical practice. While “positive” trials can show statistical significance, they may not have much meaningful clinical relevance, a conundrum neatly captured in Richard Lehman’s whimsical piece about whether to prescribe spironolactone for an elderly patient with many co-morbid diseases, where the physician recounts the concerns over analyzing complicated data from various clinical trials only to realize the patient is most concerned about quality of life rather than living as long as possible. Therefore, reformists argue, RCTs are important, but must be weighed not only against other types of evidence, but against the individual circumstances and context a patient may be facing at the time of their illness. These EBM reformists argue now against the unthinking application of guidelines, checklists or algorithms, and support individualization of care if necessary. This has given rise to a now well known term: the tyranny of the RCT. There are unintended consequences of relying too heavily on RCT evidence, with scientists questioning decades of dental dogma about flossing a prominent recent example.
As one BMJ article by Jefferson and Jørgensen points out, the process of publishing RCTs can lead to “unfathomable bias” simply because of the massive distillation required to turn the thousands of pages of details needed to run an RCT into a tight 10-page journal article. Such inevitable compressions and distortions lead the authors to argue that Cochrane may need to ignore such journal articles: “By the law of Garbage In Garbage Out, whatever we produce in our reviews will be systematically assembled and synthesised garbage with a nice Cochrane logo on it,” they wrote.
Jefferson and Jørgensen argue we should index all trial information to avoid the bias-through-distillation problem. This labor-intensive project would allow more people to peer into that specialized circle of the clinical trial so we can view them all together in context. This is an important step, but does not seem to get at a major issue of concern shared by them and Gøtzsche: the preponderance of evidence comes from industry-funded studies.
And that gets to the crux of the debate, but one that scientists seem to be talking around. If work done by the Cochrane Collaboration either to index or review high-quality trials is a public good and worthy of funding, then the data to produce those reviews should also be considered a public good. Having industry take part in clinical trials is not an issue as long as there is a significant proportion of data released with minimal concerns about conflict of interest. Yet, as the proportion of trial data coming from industry has increased, there hasn’t been a concomitant outcry for the production of more publicly funded data. Decades ago, government support for clinical trials was instrumental in the famous cases of polio and influenza vaccines. Rather than super-basic research, should governments fund more applicable science, as suggested in this New Atlantis article on Saving Science?
Moreover, despite Gøtzsche’s efforts to push Cochrane away from bias, the question is not the foolhardy errand to try to do away with all bias, but, after minimizing it to the greatest extent possible, how to be transparent and reflective about whatever biases surface in the data, as Greenhalgh and colleagues note.
RCTs and EBM have revolutionized medicine. But how scientists move past these known flaws will determine how we weigh different types of evidence. It’s unfortunate that such a weighty issue is getting lost in lurid headlines.