The Cochrane Collaboration at 10: kudos and challenges

The logic of the Cochrane Collaboration remains unassailable. No one individual or group can keep abreast of the thousands of randomized trials that appear in the medical literature each year. The origin of the Collaboration dates to the late 1980s, when Iain Chalmers, Murray Enkin and Marc Keirse systematically compiled all randomized controlled trials (RCTs) in the perinatal field.1 Chalmers, with vision and determination, made the Collaboration (named after the British physician-epidemiologist, Archie Cochrane) an international enterprise with the objective of seeing similar compendia available in every area of clinical care. Today there are over 2000 systematic reviews in the Cochrane databases.2

The Cochrane reviews have grown exponentially and now cover a steadily widening range of topics. Today, an extraordinary array of end-users consult the database, and The Cochrane Library is an essential part of the armamentarium of any agency or group interested in writing clinical guidelines or assessing a given health technology. The issue is therefore not whether the Cochrane Collaboration is a success and should be sustained. The answer is a resounding Yes. Instead, we might ask: Can it be better? Is it enough? What more needs to be done?

In this issue (see page 747) Jeremy Grimshaw provides a report card on the first 10 years of the Collaboration and identifies some scope for improvement, such as quicker responses to policy-makers' needs for evidence, improving the quality of the reviews and making them more user-friendly.3 There are also gaps. Cochrane himself noted the difficulties of doing trials in areas where clinical practice is well-established (he noted cervical cancer screening) and evaluating outcomes that can be described only in subjective terms (depression, quality of life). More recently, David Naylor wrote that while “[m]any treatments are genuinely effective, [and] some diseases remit irrespective of treatment,” there are also “grey zones” of practice, such as when patients present with “fatigue, undiagnosable odd spells, [or] vague fleeting pains” and simply getting “a dose of doctor” is beneficial. He also noted the importance of new areas as science moves rapidly into practice with genomic and molecular medicine, and new diagnostic and therapeutic technologies develop faster than they can be assessed.4 For example, while we grow increasingly uncertain about the value of mammography screening,5 breast cancer screening with MRI is in clinical trials. Can the Collaboration keep up? Can it tackle preliminary assessments of rapidly developing areas, many with few or even no large trials? Should it?

A further challenge to Cochrane methodologists and systematic reviewers is “outcome creep,” the post-hoc manipulation of a priori primary objectives in order to yield “better” results. In this issue, Chan and colleagues6(see page 735) show that a large proportion of RCTs funded by the Canadian Institutes of Health Research publish results other than those for which the trials were designed. In the future we should expect systematic reviewers (and journal editors) to check trials against protocols when they are abstracting and editing, a task that may become easier with the adoption of public registries of clinical trials.

And a last challenge? If Archie Cochrane could write this paragraph, what advice might he offer the Collaboration? In an afterword published in the 1989 reprint of his seminal essay Effectiveness and efficiency (1971),7 Cochrane regretted that his discussion had neglected quality of care. He went on to describe an experience he had while a prisoner of war:

I was faced by a ghastly medical problem. I had a young Soviet POW patient dying in great pain. He was making a fearful noise in a large ward. I had no drugs or side ward. No one could talk Russian. In despair, and purely instinctively, I sat on his bed and took him in my arms. The effect was almost magical; he quietened at once and died peacefully a few hours later. I was still with him, half asleep and very stiff. I believe that by personal intervention I improved the quality of care dramatically in this case, and I know it was based on instinct and not on reason.7

This is part of the Naylor grey zone, an important and large component of good medical care. By sticking to what can be readily measured, is the Collaboration missing other things that matter to patients and policy-makers? — CMAJ