Why we need a broad perspective on meta-analysis

In the world of clinical trials and meta-analyses there is an important debate between the “lumpers” and the “splitters.” This relates to whether the overall findings of clinical trials and meta-analyses are the appropriate outcome to apply to individuals (lumping) or whether it is better to try to match the characteristics of particular patients to characteristics of subgroups within trials or meta-analyses (splitting). Although the splitters' view seems intuitively correct, there are usually substantial clinical and methodological advantages to lumping.

The generalisability and usefulness of meta-analyses are increased considerably if the individual trials cover different patient populations, settings, and concomitant routine care. For example, when a meta-analysis showed that the use of human albumin increased mortality1 this result applied to all three groups of critically ill patients studied. For patients with hypovolaemia the difference was not conventionally significant (95% confidence interval for the odds ratio 0.99 to 3.15), but it would be wrong to interpret this result as meaning that clinicians should continue to give these patients albumin. Most significant results will disappear because of lack of power if trials in a meta-analysis are split up into a large enough number of subgroups. It is more relevant that the point estimates were similar in the three subgroups studied and that the combined estimate was homogeneous. It is therefore reasonable to assume that albumin is harmful also in hypovolaemia.

Abandoning inappropriate treatments is difficult, even when they are harmful. For albumin—and many other harmful interventions—the typical argument is that if everything else fails, it should be tried as a last resort. However, albumin is likely to be harmful even in such cases. Some critically ill patients take longer to recover than others, and those are the ones now being given albumin. This is not logical. As the ultimate test, one should ask what good evidence there is that albumin is beneficial in these patients. Alternatively, one could ask whether a drug regulatory agency would be likely to approve albumin today if it were a new drug. The answers to these questions indicate that the use of albumin should be stopped altogether.

As another example, the continuous presence and support of a caregiver during childbirth (compared with usual care) has been shown to reduce the likelihood of medication for pain relief, operative vaginal delivery, caesarean delivery, and a 5 minute Apgar score of less than 7 and to improve mothers' views of their childbirth experiences.2 The 14 trials included in the meta-analysis that showed these outcomes were performed under quite different circumstances—for example, the caregiver could be a professional, a specially trained laywoman, or a friend and the hospitals included a teaching hospital in Canada and public hospitals in Africa and Guatemala serving low income women. It strengthens the credibility of a systematic review when the results are consistent across such a varied range of settings, and it would be difficult to sustain the view that “it probably doesn't apply here”—although such arguments are sometimes heard.

As the examples illustrate, patients may be harmed or deprived of treatment benefits if the results of meta-analysis are interpreted too narrowly. Clinicians therefore need to think more broadly than they are used to by their training in subjects such as pathophysiology, pharmacology, and biochemistry. Clinical researchers often adopt unnecessarily narrow entry criteria when they write protocols for clinical trials, and the splitting approach is also prevalent in the drug industry, because it is profitable to make clinicians believe that minor differences between similar drugs are important. Meta-analyses have shown repeatedly, however, that such differences can often be ignored. It is far more important to address methodological issues, such as publication bias 3 4 and the disturbing finding that reports of trials in which the method of randomisation is not described exaggerate the treatment effect (measured as the odds ratio) by about 30% on average. 5 6

A broad meta-analysis increases power, reduces the risk of erroneous conclusions, and facilitates exploratory analyses which can generate hypotheses for future research. If the results are not homogeneous, the reasons for this could be explored. The lumping approach should therefore be preferred unless there are good reasons to the contrary. Such reasons should be empirically based and not just speculative. For example, there is no good reason to suspect that pain in osteoarthritis of the knee should respond differently to an analgesic from pain in osteoarthritis of the hip. On the other hand, adopting a broad approach in general should, of course, not prevent us from looking at subgroups when there is a good reason why a treatment may work differently in different subgroups. Thus, carotid endarterectomy is beneficial in patients with severe stenosis but harmful in those with the lowest degrees of stenosis.7

A recent meta-analysis of homoeopathy has been criticised for including all kinds of homoeopathic treatments and diseases.8 Yet this broadness of approach makes a lot of sense. There is no sound empirical basis for believing that homoeopathy should be effective for some conditions and not for others. Furthermore, the theory behind homoeopathy is speculative and far fetched, so it is important to study biasing factors carefully. The summary estimate indicated that homoeopathy was effective but the analyses revealed important biases 8 9 and the authors concluded that their study had “no major implications for clinical practice.”8

If the authors had used a narrow approach and had published several small meta-analyses, each reporting the effect of homoeopathy in just one disease and including only about two to five trials, then clinicians and patients might have been misled. Many of these meta-analyses would have been positive, but it would have been impossible to detect bias.

Bias in medical research is common,3^–10 and this fact is probably the strongest single argument in favour of broad meta-analyses. The homoeopathy example can be generalised. Patients and clinicians alike are better served by a reliable answer that there is no convincing evidence that a therapeutic principle, or a class of treatments, is effective, than by an unreliable answer that a particular example of that class of treatment is effective for a particular disease.