Original article
Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis

https://doi.org/10.1016/S0895-4356(01)00377-8Get rights and content

Abstract

Asymmetry in funnel plots may indicate publication bias in meta-analysis, but the shape of the plot in the absence of bias depends on the choice of axes. We evaluated standard error, precision (inverse of standard error), variance, inverse of variance, sample size and log sample size (vertical axis) and log odds ratio, log risk ratio and risk difference (horizontal axis). Standard error is likely to be the best choice for the vertical axis: the expected shape in the absence of bias corresponds to a symmetrical funnel, straight lines to indicate 95% confidence intervals can be included and the plot emphasises smaller studies which are more prone to bias. Precision or inverse of variance is useful when comparing meta-analyses of small trials with subsequent large trials. The use of sample size or log sample size is problematic because the expected shape of the plot in the absence of bias is unpredictable. We found similar evidence for asymmetry and between trial variation in a sample of 78 published meta-analyses whether odds ratios or risk ratios were used on the horizontal axis. Different conclusions were reached for risk differences and this was related to increased between-trial variation. We conclude that funnel plots of meta-analyses should generally use standard error as the measure of study size and ratio measures of treatment effect.

Introduction

Funnel plots—scatter plots in which the treatment effects estimated from individual studies on the horizontal axis are plotted against a measure of study precision on the vertical axis—have been proposed as a means of detecting publication bias in meta-analysis [1]. In the absence of bias the graph resembles a symmetrical inverted funnel because the treatment effect estimates from smaller studies scatter more widely at the bottom of the graph, with the spread narrowing with increasing precision among larger studies. If there is publication bias because smaller studies which show no statistically significant effects remain unpublished 2, 3, then the funnel plot will appear asymmetrical 4, 5. Funnel plot asymmetry cannot, however, be interpreted as proof of publication bias in meta-analysis [6]. Asymmetry could also result from the overestimation of treatment effects in smaller studies of inadequate methodological quality [7]. Furthermore, heterogeneity of treatment effects will lead to funnel plot asymmetry if the true treatment effect is larger in the smaller trials 6, 8. For example, if a combined outcome is considered then substantial benefit may be seen only in patients at high risk for the component of the combined outcome which is affected by the intervention [9]. Trials conducted in high-risk patients will also tend to be smaller, because of the difficulty in recruiting such patients.

Funnel plots were first used in educational research and psychology [1], mainly for meta-analyses of continuous outcome variables in which standardized mean difference was plotted against sample size. In medical research vertical axes based on the standard error or variance of the treatment effect estimate have been increasingly used. A majority of trials in medicine have binary outcomes, and treatment effects are usually expressed as risk or odds ratios, although risk differences may also be used to measure treatment effects.

Meta-analysts thus face a wide array of choices for both vertical and horizontal axes in funnel plots. This leads to the danger that the funnel plot chosen for a particular meta-analysis may be that which best conveys the message desired by the investigator, or may not be appropriate for detecting bias [10]. The purpose of this article is to provide guidelines for the choice of axes in funnel plots of meta-analyses with binary outcomes.

Section snippets

Choice of vertical axis in funnel plots: case study

The randomized controlled trials of magnesium treatment in the prevention of death following myocardial infarction (Table 1) are a well known example where publication bias, demonstrated by an asymmetrical funnel plot 5, 6, has been suggested as an explanation for the discrepancy between meta-analyses which showed a clear beneficial effect of magnesium therapy on mortality 11, 12 and a subsequent large trial which showed no effect [13]. Fig. 1 shows funnel plots for these 16 trials, using six

Choice of horizontal axis in funnel plots: empirical study

The choice of treatment effect measure may affect the interpretation of randomized trials and meta-analyses [16]. In practice, most meta-analyses use ratio measures of treatment effect (odds ratio or relative risk), although risk differences are sometimes also used. We conducted an empirical study to investigate whether the prevalence of funnel plot asymmetry in published meta-analyses depends on the choice of treatment effect.

As described in detail elsewhere [8], we hand searched volumes

Discussion

The potential for bias in the location, selection or conduct of the component studies included in meta-analyses is increasingly recognized. Funnel plots are a useful graphic means of checking whether “small study effects”—a tendency for treatment effect estimates in small studies to differ from those in larger studies—may have distorted the results of a meta-analysis [19]. This could be due to publication bias, other reporting biases, low methodological quality of smaller studies or true

References (39)

  • M. Egger et al.

    Bias in meta-analysis detected by a simple, graphical test

    Br Med J

    (1997)
  • K.F. Schulz et al.

    Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials

    JAMA

    (1995)
  • G. Davey Smith et al.

    Who benefits from medical interventions? Treating low risk patients can be a high risk strategy

    Br Med J

    (1994)
  • K.K. Teo et al.

    Effects of intravenous magnesium in suspected acute myocardial infarctionoverview of randomised trials

    Br Med J

    (1991)
  • S.M. Horner

    Efficacy of intravenous magnesium in acute myocardial infarction in reducing arrhythmias and mortalitymeta-analysis of magnesium in acute myocardial infarction

    Circulation

    (1992)
  • ISIS-4A randomised factorial trial assessing early oral captopril, oral mononitrate, and intravenous magnesium sulphate in 58,050 patients with suspected acute myocardial infarction

    Lancet

    (1995)
  • C. Poole et al.

    Random-effects meta-analyses are not always conservative

    Am J Epidemiol

    (1999)
  • C.D. Naylor et al.

    Measured enthusiasmdoes the method of reporting trial results alter perceptions of therapeutic effectiveness?

    Ann Intern Med

    (1992)
  • C.B. Begg et al.

    Operating characteristics of a rank correlation test for publication bias

    Biometrics

    (1994)
  • Cited by (0)

    View full text