Original Article
Ratio of means for analyzing continuous outcomes in meta-analysis performed as well as mean difference methods

https://doi.org/10.1016/j.jclinepi.2010.09.016Get rights and content

Abstract

Objective

Meta-analyses of continuous outcomes typically use mean differences (MDs) or standardized mean differences (SMDs) (MD in pooled standard deviation units). Ratio of means (RoM) is an alternative effect measure that performs comparably in simulation. We compared treatment effects and heterogeneity for RoM, MD, and SMD using empiric data.

Study Design and Setting

From the Cochrane Database (2008, issue 1), we included systematic reviews reporting continuous outcomes, selected the meta-analysis with the most (and ≥five) trials, and calculated MD (where possible), SMD, and RoM. For each pair of effect measures, we compared P-values separately for treatment effect and heterogeneity and assessed asymmetry of discordant pairs (statistically significant result for only one of two measures).

Results

Two hundred thirty-two of 5,053 reviews were included. Measures demonstrated similar treatment effects, with ≤6% discordant pairs and no asymmetry. A 0.5 SMD increase corresponded to 22 (95% confidence interval: 19, 24)% increase using RoM. There was less heterogeneity in RoM vs. MD (n = 143, P = 0.007), SMD vs. RoM (n = 232, P = 0.005), and SMD vs. MD (n = 143, P = 0.004). Comparing discordant pairs, fewer meta-analyses showed significant heterogeneity with SMD vs. RoM (P = 0.04), consistent with the known bias of SMD.

Conclusion

Empiric data from diverse meta-analyses demonstrate similar treatment effects and no large differences in heterogeneity of RoM compared with difference-based methods.

Introduction

What is new?

Key finding

  1. Comparison of empiric data demonstrates that the recently described ratio of means (RoM) method for pooling continuous outcomes in meta-analysis produces similar treatment effects and no large differences in heterogeneity when compared with traditionally used mean difference (MD) and standardized mean difference (SMD) methods.

What this adds to what is known?

  1. RoM does not suffer from some of the clinical limitations of MD methods, including the inability to handle outcomes expressed in different units (applicable to MD) and required knowledge of the pooled standard deviation—a quantity generally unknown to clinicians—for interpretation (applicable to SMD).

What is the implication and what should change now?

  1. Similar to binary outcome meta-analysis, for which ratio methods are commonly used, this study now also provides a ratio method option for meta-analysis of continuous outcomes.

Meta-analysis is a method of statistically combining results of similar studies, often randomized controlled trials [1]. For meta-analysis of continuous outcomes, the most commonly used measure of treatment effect is the difference in means [2]. If the outcome of interest is measured in identical units across trials, then the effect measure of choice for each trial is the difference in means and the pooled effect measure is the weighted average of mean differences (MDs). If the outcome of interest is measured in different units, then each trial’s effect measure is the difference in mean values divided by the pooled standard deviation (SD) of the two groups and the pooled effect measure is the weighted average of standardized mean differences (SMDs). In contrast, for binary outcomes, both difference (risk difference) and ratio (odds ratio and risk ratio) methods are commonly used.

We recently proposed and used a new ratio method to meta-analyze continuous outcomes, in which we calculated a ratio of means (RoM) (defined as the mean value in the experimental group divided by the mean value in the control group) instead of a difference for each study [3], [4], [5], [6]. Others have used this method [7], [8], [9] and incorporated it in freely available meta-analysis software [10]. As an illustration, Table 1 shows pooled continuous data using MD, SMD, and RoM from two meta-analyses [4], [11]. The three methods give similar results. The point estimates are similar in direction (i.e., a positive MD or SMD corresponds to RoM greater than 1, whereas a negative MD or SMD corresponds to RoM less than 1) and yield similar treatment effect P-values. In addition, statistical heterogeneity, measured as I2, the percentage of total variation in results across studies due to heterogeneity rather than chance [12], [13], is also similar. Equations for calculating RoM and a worked example are provided in the Appendix.

Advantages of RoM include the ability to pool studies with outcomes expressed in different units (vs. MD) and ease of clinical interpretation (vs. SMD) because it does not require knowledge of the pooled SD, a quantity generally unknown to clinicians. For example, the second meta-analysis in Table 1 describes the effect of acetaminophen on osteoarthritis pain [11]. MD cannot be calculated because the studies used different pain scores. Although SMD is easily calculated and shows that acetaminophen significantly decreases overall pain by 0.25 pooled SD units, it is difficult to communicate the importance of this effect to individual patients. In contrast, pooling data with RoM generates a more easily interpretable 15% decrease in overall pain. A disadvantage of RoM is that it requires the means of continuous variables in all trials included in a meta-analysis to have the same sign. Although essentially all biological continuous outcomes have positive values, meta-analyses may be used to pool changes in continuous outcomes over time or investigator-generated scales, which may be positive or negative.

We have previously demonstrated comparable statistical performance (bias, coverage, power, and heterogeneity) of RoM compared with SMD and MD using simulation methods [14]. However, in addition to statistical properties, the choice between a difference and ratio method for a specific situation should also be determined by the biological effect of the treatment as either additive or relative for different control group values. Whether absolute or relative changes are more preserved across studies can be determined through empirical comparisons. For binary outcomes, empirical comparisons between difference methods (risk difference) and ratio methods (risk ratio and odds ratio) using published meta-analyses have shown higher heterogeneity of risk difference [15], [16], suggesting that relative differences are more preserved than absolute differences as baseline risk varies. The objective of this study was to conduct a similar empirical comparison of treatment effects and heterogeneity of the ratio method, RoM, to the difference methods (MD and SMD) in a broad range of published meta-analyses of continuous outcomes.

Section snippets

Methods

We searched the Cochrane Database of Systematic Reviews (2008, issue 1) for all reviews in which any of the following words or phrases appeared in the title, abstract, or keywords: “wmd,” “weighted mean difference,” “smd,” “standardized mean difference,” or “standardised mean difference.” We chose this search strategy to identify reviews in which a pooled continuous outcome was sufficiently important to be highlighted in one of these fields. Retrieved reviews containing at least one

Search results

Among 5,053 reviews in the Cochrane Library (2008, issue 1), 897 (18%) mentioned one of the MD methods or their acronyms in the title, abstract, or keywords. As shown in Appendix Fig. 5, of these 897 reviews, 232 (26%) contained meta-analyses that met our inclusion criteria. Six hundred twenty-eight (70%) were excluded because they did not contain meta-analyses of at least five trials. Thirty-seven (4%) were excluded because they contained meta-analyses of at least five trials but with a

Discussion

Using empiric data systematically abstracted from many clinically diverse meta-analyses, we found that the recently developed RoM method provides similar treatment effect estimates compared with the traditionally used mean difference methods (MD and SMD), with SMDs of 0.2, 0.5, and 0.8, corresponding to increases in RoM of approximately 8%, 22%, and 37%, respectively. This similarity was demonstrated both when comparing discordant pairs using the standard clinical definition for statistical

Conclusions

Empiric analysis of clinically diverse published meta-analyses suggests that, on average, RoM yields similar pooled treatment effect estimates and heterogeneity compared with traditionally used mean difference methods. These results complement recent simulation findings showing comparable statistical performance characteristics to mean difference methods and suggest that RoM is a reasonable alternative to mean difference methods for pooling continuous outcomes in meta-analysis.

Acknowledgments

The authors would very much like to thank Dr Ruxandra Pinto for conducting some of the statistical analyses. The study received no specific funding. J.F. is supported by a Clinician Scientist Award from the Canadian Institutes of Health Research (CIHR) and J.B. by CIHR grant no. 84392. CIHR had no involvement in the conduct of this study.

References (22)

  • R. DerSimonian et al.

    Meta-analysis in clinical trials

    Control Clin Trials

    (1986)
  • J.J. Deeks et al.

    Statistical methods for examining heterogeneity and combining results from several studies in a meta-analysis

  • J.O. Friedrich et al.

    Meta-analysis: low-dose dopamine increases urine output but does not prevent renal dysfunction or death

    Ann Intern Med

    (2005)
  • N.K.J. Adhikari et al.

    Nitric oxide improves oxygenation but not mortality in acute lung injury: meta-analysis

    BMJ

    (2007)
  • S. Sud et al.

    Effect of mechanical ventilation in the prone position on clinical outcomes in patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis

    CMAJ

    (2008)
  • S. Sud et al.

    High frequency oscillation in patients with acute lung injury and acute respiratory distress syndrome (ARDS): systematic review and meta-analysis

    BMJ

    (2010)
  • P.J. Karanicolas et al.

    The impact of prophylactic dexamethasone on nausea and vomiting after laparoscopic cholecystectomy: a systematic review and meta-analysis

    Ann Surg

    (2008)
  • R. Kunz et al.

    Meta-analysis: effect of monotherapy and combination therapy with inhibitors of the renin–angiotensin system on proteinuria in renal disease

    Ann Intern Med

    (2008)
  • P.W.H. Peng et al.

    Use of gabapentin for perioperative pain control—a meta-analysis

    Pain Res Manag

    (2007)
  • WINPEPI program COMPARE2. Available at http://www.brixtonhealth.com/pepi4windows.html. Accessed January 25,...
  • Cited by (150)

    View all citing articles on Scopus

    Preliminary results were presented at the XVI Cochrane Colloquium; October 2008; Freiburg, Germany: Friedrich JO, Adhikari NKJ, Pinto R, Beyene J. Empiric comparison of ratio and difference methods for analyzing continuous outcomes in meta-analysis. Available at http://colloquiumabstracts.cochrane.org/.

    1

    Current address: Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences, McMaster University, MDCL 3202, Hamilton, Ontario, Canada L8N 3Z5.

    View full text