Review
A decade of individual participant data meta-analyses: A review of current practice

https://doi.org/10.1016/j.cct.2015.06.012Get rights and content

Abstract

Introduction

Individual participant data (IPD) systematic reviews and meta-analyses are often considered to be the gold standard for meta-analysis. In the ten years since the first review into the methodology and reporting practice of IPD reviews was published much has changed in the field. This paper investigates current reporting and statistical practice in IPD systematic reviews.

Methods

A systematic review was performed to identify systematic reviews that collected and analysed IPD. Data were extracted from each included publication on a variety of issues related to the reporting of IPD review process, and the statistical methods used.

Results

There has been considerable growth in the use of “one-stage” methods to perform IPD meta-analyses. The majority of reviews consider at least one covariate other than the primary intervention, either using subgroup analysis or including covariates in one-stage regression models. Random-effects analyses, however, are not often used.

Reporting of review methods was often limited, with few reviews presenting a risk-of-bias assessment. Details on issues specific to the use of IPD were little reported, including how IPD were obtained; how data was managed and checked for consistency and errors; and for how many studies and participants IPD were sought and obtained.

Conclusion

While the last ten years have seen substantial changes in how IPD meta-analyses are performed there remains considerable scope for improving the quality of reporting for both the process of IPD systematic reviews, and the statistical methods employed in them. It is to be hoped that the publication of the PRISMA-IPD guidelines specific to IPD reviews will improve reporting in this area.

Introduction

The aim of individual participant data (IPD) systematic reviews and meta-analyses is to obtain all the original, raw participant data from all studies on a specified topic, in order to reanalyse the data and pool it across studies. While obtaining all the original data from all relevant studies may be time consuming and difficult, IPD meta-analysis is recognised as having many advantages over meta-analysis based on data reported in publications, and is considered the “gold standard” for meta-analysis [1], [2], [3].

Ten years ago the first review of the practice and reporting of individual participant data meta-analyses was published in Clinical Trials [4]. At that time IPD meta-analysis was still in its early stages with numbers of publications being limited before the late 1990s. The development of statistical methods in the area was also in its infancy, with key methods papers having been published only a few years before [5], [6], [7]. The review found that reporting of the processes of IPD meta-analyses was generally poor, particularly with regard to how much of the total IPD was obtained for analysis, and with poor reporting of statistical methods. Statistical analysis was also limited, with little investigation of heterogeneity, and most meta-analyses focussing on calculating overall treatment effects, with little consideration given to subgroup analyses, or how factors such as a participant's age might modify the effectiveness of a treatment. Another review performed in 2008 and published in 2010 [8] came to broadly similar conclusions.

In the ten years since that first review was published much has changed. The number of systematic reviews and meta-analyses published continues to grow, and consequently IPD meta-analysis has also grown in popularity. Advances in computing have also made meta-analyses easier to perform; most methods for IPD meta-analysis can now be implemented in all major statistical software packages. As the number of IPD meta-analyses grows there is an increased need to ensure high quality of conduct reporting of these analyses, to avoid some of the problems identified in the original review.

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [9] is widely used as guidance on how to report a systematic review and meta-analysis. Recently a new PRISMA-IPD statement has been released specifically to guide the conduct and reporting of IPD reviews and meta-analyses [10]. As part of the process of creating this statement, a review of current practice in reporting of IPD reviews was conducted to identify areas where reporting was poor to inform development of the guidance. This paper presents the results of this review, updated to 2014, and expanded to consider statistical methods used in IPD meta-analyses.

Section snippets

Methods

The aim of this review was to identify published systematic reviews of medical interventions that sought to obtain and meta-analyse IPD. It was intended to obtain a representative sample of recent reviews for analysis, rather than find all such reviews, so an exhaustive database search was not performed. A MEDLINE search was performed including terms “individual participant/patient data”, “meta-analysis” and “systematic review”. This search was originally performed in January 2013 and was

Results

The original and updated search together identified 1371 potentially relevant records. After checking titles and abstracts 184 papers were considered for inclusion. After obtaining full texts (where readily available) and further checking, 100 systematic reviews of IPD were included in this review [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43],

Quality of reporting

The included reviews varied considerably in how they reported aspects of the review process, such as the search strategy, and quality assessment process. The numbers of reviews reporting key aspects of the IPD review process are summarised in Table 1. In general, aspects of the review process common to all systematic reviews were well reported, with, for example, 80% of reviews reporting details of the search strategy and search process. Inclusion and exclusion criteria were also generally

Statistical methods

The review of 2005 identified two distinct statistical approaches for IPD meta-analysis [4]. In a two-stage approach studies are analysed individually to obtain summary results for each study in the first stage, and the summary results pooled across studies as in a conventional meta-analysis in the second stage. In a one-stage approach all data from all studies are analysed simultaneously, in a single model, usually a regression model. This distinction in approaches remains in current practice.

Discussion

Much has changed in IPD systematic reviews and meta-analyses in the last ten years. The most notable change is the switch from a predominance of two-stage methods, where studies are analysed separately and summary results poled across studies; to one-stage methods where all IPD are analysed simultaneously. This is probably driven by growing familiarity with these methods, improved software, and recognition that regression models offer the greatest flexibility for IPD analysis. Another key

Conclusion

The last ten years have seen substantial changes in how IPD meta-analyses are performed, particularly with the growth of the use of one-stage regression models. There remains considerable scope for improving the quality of reporting both the process of IPD systematic reviews, and the statistical methods employed in them. It is to be hoped that the publication of the PRISMA-IPD guidelines specific to IPD reviews will improve reporting in this area [10].

Funding

The systematic review described in this paper was performed, in part, to inform the development of PRISMA-IPD. That project was supported, in part, through a National Institute for Health Research (NIHR) Senior Investigator personal award (Dr L. Stewart), and as part of the MRC–National Institute for Health Research Methodology Research Programme (grant ID 88053). The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data;

Acknowledgements

We would like to thank Kath Wright for performing the literature searches for this review, and Dean Langan and Alison Smith for their assistance in the study selection and data extraction processes. We also thank the members of the PRISMA-IPD development group for their comments on early versions of this review.

References (112)

  • M.W.J. Koeter et al.

    Effect of early and late compliance on the effectiveness of acamprosate in the treatment of alcohol dependence

    J. Subst. Abus. Treat.

    (2010)
  • S. Kufner et al.

    A meta-analysis of specifically designed randomized trials of sirolimus-eluting versus paclitaxel-eluting stents in diabetic patients with coronary artery disease

    Am. Heart J.

    (2011)
  • S. Laporte et al.

    Individual patient data meta-analysis of enoxaparin vs. unfractionated heparin for venous thromboembolism prevention in medical patients

    J. Thromb. Haemost.

    (2011)
  • P. Lehert et al.

    Racecadotril for childhood gastroenteritis: an individual patient data meta-analysis

    Dig. Liver Dis.

    (2011)
  • M.N. Mavros et al.

    A systematic review: treatment and prognosis of patients with fibrolamellar hepatocellular carcinoma

    J. Am. Coll. Surg.

    (2012)
  • W.G. Melsen et al.

    Attributable mortality of ventilator-associated pneumonia: a meta-analysis of individual patient data from randomised prevention studies

    Lancet Infect. Dis.

    (2013)
  • S.G. Muthuri et al.

    Effectiveness of neuraminidase inhibitors in reducing mortality in patients admitted to hospital with influenza A H1N1pdm09 virus infection: a meta-analysis of individual participant data

    Lancet Respir. Med.

    (2014)
  • S.J. Nicholls et al.

    Meta-analysis of comparative efficacy of increasing dose of atorvastatin versus rosuvastatin versus simvastatin on lowering levels of atherogenic lipids (from VOYAGER)

    Am. J. Cardiol.

    (2010)
  • G.I. Papakostas et al.

    Efficacy of bupropion and the selective serotonin reuptake inhibitors in the treatment of anxiety symptoms in major depressive disorder: a meta-analysis of individual patient data from 10 double-blind, randomized clinical trials

    J. Psychiatr. Res.

    (2008)
  • F.A. Raja et al.

    Platinum versus platinum-combination chemotherapy in platinum-sensitive recurrent ovarian cancer: a meta-analysis using individual patient data

    Ann. Oncol.

    (2013)
  • R.N. Rodseth et al.

    The prognostic value of pre-operative and post-operative B-type natriuretic peptides in patients undergoing noncardiac surgery: B-type natriuretic peptide and N-terminal fragment of pro-B-type natriuretic peptide: a systematic review and individual patient data meta-analysis

    J. Am. Coll. Cardiol.

    (2014)
  • U. Ronellenfitsch et al.

    Preoperative chemo(radio)therapy versus primary surgery for gastroesophageal adenocarcinoma: systematic review with meta-analysis combining individual patient and aggregate data

    Eur. J. Cancer

    (2013)
  • A. Rossi et al.

    Six versus fewer planned cycles of first-line platinum-based chemotherapy for non-small-cell lung cancer: a systematic review and meta-analysis of individual patient data

    Lancet Oncol.

    (2014)
  • L. Stewart et al.

    To IPD or not to IPD?

    Eval. Health Prof.

    (2002)
  • L.A. Stewart et al.

    Practical methodology of meta-analyses (overviews) using updated individual patient data

    Stat. Med.

    (1995)
  • M.C. Simmonds et al.

    Meta-analysis of individual patient data from randomized trials; a review of methods used in practice

    Cinical Trials

    (2005)
  • J.P.T. Higgins et al.

    Meta-analysis of continuous outcome data from individual patients

    Stat. Med.

    (2001)
  • R.M. Turner et al.

    A multilevel model framework for meta-analysis of clinical trials with binary outcomes

    Stat. Med.

    (2000)
  • A. Whitehead et al.

    Meta-analysis of ordinal outcomes using individual patient data

    Stat. Med.

    (2001)
  • R.D. Riley et al.

    Meta-analysis of individual participant data: rationale, conduct, and reporting

    BMJ

    (2010)
  • D. Moher et al.

    Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement

    BMJ

    (2009)
  • L.A. Stewart et al.

    Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement

    JAMA

    (2015)
  • Antithrombotic Trialists Collaboration et al.

    Aspirin in the primary and secondary prevention of vascular disease: collaborative meta-analysis of individual participant data from randomised trials

    Lancet

    (2009)
  • A. Auperin et al.

    Meta-analysis of concomitant versus sequential radiochemotherapy in locally advanced non-small-cell lung cancer

    J. Clin. Oncol.

    (2010)
  • T.S. Baman et al.

    Safety of pacemaker reuse: a meta-analysis with implications for underserved nations

    Circ.

    (2011)
  • K.J.L. Bell et al.

    Monitoring initial response to angiotensin-converting enzyme inhibitor-based regimens: an individual patient data meta-analysis from randomized, placebo-controlled trials

    Hypertension

    (2010)
  • K. Bennett et al.

    Cognitive behavioral therapy age effects in child and adolescent anxiety: an individual patient data meta-analysis

    Depress. Anxiety

    (2013)
  • D.A. Berry et al.

    High-dose chemotherapy with autologous stem-cell support as adjuvant therapy in breast cancer: overview of 15 randomized trials

    J. Clin. Oncol.

    (2011)
  • A. Beyerlein et al.

    Infant formula supplementation with long-chain polyunsaturated fatty acids has no effect on Bayley developmental scores at 18 months of age—IPD meta-analysis of 4 large clinical trials

    J. Pediatr. Gastroenterol. Nutr.

    (2010)
  • T. Bongartz et al.

    Etanercept therapy in rheumatoid arthritis and the risk of malignancies: a systematic review and individual patient data meta-analysis of randomised controlled trials

    Ann. Rheum. Dis.

    (2009)
  • J. Bourhis et al.

    Effect of amifostine on survival among patients treated with radiotherapy: a meta-analysis of individual patient data

    J. Clin. Oncol.

    (2011)
  • M.F. Brin et al.

    Safety and tolerability of onabotulinumtoxinA in the treatment of facial lines: a meta-analysis of individual patient data from global clinical registration studies in 1678 participants

    J. Am. Acad. Dermatol.

    (2009)
  • F. Burzotta et al.

    Clinical impact of thrombectomy in acute ST-elevation myocardial infarction: an individual patient-data pooled analysis of 11 trials

    Eur. Heart J.

    (2009)
  • Systematic review of purine analog treatment for chronic lymphocytic leukemia: lessons for future trials

    Haematologica

    (2012)
  • L. Chambrone et al.

    Effects of smoking cessation on the outcomes of non-surgical periodontal therapy: a systematic review and individual patient data meta-analysis

    J. Clin. Periodontol.

    (2013)
  • Reducing uncertainties about the effects of chemoradiotherapy for cervical cancer: a systematic review and meta-analysis of individual patient data from 18 randomized trials

    J. Clin. Oncol.

    (2008)
  • Beneficial and harmful effects of anthracyclines in the treatment of childhood acute lymphoblastic leukaemia: a systematic review and meta-analysis

    Br. J. Haematol.

    (2009)
  • R. Clarke et al.

    Effects of lowering homocysteine levels with B vitamins on cardiovascular disease, cancer, and cause-specific mortality: meta-analysis of 8 randomized trials involving 37,485 individuals

    Arch. Intern. Med.

    (2010)
  • J.P. Daniels et al.

    Individual patient data meta-analysis of randomized evidence to assess the effectiveness of laparoscopic uterosacral nerve ablation in chronic pelvic pain

    Hum. Reprod. Update

    (2010)
  • T. De Backer et al.

    Naftidrofuryl for intermittent claudication: meta-analysis based on individual patient data

    BMJ

    (2009)
  • Cited by (82)

    • Modernizing evidence synthesis for evidence-based medicine

      2023, Clinical Decision Support and beyond: Progress and Opportunities in Knowledge-Enhanced Health and Healthcare
    View all citing articles on Scopus
    View full text