Skip to main content

Main menu

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN

User menu

Search

  • Advanced search
CMAJ
  • CMAJ JOURNALS
    • CMAJ Open
    • CJS
    • JAMC
    • JPN
CMAJ

Advanced Search

  • Home
  • COVID-19
    • Articles & podcasts
    • Blog posts
    • Collection
    • News
  • Content
    • Current issue
    • Past issues
    • Early releases
    • Collections
    • Sections
    • Blog
    • Infographics & illustrations
    • Podcasts
    • Classified ads
  • Authors
    • Overview for authors
    • Submission guidelines
    • Submit a manuscript
    • Forms
    • Editorial process
    • Editorial policies
    • Peer review process
    • Publication fees
    • Reprint requests
  • CMA Members
    • Overview for members
    • Earn CPD Credits
    • Print copies of CMAJ
    • Career Ad Discount
  • Subscribers
    • General information
    • View prices
  • Alerts
    • Email alerts
    • RSS
  • JAMC
    • À propos
    • Numéro en cours
    • Archives
    • Sections
    • Abonnement
    • Alertes
  • Visit CMAJ on Facebook
  • Follow CMAJ on Twitter
  • Follow CMAJ on Pinterest
  • Follow CMAJ on Youtube
  • Follow CMAJ on Instagram
Commentary

Appraising descriptive and analytic findings of large cohort studies

Barnaby C. Reeves
CMAJ July 29, 2019 191 (30) E828-E829; DOI: https://doi.org/10.1503/cmaj.190882
Barnaby C. Reeves
Bristol Medical School, University of Bristol, Bristol, UK
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Responses
  • Metrics
  • PDF
Loading

See related article at www.cmaj.ca/lookup/doi/10.1503/cmaj.190221

KEY POINTS
  • Descriptions of variation in health care practice and outcome provide extremely valuable information.

  • Estimation of associations between predictors and outcome are at risk of several biases, which must be carefully appraised.

  • The credibility of attributable fractions depends on the validity of the associations on which they are based and the assumption that these associations are entirely causal.

Large representative cohorts can provide compelling descriptions of variation in both health care practice and health outcomes, and their value is often underestimated. Such information is vital to health policy-makers for planning services, but also to clinicians and researchers, as substantial variation often represents an opportunity to target improvements in practice and influence future outcomes. In linked research, Devereaux and colleagues report rates of perioperative mortality and serious complications after surgery1 among people undergoing surgery and requiring at least 1 night in hospital, and associations between complications and 30-day mortality adjusted for baseline comorbidities. The scale of the Vascular Events in Noncardiac Surgery Patients Cohort Evaluation (VISION) study (> 40 000 participants recruited over 6 years) and the VISION collaboration (which spans 6 continents) is staggering. However, bias in analyses of observational studies is almost unavoidable and estimates of relationships between complications and mortality should be interpreted carefully before their findings are applied to clinical and policy decisions.

Confidence in the results of any study starts with reviewing the authors’ prespecified objectives, ideally through registration details and a protocol. The VISION study was established with multiple aims, and the registration details (ClinicalTrials.gov, no. NCT00512109) refer to prognostic analyses of troponin assays,2–4 making the origins of this report difficult to trace. A statistical analysis plan supplied by the authors in an appendix sets out 6 objectives: to determine the incidence of postoperative complications within 30 days after surgery; the time-dependent relationship between these complications and 30-day mortality; the attributable fraction of death at 30 days of each postoperative complication independently associated with mortality; the timing of death during the first 30 days after noncardiac surgery; the proportion of patients who died after noncardiac surgery in hospital and separately after hospital discharge during a 30-day follow-up period; and the risk of death at 30 days after noncardiac surgery by surgical category.

Four objectives are descriptive, but the second and third objectives require analytic quantification of associations between perioperative complications and 30-day mortality.

For descriptive objectives, appraisal should focus on recruitment to the cohort and the quality and completeness of the data set. The VISION study accounts for all patients determined as being eligible, showing that mortality was available for 99.9% (Supplemental Fig. 1 in Appendix 1 of the linked research,1 available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190221/-/DC1). Patients who did not consent made up only 25% of the total, with mainly logistical reasons explaining why others were not enrolled. Here, potential concern about the validity of the descriptive results might arise owing to the varied methods of consenting patients across sites. For example, patients at differential risk of perioperative mortality or complications might have been excluded between admission and consent, and not be accounted for.

Findings for analytic objectives need particular scrutiny because several sources of bias can undermine their validity.5 Selection into the VISION cohort could also have biased quantification of associations between complications and mortality if patients were excluded for reasons related to the predictor of interest (the occurrence of a complication) and outcome (perioperative mortality).5 This might have arisen, albeit for a few patients, if a patient had a complication and died before retrospective consent could be sought.

It is important to consider whether residual confounding can explain the associations observed or bias their magnitude substantially.5 In the linked study, hazard ratios (HRs) for associations between major bleeding and acute kidney injury with 30-day mortality, estimated with and without adjustment for preoperative hemoglobin and estimated glomerular filtration rate (eGFR), suggest some residual confounding in the primary analyses (Supplemental Table 9 in Appendix 1 of the linked research1). The HR for major bleeding (but not acute kidney injury) is reduced by adjusting for preoperative hemoglobin and the HR for acute kidney injury (but not major bleeding) is reduced by adjusting for preoperative eGFR. These shifts in the HRs are small compared with the overall magnitude of the HRs, so there is no reason to doubt that the associations are real; however, their magnitude may be more uncertain than indicated by their confidence intervals.

Differential misclassification of predictors and outcomes5 can happen when outcomes are classified with knowledge of the predictors, or vice versa. In the linked study, these risks seem unlikely, as definitions of complications were set out and applied to data that had already been collected, and complications that involved clinical judgment were adjudicated.

Missing data, for predictors, confounders or outcomes, can also introduce bias5 but, in the linked study, data were available for 99.0% of the entire cohort. This high percentage may be accounted for by the limited number of predictors included in the model, which in turn introduces the risk of residual confounding. Supplementary analyses adjusting for the additional risk factors preoperative hemoglobin and eGFR show that these preoperative characteristics were missing for 3.4% and 6.8% of patients.

Finally, cherry-picking results from other results generated — such as from multiple outcome measurements, multiple analyses of the predictor-outcome relationship or different subgroups5 — is increasingly being recognized as a pervasive source of bias.6,7 It is notable that the authors of the linked study had a statistical analysis plan (many analyses of observational cohorts do not), but it is dated some years after recruitment ended (April 2017 v. November 2013). There was no selection of results for multiple outcomes or different subgroups, but selection of the reported results for relationships between complications and mortality cannot be excluded.

Estimates of the relationships between complications and mortality underpin the calculation of attributable fractions — if the former are uncertain owing to potential bias, so are the latter. Attributable fractions should also be interpreted cautiously: basing clinical practice or policy decisions on these statistics requires the user to question closely the plausibility of the assumption that relationships between complications and 30-day mortality are entirely causal. I salute the achievement of the VISION study investigators but advise caution in applying the relationships of complications with 30-day mortality and the attributable fractions.

Footnotes

  • Competing interests: Barnaby Reeves reports being an applicant on a grant application to fund the development of the ROBINS-I tool and a member of the core team that developed the tool.

  • This article was solicited and has not been peer reviewed.

References

  1. ↵
    The Vascular Events in Noncardiac Surgery Patients Cohort Evaluation (VISION) Study Investigators. Association between complications and death within 30 days after noncardiac surgery. CMAJ 2019;191:E830–7.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study InvestigatorsDevereaux PJ, Chan MT, Alonso-Coello P, et al. Association between postoperative troponin levels and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2012;307:2295–304.
    OpenUrlCrossRefPubMed
  3. Writing Committee for the VISION Study InvestigatorsDevereaux PJ, Biccard BM, Sigamani A, et al. Association of postoperative high-sensitivity troponin levels with myocardial injury and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2017;317:1642–51.
    OpenUrl
  4. ↵
    Vascular Events In Noncardiac Surgery Patients Cohort Evaluation Study (VISION). ClinicalTrials.gov trial no. NCT00512109; 2018. Available: https://clinicaltrials.gov/ct2/show/NCT00512109 (accessed 2019 July 1).
  5. ↵
    The ROBINS-E tool (Risk Of Bias In Non-randomized Studies — of Exposures). Bristol (UK): Centre for Research Synthesis and Decision Analysis, University of Bristol; 2017. Available: www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/robins-e/ (accessed 2019 July 1).
  6. ↵
    1. Sterne JAC,
    2. Hernán MA,
    3. Reeves BC,
    4. et al
    . ROBINS-I: a tool for assessing risk of bias in non-randomized studies of Interventions. BMJ 2016;355:i4919.
    OpenUrlFREE Full Text
  7. ↵
    1. Sterne JAC,
    2. Savovic J,
    3. Page M,
    4. et al
    . RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. In press.
PreviousNext
Back to top

In this issue

Canadian Medical Association Journal: 191 (30)
CMAJ
Vol. 191, Issue 30
29 Jul 2019
  • Table of Contents
  • Index by author

Article tools

Respond to this article
Print
Download PDF
Article Alerts
To sign up for email alerts or to access your current email alerts, enter your email address below:
Email Article

Thank you for your interest in spreading the word on CMAJ.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Appraising descriptive and analytic findings of large cohort studies
(Your Name) has sent you a message from CMAJ
(Your Name) thought you would like to see the CMAJ web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Appraising descriptive and analytic findings of large cohort studies
Barnaby C. Reeves
CMAJ Jul 2019, 191 (30) E828-E829; DOI: 10.1503/cmaj.190882

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
‍ Request Permissions
Share
Appraising descriptive and analytic findings of large cohort studies
Barnaby C. Reeves
CMAJ Jul 2019, 191 (30) E828-E829; DOI: 10.1503/cmaj.190882
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Footnotes
    • References
  • Responses
  • Metrics
  • PDF

Related Articles

  • Association between complications and death within 30 days after noncardiac surgery
  • PubMed
  • Google Scholar

Cited By...

  • Most deaths related to non-cardiac surgery occur outside operating room, finds study
  • Google Scholar

More in this TOC Section

  • Not neutral: reimagining antiracism as a professional competence
  • Transforming race-based health research in Canada
  • Time to dismantle systemic anti-Black racism in medicine in Canada
Show more Commentary

Similar Articles

Collections

  • Topics
    • Research methods & statistics

Content

  • Current issue
  • Past issues
  • Collections
  • Sections
  • Blog
  • Podcasts
  • Alerts
  • RSS
  • Early releases

Information for

  • Advertisers
  • Authors
  • Reviewers
  • CMA Members
  • Media
  • Reprint requests
  • Subscribers

About

  • General Information
  • Journal staff
  • Editorial Board
  • Governance Council
  • Journal Oversight
  • Careers
  • Contact
  • Copyright and Permissions

Copyright 2021, Joule Inc. or its licensors. All rights reserved. ISSN 1488-2329 (e) 0820-3946 (p)

All editorial matter in CMAJ represents the opinions of the authors and not necessarily those of the Canadian Medical Association or its subsidiaries.

Powered by HighWire