Appraising descriptive and analytic findings of large cohort studies

• Descriptions of variation in health care practice and outcome provide extremely valuable information.

• Estimation of associations between predictors and outcome are at risk of several biases, which must be carefully appraised.

• The credibility of attributable fractions depends on the validity of the associations on which they are based and the assumption that these associations are entirely causal.

Large representative cohorts can provide compelling descriptions of variation in both health care practice and health outcomes, and their value is often underestimated. Such information is vital to health policy-makers for planning services, but also to clinicians and researchers, as substantial variation often represents an opportunity to target improvements in practice and influence future outcomes. In linked research, Devereaux and colleagues report rates of perioperative mortality and serious complications after surgery1 among people undergoing surgery and requiring at least 1 night in hospital, and associations between complications and 30-day mortality adjusted for baseline comorbidities. The scale of the Vascular Events in Noncardiac Surgery Patients Cohort Evaluation (VISION) study (> 40 000 participants recruited over 6 years) and the VISION collaboration (which spans 6 continents) is staggering. However, bias in analyses of observational studies is almost unavoidable and estimates of relationships between complications and mortality should be interpreted carefully before their findings are applied to clinical and policy decisions.

Confidence in the results of any study starts with reviewing the authors’ prespecified objectives, ideally through registration details and a protocol. The VISION study was established with multiple aims, and the registration details (ClinicalTrials.gov, no. NCT00512109) refer to prognostic analyses of troponin assays,2^–4 making the origins of this report difficult to trace. A statistical analysis plan supplied by the authors in an appendix sets out 6 objectives: to determine the incidence of postoperative complications within 30 days after surgery; the time-dependent relationship between these complications and 30-day mortality; the attributable fraction of death at 30 days of each postoperative complication independently associated with mortality; the timing of death during the first 30 days after noncardiac surgery; the proportion of patients who died after noncardiac surgery in hospital and separately after hospital discharge during a 30-day follow-up period; and the risk of death at 30 days after noncardiac surgery by surgical category.

Four objectives are descriptive, but the second and third objectives require analytic quantification of associations between perioperative complications and 30-day mortality.

For descriptive objectives, appraisal should focus on recruitment to the cohort and the quality and completeness of the data set. The VISION study accounts for all patients determined as being eligible, showing that mortality was available for 99.9% (Supplemental Fig. 1 in Appendix 1 of the linked research,1 available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.190221/-/DC1). Patients who did not consent made up only 25% of the total, with mainly logistical reasons explaining why others were not enrolled. Here, potential concern about the validity of the descriptive results might arise owing to the varied methods of consenting patients across sites. For example, patients at differential risk of perioperative mortality or complications might have been excluded between admission and consent, and not be accounted for.

Findings for analytic objectives need particular scrutiny because several sources of bias can undermine their validity.5 Selection into the VISION cohort could also have biased quantification of associations between complications and mortality if patients were excluded for reasons related to the predictor of interest (the occurrence of a complication) and outcome (perioperative mortality).5 This might have arisen, albeit for a few patients, if a patient had a complication and died before retrospective consent could be sought.

It is important to consider whether residual confounding can explain the associations observed or bias their magnitude substantially.5 In the linked study, hazard ratios (HRs) for associations between major bleeding and acute kidney injury with 30-day mortality, estimated with and without adjustment for preoperative hemoglobin and estimated glomerular filtration rate (eGFR), suggest some residual confounding in the primary analyses (Supplemental Table 9 in Appendix 1 of the linked research1). The HR for major bleeding (but not acute kidney injury) is reduced by adjusting for preoperative hemoglobin and the HR for acute kidney injury (but not major bleeding) is reduced by adjusting for preoperative eGFR. These shifts in the HRs are small compared with the overall magnitude of the HRs, so there is no reason to doubt that the associations are real; however, their magnitude may be more uncertain than indicated by their confidence intervals.

Differential misclassification of predictors and outcomes5 can happen when outcomes are classified with knowledge of the predictors, or vice versa. In the linked study, these risks seem unlikely, as definitions of complications were set out and applied to data that had already been collected, and complications that involved clinical judgment were adjudicated.

Missing data, for predictors, confounders or outcomes, can also introduce bias5 but, in the linked study, data were available for 99.0% of the entire cohort. This high percentage may be accounted for by the limited number of predictors included in the model, which in turn introduces the risk of residual confounding. Supplementary analyses adjusting for the additional risk factors preoperative hemoglobin and eGFR show that these preoperative characteristics were missing for 3.4% and 6.8% of patients.

Finally, cherry-picking results from other results generated — such as from multiple outcome measurements, multiple analyses of the predictor-outcome relationship or different subgroups5 — is increasingly being recognized as a pervasive source of bias.6^,7 It is notable that the authors of the linked study had a statistical analysis plan (many analyses of observational cohorts do not), but it is dated some years after recruitment ended (April 2017 v. November 2013). There was no selection of results for multiple outcomes or different subgroups, but selection of the reported results for relationships between complications and mortality cannot be excluded.

Estimates of the relationships between complications and mortality underpin the calculation of attributable fractions — if the former are uncertain owing to potential bias, so are the latter. Attributable fractions should also be interpreted cautiously: basing clinical practice or policy decisions on these statistics requires the user to question closely the plausibility of the assumption that relationships between complications and 30-day mortality are entirely causal. I salute the achievement of the VISION study investigators but advise caution in applying the relationships of complications with 30-day mortality and the attributable fractions.