Abstract
Background: Predicting long-term survival after admission to hospital is helpful for clinical, administrative and research purposes. The Hospital-patient One-year Mortality Risk (HOMR) model was derived and internally validated to predict the risk of death within 1 year after admission. We conducted an external validation of the model in a large multicentre study.
Methods: We used administrative data for all nonpsychiatric admissions of adult patients to hospitals in the provinces of Ontario (2003–2010) and Alberta (2011–2012), and to the Brigham and Women’s Hospital in Boston (2010–2012) to calculate each patient’s HOMR score at admission. The HOMR score is based on a set of parameters that captures patient demographics, health burden and severity of acute illness. We determined patient status (alive or dead) 1 year after admission using population-based registries.
Results: The 3 validation cohorts (n = 2 862 996 in Ontario, 210 595 in Alberta and 66 683 in Boston) were distinct from each other and from the derivation cohort. The overall risk of death within 1 year after admission was 8.7% (95% confidence interval [CI] 8.7% to 8.8%). The HOMR score was strongly and significantly associated with risk of death in all populations and was highly discriminative, with a C statistic ranging from 0.89 (95% CI 0.87 to 0.91) to 0.92 (95% CI 0.91 to 0.92). Observed and expected outcome risks were similar (median absolute difference in percent dying in 1 yr 0.3%, interquartile range 0.05%–2.5%).
Interpretation: The HOMR score, calculated using routinely collected administrative data, accurately predicted the risk of death among adult patients within 1 year after admission to hospital for nonpsychiatric indications. Similar performance was seen when the score was used in geographically and temporally diverse populations. The HOMR model can be used for risk adjustment in analyses of health administrative data to predict long-term survival among hospital patients.
The life expectancy of individual patients can be important for both medical decision-making and research. Patients with a short life expectancy may choose to defer preventive treatments, screening interventions or interventional procedures for conditions that are currently asymptomatic. An accurate assessment of risk of death, particularly if that risk is high, could motivate and inform discussions between patients and physicians regarding goals of care. In addition, accurate prognostications are essential for adjusting statistical models that have death as an outcome (or as a competing risk for other outcomes) in both research and administration.
We recently derived and internally validated a model that predicts the risk of death from any cause at 1 year after admission to hospital.1 The Hospital-patient One-year Mortality Risk (HOMR) model consists of covariates whose values are determined at admission using routinely collected health administrative data (Figure 1). These covariates include patient demographics (age, sex and living status); health burden (measured using the Charlson Comorbidity Index score, home oxygen status and the number of visits to emergency departments and admissions to hospital by ambulance in the previous year); and acuity of illness (admission urgency and hospital service, direct admission to an intensive care unit and whether the admission was an urgent readmission to hospital). The latter category was also gauged using the Diagnostic Risk Score, which quantifies risk of death for particular diagnoses beyond that explained by the other covariates (Appendix 1, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1).
Covariates used to calculate a patient’s Hospital-patient One-year Mortality Risk (HOMR) score at the time of admission to hospital. The Diagnostic Risk Score (Appendix 1) quantifies risk of death for diagnostic groups beyond that explained by the other covariates. Points for the interacting covariates of age and Charlson Comorbidity Index score include the risk of patient age, comorbidity score and their interaction. In contrast, points for living status and admission urgency include the risk of these covariates and their interaction with admissions by ambulance in the previous year; points for the latter covariate are considered separately. See Table 3 for the expected risk of death within 1 year after hospital admission for each HOMR score. ED = emergency department, ICU = intensive care unit. *In the year before admission. †See Appendix 2 for definitions of each service. (Appendices are available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1)
Discrete values for each covariate are given specific points, which are summed to create the HOMR score (Figure 1). In an internal validation population, the HOMR score accurately predicted the risk of death from any cause within 1 year after admission, with a C statistic of 0.92 and excellent calibration among adult residents of Ontario admitted to hospital for nonpsychiatric indications in 2011.1
Although these statistics are impressive, external validation is required to determine the true usefulness of any statistical model. External validation is necessary to prove that the model’s performance is not idiosyncratic to the patients, physicians, institutions or data systems used to derive and internally test it.2,3 A prognostic model should remain accurate when retested with different patients (reproducibility), during different periods (historical transportability) and in different locations (geographic transportability).4 We conducted an external validation of the HOMR model in a multicentre study that included Canadian and American hospitals.
Methods
Study settings and data sources
We used health administrative data from 2 Canadian provinces (Ontario and Alberta) and the Brigham and Women’s Hospital in Boston. All data were linked and analyzed anonymously. The study was approved by the research ethics board at each study site (Sunnybrook and Women’s College Health Sciences Centre Research Ethics Board, the University of Alberta Health Research Ethics Board and the Partners Human Research Committee).
All health care activities in Ontario and Alberta are publicly funded and are captured in population-based administrative datasets. These datasets include the Discharge Abstract Database, which captures all hospital admissions and, in the years we sampled, used diagnostic codes from the International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10); the Registered Persons Database, which captures each person’s date of death (including deaths that occur out of province); and the National Ambulatory Care Reporting System, which captures all visits to emergency departments. For the Ontario cohort, we also used the Assistive Devices Program, which captures all patients who receive home oxygen; the Continuing Care Reporting System, which captures all residents of registered nursing homes and hospitals (i.e., inpatients who are unable to live at home or in a nursing home, usually because they have advanced nursing requirements); and the Home Care Database, which captures all publicly funded in-home assistance. Home oxygen status could not be determined for the patients in Alberta. We linked all of the databases deterministically using encrypted health insurance numbers.
The Brigham and Women’s Hospital is a 779-bed not-for-profit teaching affiliate of Harvard Medical School in Boston. It has about 44 000 inpatient admissions, 54 000 emergency department visits and more than 950 000 ambulatory visits each year. Patients come from a variety of referral sources, including practices based in the hospital, community-based private practices and community health centres affiliated with Partners Community Healthcare, Inc., and nonaffiliated practices within and outside of the Boston area. Because the Brigham and Women’s Hospital acts as both a high-level academic tertiary hospital and a primary hospital for the surrounding (mainly low-income) community, we considered it to be similar to other academic American hospitals. All health activities for these patients are captured in health administrative datasets, including the Research Patient Data Registry, which records diagnoses using codes from the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), Diagnosis-Related Group codes and dates of death from the Social Security Death Registry; the Longitudinal Medical Record, where clinical data can be extracted from medication or problem lists; the Enterprise Master Patient Index, which coordinates patient identification across the multiple administrative datasets and maintains a database of selected demographic information for patients; and the Brigham Integrated Computing System, which captures all inpatient data. Home nursing and chronic hospital status were unavailable for patients at the Brigham and Women’s Hospital.
Study cohorts
In all 3 cohorts, we included adults (age ≥ 18 yr in Ontario and Alberta; age ≥ 20 in Boston) with a valid identifier who had at least 1 admission to a hospital for an acute, nonpsychiatric indication during the study period. For people who had multiple admissions, we randomly selected 1 admission per person during the study period. The Ontario cohort included eligible admissions to all acute care hospitals in the province between Jan. 1, 2003 (corresponding to the introduction of ICD-10) and Dec. 31, 2010 (final date for which patients not included in the HOMR derivation group were available). The Alberta cohort included eligible admissions to all hospitals in the province between Apr. 1, 2011, and Mar. 31, 2012. The Boston cohort included eligible admissions to the Brigham and Women’s Hospital between Jan. 1, 2010, and Dec. 31, 2012. In all 3 cohorts, we excluded patients who underwent same-day surgery and those who had been transferred from another hospital.
Outcome measure
The primary outcome measure was death from any cause within 1 year after admission to hospital. All deaths, including those that happened during the index admission, were captured. For the Ontario and Alberta cohorts, outcome status was determined by linking to the Registered Persons Database. For the Boston cohort, outcome status was determined by linking to the Research Patient Data Registry.
Other covariates
Definitions and database sources of the covariates in the HOMR model are provided in Appendix 2, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.150209/-/DC1. The HOMR score is calculated using data values at patient admission. The Diagnostic Risk Score (Appendix 1) was previously derived using ICD-10 codes.1 Because ICD-9-CM codes were used for diagnoses in the Boston cohort, we used crosswalk tables (provided by the Canadian Institute for Health Information) to identify the ICD-9-CM code(s) that most accurately corresponded to the ICD-10 codes included in the Diagnostic Risk Score.
Statistical analysis
We calculated each patient’s HOMR score. Within each cohort, we calculated the proportion of patients who died within 1 year after admission for each distinct HOMR score value. To measure discrimination of the HOMR scores, we used a binomial logistic regression model to calculate the C statistic, where death within 1 year after hospital admission was the outcome and the HOMR score (expressed as a linear term) was the sole independent variable. To measure calibration, we calculated 95% confidence intervals (CIs) around the observed risk of death associated with each HOMR score and determined whether this included the expected risk of death calculated from the original study cohort. For each HOMR score, we also calculated the absolute difference between the observed and the expected proportion of patients who died within 1 year after admission and the relative difference (calculated as the absolute difference divided by the expected proportion).
We produced an expected risk of death within 1 year for each HOMR score using data from all 3 validation populations and a random effects model that accounted for patients clustered within each cohort. This model had a single independent covariate — the HOMR score expressed as a linear term — and we used it to generate 95% prediction intervals (not confidence intervals) for the expected outcome risk for each HOMR score.
We performed all statistical analyses using SAS version 9.3 (SAS Institute, Inc.).
Results
Characteristics of the patients in the 3 validation cohorts and in the derivation cohort1 are shown in Table 1. We observed several notable differences between the cohorts. Patients in the Alberta cohort were younger than those in the other cohorts. Patients in the derivation cohort were much more likely to be categorized as living at home with home care services. Patients in the Boston cohort had notably higher Charlson Comorbidity Index scores (with only 36.8% having a score of 0); they also were much more likely to have been admitted to hospital by ambulance in the previous year but were less likely to be admitted to the intensive care unit. General surgery was more prominent within admission services in the Alberta cohort, whereas hematology/oncology, cardiovascular surgery and neurosurgery were more prominent in the Boston cohort.
Characteristics of patients in the derivation cohort and in the 3 cohorts for external validation of the HOMR score
Despite their disparate characteristics, the cohorts had similar mean Diagnostic Risk Scores (Table 1) and HOMR scores. The median HOMR score was 22 (interquartile range [IQR] 9 to 32, range −12 to 73) in Ontario, 23 (IQR 12 to 50, range −7 to 69) in Alberta and 27 (IQR 20 to 34, range −8 to 59) in Boston. In the derivation cohort, the median HOMR score was 25 (IQR 12 to 36, range −12 to 76).
In the validation cohorts, the overall all-cause mortality rate within 1 year after admission was 8.7%. It varied slightly between the patient groups: 8.76% (95% CI 8.73% to 8.79%) in the Ontario cohort, 8.45% (95% CI 8.33% to 8.57%) in the Alberta cohort and 8.70% (95% CI 8.49% to 8.91%) in the Boston cohort. In the derivation cohort, the 1-year mortality was 11.73% (95% CI 11.65% to 11.81%).
The HOMR score was strongly associated with 1-year risk of death and was highly discriminative in each validation cohort. When entered as a linear term in a logistic regression model, the score was strongly associated with risk of death in all 3 validation cohorts, with an odds ratio for death per 1-point increase in HOMR score of 1.23 (95% CI 1.23 to 1.23) in the Ontario cohort, 1.24 (95% CI 1.23 to 1.24) in the Alberta cohort and 1.24 (95% CI 1.24 to 1.25) in the Boston cohort. The HOMR score was discriminative, with C statistics of 0.92 (95% CI 0.91 to 0.92) in Ontario, 0.92 (95% CI 0.90 to 0.93) in Alberta and 0.89 (95% CI 0.87 to 0.91) in Boston.
In all 3 validation cohorts, the 1-year risk of death for distinct HOMR scores was similar to the pattern found in the original study1 (Figure 2). For HOMR scores between 30 and 50, the observed risk of death tended to exceed the expected risk, but absolute differences were small. For scores above 50, the observed risk of death tended to be lower than the expected risk in the Boston cohort; there were fewer patients with such scores in this cohort than in the Ontario and Alberta cohorts.
Observed and expected risks of death within 1 year after admission to hospital in the 3 validation cohorts (Ontario, Alberta and Boston) and the derivation cohort,1 as calculated by the Hospital-patient One-year Mortality Risk (HOMR) model. Error bars indicate 95% confidence intervals. A summary of the calibration of each cohort to the expected risk of death is given in Table 2.
The HOMR score was also well calibrated with the 1-year risk of death. In the validation cohorts, the expected risk of death fell within the 95% CIs of the observed risk 31.0% of the time in Ontario, 57.1% of the time in Alberta and 53.8% of the time in the Boston cohort (Table 2); in the derivation cohort, this happened 66.7% of the time. The relative differences between the observed and expected percentage of patients dead at 1 year were notable, reaching a median value of 0.26 (i.e., 26%) in the Boston cohort. However, the absolute differences between the observed and expected percentages of patients dead at 1 year were small in all of the validation cohorts, with a median value of 0.3% in all 3 patient groups.
Calibration of the HOMR score in the derivation and validation cohorts
We used data from the validation cohorts to calculate the expected 1-year risk of death for each HOMR score (Table 3). The uncertainty around these predicted risks was small, with relative differences between expected values and the upper 95% prediction interval having a median value of 0.3% (IQR 0.2% to 0.7%).
Expected risk of death within 1 year after admission to hospital, by HOMR score*
Interpretation
This multicentre study included a large and diverse population of patients and showed that the HOMR score readily discriminated between patients who died and those who survived during the year after admission to hospital. The score was well calibrated and was, within this sample, both temporally and geographically transportable. Therefore, the HOMR score can be used with confidence to predict risk of death within 1 year after admission using health administrative data.
We believe that our external validation of the HOMR model has important implications for research and monitoring of health care systems. Researchers and analysts will be able to adjust for risk of death in analyses involving a broad range of hospital patients. Use of the HOMR model will also allow administrators and decision-makers to compare more accurately in-hospital and post-discharge mortality outcomes between institutions, regions or health care systems. Most of the information required to calculate the HOMR score can be determined from fields typically available in hospital discharge abstract databases (Appendix 2). Finally, we believe that the HOMR model can be used to study, at a population-based level, the expected survival of patients undergoing preventive treatments or screening interventions, to assess the appropriateness of their use.
The HOMR model joins several other points-based models that predict risk of death among hospital patients (Table 4), most of which have been discussed in detail by Yourman and colleagues.20 The HOMR model had a much larger derivation population than the other indexes and is applicable to a broader range of patients, since other indexes were limited to particular age groups, admission services or diagnoses. The discrimination of the HOMR model significantly exceeds that of other indexes, although some of this difference may be due to its inclusion of a large population with widely varying risks of death. Finally, in addition to the HOMR model, only 3 other indexes have been externally validated. This finding supports a recently published systematic review, which found that validation of risk-prediction models is uncommon.21
Summary of point-based models for predicting risk of death among hospital patients
Limitations
Several issues about this validation and the HOMR score should be highlighted. First, the validation cohorts were selected from all hospital patients in 3 regions that were chosen on the basis of availability rather than randomly. As such, the generalizability of our results to other hospital populations is uncertain. We did, however, include 3 large and disparate patient populations from distinct periods, health care systems, countries and coding systems. The fact that the HOMR model was both discriminative and well calibrated in these disparate populations is reassuring to future users who plan to apply the model to other patient groups.
Second, further testing of the HOMR model is required to prove its accuracy in other, more geographically and temporally distinct populations.
Third, not all of the data components required to calculate the HOMR score were available in the validation cohorts. The fact that the HOMR model remained discriminative and well calibrated despite these missing data indicates its robustness.
Finally, the HOMR model cannot necessarily be used to predict outcomes with the use of primary data. For most of the HOMR covariates (e.g., patient age, admission service), the difference between administrative data and primary data would likely be small to nonexistent. However, differences could be considerable for the Charlson Comorbidity Index score, because administrative data tend to underestimate comorbid conditions.22–24 As such, the number of points assigned to the Charlson score might be inappropriately high if the HOMR score were calculated using primary data. On the other hand, the predictive capability of the HOMR score may be improved with the use of primary data, because this would improve the accuracy of comorbidity capture. In addition, use of primary data would permit the proper quantification of the severity of comorbid conditions and the functional status of patients, both of which have important prognostic implications.25–27 Thus, evaluation of the performance of the HOMR model in cohorts with primary data capture would be an important direction for future research.
Conclusion
We have shown in this large multicentre study that the HOMR score, calculated using routinely collected administrative data, accurately predicted the risk of death within 1 year after admission to hospital in a diverse population of patients admitted for nonpsychiatric indications. The HOMR model can be used for risk adjustment in analyses of health administrative data to predict long-term survival among hospital patients.
Acknowledgements
Carl van Walraven is supported by a Clinician Scientist Chair Award from the Department of Medicine, University of Ottawa. Finlay McAlister is supported by a salary award from Alberta Innovates – Health Solutions and holds the Chair in Cardiovascular Outcomes Research at the Mazankowski Alberta Heart Institute, University of Alberta.
Footnotes
Competing interests: None declared.
This article has been peer reviewed.
Contributors: All of the authors contributed to the study design. Carl van Walraven, Jacques Donzé, Steven Hawken and Jeffrey Bakal contributed to the data analysis. All of the authors contributed to the writing of the manuscript, approved the final version to be published and agreed to act as guarantors of the work.
- Accepted April 13, 2015.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- Association between neuraxial anaesthesia or general anaesthesia for lower limb revascularisation surgery in adults and clinical outcomes: population based comparative effectiveness study
- Estimating Real World Performance of a Predictive Model: A Case-Study in Predicting End-of-Life
- The risk of death within 5 years of first hospital admission in older adults
- mHOMR: a feasibility study of an automated system for identifying inpatients having an elevated risk of 1-year mortality
- External validation of the Hospital Frailty Risk Score and comparison with the Hospital-patient One-year Mortality Risk Score to predict outcomes in elderly hospitalised patients: a retrospective cohort study
- The association of female sex with application of evidence-based practice recommendations for perioperative care in hip fracture surgery
- The "surprise question" for predicting death in seriously ill patients: a systematic review and meta-analysis