Original Article
Time-dependent bias was common in survival analyses published in leading clinical journals

https://doi.org/10.1016/j.jclinepi.2003.12.008Get rights and content

Abstract

Objective

In survival analysis, “baseline immeasurable” time-dependent factors cannot be recorded at baseline, and change value after patient observation starts. Time-dependent bias can occur if such variables are not analyzed appropriately. This study sought to determine the prevalence of such time-dependent bias in highly-cited medical journals.

Study design and setting

We searched Medline databases to identify all observational studies that used a survival analysis in American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal, Chest, Circulation, Journal of the American Medical Association, Lancet, and New England Journal of Medicine between 1998 and 2002. Studies with “baseline immeasurable” time-dependent factors were susceptible to time-dependent bias if a time-dependent covariate analysis was not used.

Results

Of 682 eligible studies, 127 (18.6%, 95% CI 15.8–21.8%) contained a “baseline immeasurable” time-dependent factor and 52 (7.6% [5.8–9.9%] of all survival analyses/40.9% [32.3–50.0%] of studies with a time-dependent factor) were susceptible to time-dependent bias. In 35 studies (5.1% [3.7–7.1%]/27.6% [20.5–35.9%]), the bias affected a variable highlighted in the study abstract and correction of the bias could have qualitatively changed the study's conclusion in over half of studies.

Conclusion

In medical journals, time-dependent bias is concerningly common and frequently affects key factors and the study's conclusion.

Introduction

Bias is the systematic deviation of study results or inferences from the truth. Because bias can lead to erroneous conclusions, its minimization is a critical goal of all good research [1]. Bias can arise during all study phases, including its design, conduct, or analysis [2]. Several studies have found extensive amounts of inappropriate statistical methodology in published papers [3], [4], [5]. Analytical bias is important because it often can be avoided with proper statistical methodology.

In a time-to-event or survival analysis [6], problems can occur when variables in the model change value after the start of patient observation. Such variables are called “time-dependent,” because their value can change over time. There are two general categories of time-dependent variables. “Baseline measurable” time-dependent variables, like systolic blood pressure and body mass index, can change over time but are measurable at baseline. These “baseline measurable” time-dependent variables are frequently analyzed as fixed, or unchanging, variables in survival analyses. In contrast, “baseline immeasurable” time-dependent variables cannot be measured at baseline and indicate what happened to patients during observation.

Biased estimates can occur if “baseline immeasurable” time-dependent variables are analyzed as fixed variables. Consider a hypothetic study determining prognosticators for patients who have a perforation of the sigmoid and undergo emergency hemicolectomy with colostomy. Patients who die in the first several months after the operation will never undergo closure of their colostomy. If this “baseline immeasurable” time-dependent factor (“Was colostomy closed?”) is analyzed in a survival analysis as a fixed variable, one would associate no colostomy closure with a worse survival. This association is erroneous, because death results in the colostomy not getting closed, rather than vice versa.

For a real example, consider a study in which we examined the effect of discharge summary dissemination upon readmission to hospital [7]. Our “baseline-immeasurable” time-dependent variable was “Did the patient have a follow-up visit with a physician who had received the discharge summary.” When it was analyzed as a fixed variable, we found a large reduction in readmission to hospital when patients saw physicians with the summary (adjusted hazard ratio [HR] 0.35, 95% CI 0.24–0.52). This is a biased association, because patients who are readmitted to the hospital early after discharge do not have a chance to see such physicians and are placed in the “no-summary” group. This makes outcomes for the “no-summary” group particularly poor, thereby making follow-up with physicians who had the summary look beneficial by comparison. When a time-dependent analysis is used, we found a much smaller effect of follow-up with a physician who had received the summary (adjusted HR 0.74, 95% CI 0.50–1.11).

In these two examples, biased conclusions occur because patient categorization with respect to the “baseline-immeasurable” time-dependent variable was partially due to their outcome. Glesby and Hoover termed this “survivor treatment selection bias” [8]. Because this bias is not limited to treatment variables or analyses in which survival is the outcome, we will use the more generic term “time-dependent bias” in this review. Time-dependent bias can be avoided with the proper use of a time-dependent covariate analysis [8], [9], [10], [11].

There are several reasons why time-dependent bias could be common in the medical literature. The use of time-to-event analyses has increased in the medical literature over the last 20 years [12]. In our experience, time-dependent analysis of survival data is infrequently taught during research training and is often given a cursory review in some epidemiologic textbooks. In many journals, articles infrequently undergo formal statistical review prior to publication [13].

We critically reviewed the survival analyses published in prominent general medical journals to determine whether time-dependent bias due to improper analytical methodology is common. We also wanted to determine if its prevalence has recently changed, and how often study conclusions would qualitatively change if the analysis had properly accounted for the time-dependent nature of the data.

Section snippets

Study selection and inclusion criteria

We reviewed studies containing survival analyses that were published in nine medical journals including the American Journal of Medicine, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal (BMJ), Circulation, Chest, Journal of the American Medical Association (JAMA), Lancet, and New England Journal of Medicine. These journals were selected because they are general medical, respiratory, and cardiovascular journals with a broad readership, large circulation, and a

Results

Our search strategy resulted in 1,184 papers (see Appendix A). Four hundred seventy-seven papers did not have a survival analysis (Fig. 1). Of the 707 survival analyses, 25 were excluded because they did not determine the association of an outcome with any patient factor.

This left 682 survival analyses that determined the association of at least one variable with an outcome (Table 1). The number of survival analyses published per year increased during the study period. The Kaplan-Meier method,

Discussion

We found that time-dependent bias due to improper analytic methodology is surprisingly common in these medical journals. When present, time-dependent bias frequently affected the strength of a purported association that was highlighted in the abstract. The prevalence of time-dependent bias appears to be decreasing. In over half of the papers susceptible to time-dependent bias, its correction could have changed the studies' conclusion.

Time-dependent bias is important for several reasons. It can

Acknowledgements

Dr. van Walraven is an Ontario Ministry of Health Career Scientist. We greatly appreciate comments from Drs. Don Redelmeier, Dean Fergusson, Paul Hebert, and David Sackett regarding previous versions of this study.

References (97)

  • W. Zhou et al.

    Counting alleles to predict recurrence of early-stage colorectal cancers

    Lancet

    (2002)
  • A. Coutsoudis et al.

    Influence of infant-feeding patterns on early mother-to-child transmission of HIV-1 in Durban, South Africa: a prospective cohort study. South African Vitamin A Study Group

    Lancet

    (1999)
  • J.B. Cologne et al.

    Longevity of atomic-bomb survivors

    Lancet

    (2000)
  • L.M. Brumble et al.

    Prevention of cytomegalovirus infection and disease after lung transplantation : results using a unique regimen employing delayed ganciclovir

    Chest

    (2002)
  • M. Brenner et al.

    Survival following bilateral staple lung volume reduction surgery for emphysema

    Chest

    (1999)
  • T. Rokkas et al.

    The association between CagA status and the development of esophagitis after the eradication of Helicobacter pylori

    Am J Med

    (2001)
  • R.W. Morse et al.

    Rest-redistribution 201-Tl single-photon emission CT imaging for determination of myocardial viability: relationship among viability, mode of therapy, and long-term prognosis

    Chest

    (1999)
  • P. Keiser et al.

    Granulocyte colony-stimulating factor use is associated with decreased bacteremia and increased survival in neutropenic HIV-infected patients

    Am J Med

    (1998)
  • J.H. Khan et al.

    Pulmonary metastases of endocrine origin: the role of surgery

    Chest

    (1998)
  • Y.Y. Liu et al.

    Prognosis and recurrent patterns in bronchioloalveolar carcinoma

    Chest

    (2000)
  • M. Gilljam et al.

    Pregnancy in cystic fibrosis. Fetal and maternal outcome

    Chest

    (2000)
  • P.A. Rochon et al.

    Rate of heart failure and 1-year survival for older people receiving low-dose beta-blocker therapy after myocardial infarction

    Lancet

    (2000)
  • S. Kobayashi et al.

    Poor bone quality or hip structure as risk factors affecting survival of total-hip arthroplasty

    Lancet

    (2000)
  • W.F. Kanasky et al.

    Impact of body weight on long-term survival after lung transplantation

    Chest

    (2002)
  • D.D. Sin et al.

    The impact of follow-up physician visits on emergency readmissions for patients with asthma and chronic obstructive pulmonary disease: a population-based study

    Am J Med

    (2002)
  • P. Chassagne et al.

    Fecal incontinence in the institutionalized elderly: incidence, risk factors, and prognosis

    Am J Med

    (1999)
  • C.A. Gutierrez et al.

    Cytomegalovirus viremia in lung transplant recipients receiving ganciclovir and immune globulin

    Chest

    (1998)
  • P.N. Chhajed et al.

    Interventional bronchoscopy for the management of airway complications following lung transplantation

    Chest

    (2001)
  • B. Ledergerber et al.

    Clinical progression and virological failure on highly active antiretroviral therapy in HIV-1 patients: a prospective cohort study. Swiss HIV Cohort Study

    Lancet

    (1999)
  • C.E. Behrendt

    Acute respiratory failure in the United States: incidence and 31-day survival

    Chest

    (2000)
  • J.A. Henke et al.

    Persistent increases of BAL neutrophils as a predictor of mortality following lung transplant

    Chest

    (1999)
  • S. Itescu et al.

    An immunological algorithm to predict risk of high-grade rejection in cardiac transplant recipients

    Lancet

    (1998)
  • A.A. El Solh et al.

    Clinical and radiographic manifestations of uncommon pulmonary nontuberculous mycobacterial disease in AIDS patients

    Chest

    (1998)
  • R. Hubbard et al.

    Survival in patients with cryptogenic fibrosing alveolitis: a population-based cohort study

    Chest

    (1998)
  • M. Metintas et al.

    Cisplatin, mitomycin, and interferon-alpha2a combination chemoimmunotherapy in the treatment of diffuse malignant pleural mesothelioma

    Chest

    (1999)
  • J.B. Cavender et al.

    Effects of smoking on survival and morbidity in patients randomized to medical or surgical therapy in the Coronary Artery Surgery Study (CASS): 10-year follow-up. CASS Investigators

    J Am Coll Cardiol

    (1992)
  • D.G. Kleinbaum et al.

    Fundamentals of epidemiologic research

  • S.M. McGuigan

    The use of statistics in the British Journal of Psychiatry

    Br J Psychiatry

    (1995)
  • D.G. Altman et al.

    Time to event (survival) data

    BMJ

    (1998)
  • C. van Walraven et al.

    The effect of discharge summary availability during post-discharge outpatient visits on readmission to hospital

    J Gen Intern Med

    (2002)
  • M.J. Glesby et al.

    Survivor treatment selection bias in observational studies: examples from the AIDS literature

    Ann Intern Med

    (1996)
  • D.R. Cox

    Regression models and life tables

    J R Stat Soc

    (1972)
  • P.D. Allison

    Estimating Cox-regression models with PROC PHREG. Survival analysis using the SAS system

    (2000)
  • L.D. Fisher et al.

    Time-dependent covariates in the Cox proportional-hazards regression model

    Annu Rev Public Health

    (1999)
  • T.R. Fleming et al.

    Survival analysis in clinical trials: past developments and future directions

    Biometrics

    (2000)
  • S.N. Goodman et al.

    Statistical reviewing policies of medical journals: caveat lector?

    J Gen Intern Med

    (1998)
  • H. Yoshida et al.

    Interferon therapy reduces the risk for hepatocellular carcinoma: national surveillance program of cirrhotic and noncirrhotic patients with chronic hepatitis C in Japan. IHIT Study Group. Inhibition of Hepatocarcinogenesis by Interferon Therapy

    Ann Intern Med

    (1999)
  • A.P. Guerin et al.

    Impact of aortic stiffness attenuation on survival of patients in end-stage renal failure

    Circulation

    (2001)
  • Cited by (272)

    • Comparative effectiveness of individual pneumococcal vaccines with dual pneumococcal vaccination in older United States Veterans

      2022, Vaccine
      Citation Excerpt :

      We identified three exposure groups: PCV 13, only PPSV 23 and dual vaccination (combined PCV 13 and PPSV 23). As our exposure groups are based on a time-varying covariate – pneumococcal vaccination status, simply assigning individuals to their initial vaccination status would introduce time-dependent bias.[22] Hence, we employed a within-patient crossover design to appropriately assign exposure time in person-days to the appropriate exposure groups.[23,24]

    View all citing articles on Scopus
    View full text