Elsevier

European Journal of Cancer

Volume 43, Issue 17, November 2007, Pages 2559-2579
European Journal of Cancer

Almost all articles on cancer prognostic markers report statistically significant results

https://doi.org/10.1016/j.ejca.2007.08.030Get rights and content

Abstract

We aimed to understand the extent of the pursuit for statistically significant results in the prognostic literature of cancer. We evaluated 340 articles included in prognostic marker meta-analyses (Database 1) and 1575 articles on cancer prognostic markers published in 2005 (Database 2). For each article, we examined whether the abstract reported any statistically significant prognostic effect for any marker and any outcome (‘positive’ articles). ‘Negative’ articles were further examined for statements made by the investigators to overcome the absence of prognostic statistical significance. We also examined how the articles of Database 1 had presented the relative risks that were included in the respective meta-analyses. ‘Positive’ prognostic articles comprised 90.6% and 95.8% in Databases 1 and 2, respectively. Most of the ‘negative’ prognostic articles claimed significance for other analyses, expanded on non-significant trends or offered apologies that were occasionally remote from the original study aims. Only five articles in Database 1 (1.5%) and 21 in Database 2 (1.3%) were fully ‘negative’ for all presented results in the abstract and without efforts to expand on non-significant trends or to defend the importance of the marker with other arguments. Of the statistically non-significant relative risks in the meta-analyses, 25% had been presented as statistically significant in the primary papers using different analyses compared with the respective meta-analysis. We conclude that almost all articles on cancer prognostic marker studies highlight some statistically significant results. Under strong reporting bias, statistical significance loses its discriminating ability for the importance of prognostic markers.

Introduction

Cancer prognosis has been a field of intensive research for many years. Besides traditional clinical markers, basic and translational research have generated hundreds of candidate markers for prediction of outcomes in cancer patients.1, 2 The expectation is that eventually some of these markers should also be clinically useful to change clinical practice.3, 4, 5, 6 However, progress in ‘individualised’ medicine based on prognostic information has been slow, in contrast to the vast amount of published data. Several methodological problems have been implicated for prognostic marker studies. They include poor study design and execution and poor and selective reporting of results.1, 7, 8, 9, 10

Selective reporting is a particular threat to the credibility of this literature. It includes both publication bias11, 12 and selective reporting of specific analyses and outcomes favouring results that pass the threshold of nominal statistical significance.9 These biases have been well documented even for rigorous study designs, such as randomised trials.13, 14 Selective reporting has been difficult to probe for prognostic investigations. There is no study registration mechanism and study protocols are typically not available.15 Therefore, here we aimed to probe into these biases using an indirect approach. We evaluated two large samples of articles on cancer prognostic markers and estimated how many of them claim nominally statistically significant findings. Publication and other selective reporting biases all cause an excess of statistically significant results in the literature.16, 17 With publication bias, studies with non-significant results would be left unpublished; thus, the published literature would be relatively enriched in statistically significant findings. With selective reporting of specific analyses or outcomes, the end result is the same: studies that should have been presented as ‘negative’ (non-statistically significant) based on their primary analyses get published with ‘positive’ (statistically significant) results based on data dredging and manipulated analyses.

Section snippets

Search strategy and eligibility criteria for cancer prognostic marker studies

We used two large databases of published articles. The first (Database 1) comprised 340 articles on cancer prognostic markers with data that were included in meta-analyses of cancer prognostic markers published until 2005. The database has been built as part of a previous project.10 In that project, we have identified 20 meta-analyses of prognostic markers for cancer by searching MEDLINE and EMBASE up to 2005. The primary studies included in these meta-analyses were used to create Database 1 in

Eligible articles

Database 1 included 340 articles and Database 2 incorporated 1575 articles.The articles had been published in 97 and 343 different journals, respectively (Appendix). With the exception of seven articles in Lancet, five in New England Journal of Medicine, four in JAMA and one in Nature Medicine, these were specialty journals. The 10 most common venues (along with the number of articles per database) were Clinical Cancer Research (24 + 127 = 151), Journal of Clinical Oncology (22 + 75 = 97), Cancer (27 + 62

Discussion

This survey shows that articles on cancer prognostic markers almost ubiquitously highlight significant prognostic associations. In the rare articles where no prognostic markers are presented as significant, authors often have other (non-prognostic) statistically significant analyses to show, they expand on the importance of non-significant trends, or defend the importance of the cancer marker with other arguments. Eventually, totally ‘negative’ articles on prognostic cancer markers represent

Conflict of interest statement

The authors declare that there is no conflict of interest regarding this submission.

References (39)

  • J.P. Ioannidis

    Is molecular profiling ready for use in clinical decision making?

    Oncologist

    (2007)
  • D.G. Altman et al.

    Methodological challenges in the evaluation of prognostic factors in breast cancer

    Breast Cancer Res Treat

    (1998)
  • A. Burton et al.

    Missing covariate data within cancer prognostic studies:a review of current reporting and proposed guidelines

    Br J Cancer

    (2004)
  • P.A. Kyzas et al.

    Selective reporting biases in cancer prognostic factor studies

    J Natl Cancer Inst

    (2005)
  • P.A. Kyzas et al.

    Quality of reporting of cancer prognostic marker studies:association with reported prognostic effect

    J Natl Cancer Inst

    (2007)
  • K. Dickersin et al.

    Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards

    JAMA

    (1992)
  • A.W. Chan et al.

    Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles

    JAMA

    (2004)
  • A.W. Chan et al.

    Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research

    CMAJ

    (2004)
  • J.P. Ioannidis et al.

    An exploratory test for an excess of significant findings

    Clin Trials

    (2007)
  • Cited by (192)

    • Management of the cN0 neck in early oral cancer: time to revise the guidance?

      2021, British Journal of Oral and Maxillofacial Surgery
    View all citing articles on Scopus
    View full text