Almost all articles on cancer prognostic markers report statistically significant results
Introduction
Cancer prognosis has been a field of intensive research for many years. Besides traditional clinical markers, basic and translational research have generated hundreds of candidate markers for prediction of outcomes in cancer patients.1, 2 The expectation is that eventually some of these markers should also be clinically useful to change clinical practice.3, 4, 5, 6 However, progress in ‘individualised’ medicine based on prognostic information has been slow, in contrast to the vast amount of published data. Several methodological problems have been implicated for prognostic marker studies. They include poor study design and execution and poor and selective reporting of results.1, 7, 8, 9, 10
Selective reporting is a particular threat to the credibility of this literature. It includes both publication bias11, 12 and selective reporting of specific analyses and outcomes favouring results that pass the threshold of nominal statistical significance.9 These biases have been well documented even for rigorous study designs, such as randomised trials.13, 14 Selective reporting has been difficult to probe for prognostic investigations. There is no study registration mechanism and study protocols are typically not available.15 Therefore, here we aimed to probe into these biases using an indirect approach. We evaluated two large samples of articles on cancer prognostic markers and estimated how many of them claim nominally statistically significant findings. Publication and other selective reporting biases all cause an excess of statistically significant results in the literature.16, 17 With publication bias, studies with non-significant results would be left unpublished; thus, the published literature would be relatively enriched in statistically significant findings. With selective reporting of specific analyses or outcomes, the end result is the same: studies that should have been presented as ‘negative’ (non-statistically significant) based on their primary analyses get published with ‘positive’ (statistically significant) results based on data dredging and manipulated analyses.
Section snippets
Search strategy and eligibility criteria for cancer prognostic marker studies
We used two large databases of published articles. The first (Database 1) comprised 340 articles on cancer prognostic markers with data that were included in meta-analyses of cancer prognostic markers published until 2005. The database has been built as part of a previous project.10 In that project, we have identified 20 meta-analyses of prognostic markers for cancer by searching MEDLINE and EMBASE up to 2005. The primary studies included in these meta-analyses were used to create Database 1 in
Eligible articles
Database 1 included 340 articles and Database 2 incorporated 1575 articles.The articles had been published in 97 and 343 different journals, respectively (Appendix). With the exception of seven articles in Lancet, five in New England Journal of Medicine, four in JAMA and one in Nature Medicine, these were specialty journals. The 10 most common venues (along with the number of articles per database) were Clinical Cancer Research (24 + 127 = 151), Journal of Clinical Oncology (22 + 75 = 97), Cancer (27 + 62
Discussion
This survey shows that articles on cancer prognostic markers almost ubiquitously highlight significant prognostic associations. In the rare articles where no prognostic markers are presented as significant, authors often have other (non-prognostic) statistically significant analyses to show, they expand on the importance of non-significant trends, or defend the importance of the cancer marker with other arguments. Eventually, totally ‘negative’ articles on prognostic cancer markers represent
Conflict of interest statement
The authors declare that there is no conflict of interest regarding this submission.
References (39)
- et al.
Tumor markers as clinical cancer tests–are we there yet?
Semin Oncol
(2002) - et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Clinical trial registration: a statement from the International Committee of Medical Journal Editors
Lancet
(2004) - et al.
Reporting recommendations for tumor marker prognostic studies (REMARK)
Eur J Cancer
(2005) - et al.
Expression and prognostic significance of hypoxia-inducible factor 1alpha (HIF-1alpha) in malignant pleural mesothelioma (MPM)
Lung Cancer
(2006) - et al.
An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods
J Clin Epidemiol
(2004) Systematic reviews of evaluations of prognostic variables
BMJ
(2001)- et al.
Statistical aspects of prognostic factor studies in oncology
Br J Cancer
(1994) - et al.
Tumor marker utility grading system: a framework to evaluate clinical utility of tumor markers
J Natl Cancer Inst
(1996) - et al.
update of recommendations for the use of tumor markers in breast and colorectal cancer:clinical practice guidelines of the American Society of Clinical Oncology
J Clin Oncol
(2001)
Is molecular profiling ready for use in clinical decision making?
Oncologist
Methodological challenges in the evaluation of prognostic factors in breast cancer
Breast Cancer Res Treat
Missing covariate data within cancer prognostic studies:a review of current reporting and proposed guidelines
Br J Cancer
Selective reporting biases in cancer prognostic factor studies
J Natl Cancer Inst
Quality of reporting of cancer prognostic marker studies:association with reported prognostic effect
J Natl Cancer Inst
Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards
JAMA
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles
JAMA
Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research
CMAJ
An exploratory test for an excess of significant findings
Clin Trials
Cited by (192)
The role of randomized controlled trials, registries, observational databases in evaluating new interventions
2023, Best Practice and Research: Clinical HaematologyManagement of the cN0 neck in early oral cancer: time to revise the guidance?
2021, British Journal of Oral and Maxillofacial Surgery