Review Article
Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials

https://doi.org/10.1016/j.jclinepi.2004.10.019Get rights and content

Abstract

Background and Objective

Divergent results on the same scientific question generate controversy. We hypothesized that controversial data are attractive to investigators and editors, and thus the most extreme, opposite results would appear very early rather than late, as data accumulate, provided data can be generated rapidly.

Methods

We used data from MEDLINE-indexed meta-analyses of case–control studies on genetic associations (retrospective, hypothesis-generating research with usually rapid turnaround) and meta-analyses of randomized trials of health care interventions (prospective, targeted research that usually takes longer) sampled from the Cochrane Library. Using cumulative meta-analysis, we evaluated how the between-study variance for studies on the same question changed over time and at what point the studies with the most extreme results ever observed had been published.

Results

The maximal between-study variance was more likely to be recorded early in the 44 eligible meta-analyses of genetic associations than in the 37 meta-analyses of health care interventions (P = .013). At the time of the first heterogeneity assessment, the most favorable-ever result in support of a specific association was more likely to appear than the least favorable-ever result (22 vs. 10, P = .017); the opposite was seen at the second heterogeneity assessment (15 vs. 5, P = .031). Such a sequence of extreme opposite results was not seen in the clinical trials meta-analyses. The estimated between-study variance decreased over time in genetic association studies (P = .010), but not in clinical trials (P = .30).

Conclusion

In contrast to prospective trials, a rapid early sequence of extreme, opposite results is frequent in retrospective hypothesis-generating molecular research.

Introduction

Diverse investigators may generate data on the same scientific question, and these data may or may not agree among themselves. Although some uncertainty is unavoidable in science, extreme disagreements create confusion and controversy. It would be useful to understand how these controversies arise. An important issue is whether disagreements and the most extreme discrepancies between studies on the same question are more likely to appear early or late, as accumulating data are being published over time.

Scientific research is often subject to publication bias [1], [2] and time-lag bias [3]. It is well documented that small studies with nonsignificant results may remain unpublished [1], [2], and studies with significant results take a shorter time to be published than studies with nonsignificant results [3]. The more extreme an observed result, the more likely it is to be formally statistically significant and published faster. As a consequence, early publications may present findings that are out of proportion to the truth. If many more studies are performed on the same topic, their results should be, on average, less extreme than the results of the first study [4], [5], [6]. Yet is the order of publication of these subsequent studies determined by the results of the first study? We hypothesized that highly contradictory results are most tantalizing and attractive to investigators and editors; thus, they may also have an advantage for rapid publication against other results that are closer to the original first publications. In this case, one would expect to see early on in the literature a succession of the most extreme opposite results. Studies with intermediate, potentially less spectacular results may then be published at a slower pace, filling in the gap between the early extremes.

According to our hypothesis, an early rapid succession of extremes would be most common when a large number of studies can be performed worldwide in a short period of time. This may be typical of retrospective study designs, especially in fields with intense research activity. Conversely, this rapid succession of extremes would not be observed for prospective research where studies take considerable time to perform and their conduct is spread over a longer time span. Even if the most contradictory findings have some advantage for more rapid publication, once an extreme result has been reported, this will not suffice to make them appear in the literature before studies with less contradictory findings that have been completed much earlier. To test our hypothesis, we used two large databases of meta-analyses of case–control studies on genetic associations for complex diseases and prospective clinical trials on the efficacy of health care interventions.

Section snippets

Meta-analysis framework

We used a meta-analytic perspective [7]. A meta-analysis gathers all the studies on the same question, so it is possible to examine when the studies with the most extreme observed results have been published relative to others. In cumulative meta-analysis [8], the results of all studies on a given question are sequentially summarized as they are published over time. Both the summary results and variance estimates may change over time [9]. At the end of each year, summary results are routinely

Comparison of meta-analyses

The two sets of meta-analyses had a similar number of studies and heterogeneity assessments, and were not significantly different in the proportion of cases with overall statistically significant odds ratios or between-study heterogeneity. However, as expected, clinical trials had been published over a significantly broader time span than the genetic association studies. Clinical trials also had a smaller sample size (Table 1).

Early extreme contradictory estimates in genetic associations

In 40 of the 44 meta-analyses of genetic association studies, some

Discussion

Publication bias and time lag bias are well-established problems in the biomedical research literature [1], [2], [3] and beyond [24], [25], [26], [27] and are responsible for the appearance of spuriously favorable results in the peer-reviewed literature. Here we show an additional bias that results in the appearance of extreme, contradictory findings very early during the accumulation of scientific evidence. This phenomenon may occur when a large pool of potential analyses is available and many

Acknowledgments

The work was supported by a PENED grant from the General Secretariat for Research and Technology, Greece, and the European Commission. There were no competing financial interests. We thank Evangelia E. Ntzani and Despina G. Contopoulos-Ioannidis for their contribution in background work.

References (31)

  • K. Dickersin et al.

    Publication bias: the problem that won't go away

    Ann N Y Acad Sci

    (1993)
  • J.P. Ioannidis

    Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials

    JAMA

    (1998)
  • J.P. Ioannidis et al.

    Replication validity of genetic association studies

    Nat Genet

    (2001)
  • J. Lau et al.

    Cumulative meta-analysis of therapeutic trials for myocardial infarction

    N Engl J Med

    (1992)
  • Cited by (0)

    View full text