Systematic reviews are useful tools because they combine the efficacy results of individual studies and provide health care providers and others with a picture of the overall impact of an intervention. Resources exist to help ensure that appropriate methods are used to conduct and report systematic reviews.1,2
For a systematic review of an intervention that has been in the marketplace for a considerable time, the review team will search a number of sources for reports of studies that might be relevant for inclusion in the review. Most of these reports will have already been published, and only a minority will be unpublished studies such as internal reports and abstracts. New technologies, including drugs, are those that have only recently gained regulatory approval, and thus the complete body of evidence may not be available for public scrutiny. This poses additional problems for systematic reviewers.3 In this article we highlight some of these challenges along with possible solutions.
It is difficult to obtain published reports evaluating a new technology. Perhaps this is because of the considerable delay from the time of ethics approval to the subsequent publication of statistically positive results (median 5 years) or statistically negative results (median 8 years).4 Thus, systematic reviewers must rely on grey literature (e.g., unpublished reports, with limited distribution). The inclusion of unpublished data can have its advantages, however. A systematic review of data from only published reports can present a misleading picture of an intervention's efficacy. The exclusion of grey literature may exaggerate the estimates of the intervention's efficacy by 15%–38% depending on the type of grey literature.5 Abstracts are typically the most common type, one report estimating they account for 60% of the grey literature,5 and their exclusion results in the largest amount of bias in a systematic review.5 It is unclear whether there would be more or less bias if the unpublished studies were in the form of full reports rather than abstracts. As well, the typical abstract of 250 words likely cannot report all of the possible outcome data or all of the information needed to know to which population(s) and intervention(s) any reported results can be generalized. Moreover, by virtue of the relative paucity of details about study characteristics in abstracts, different abstracts might not be recognized as referring to the same study; this “covert duplication” can lead to biased estimates of the efficacy of an intervention.6 Thus, both the exclusion and the inclusion of grey literature (e.g., abstracts) create specific difficulties for systematic reviewers of new health technologies. Nonetheless, the inclusion of unpublished data along with the results of published reports is of paramount importance. Without the former, the question of whether an intervention is “useful” or “not useful” cannot be answered completely.
What, then, can be done to rectify these problems? One solution is to publish all abstracts presented at scientific meetings and provide hypertext links to more complete information or data. Electronic publishing houses and print journals with Web sites can greatly facilitate this service. Full-text searching of these hypertext links and the use of unique identifiers for each study, such as the International Standard Randomized Controlled Trial Number (ISRCTN),7 could go a long way to ensuring that the data archives are easily and efficiently accessible. Perhaps a committee spearheaded by journal editors and comprising all interested parties would facilitate this process. This initiative could be coupled with one proposed by Sim and associates,8 whereby journals, in addition to publishing reports of trials, would deposit the trial information necessary for completing a systematic review into an electronic trial bank.
One way to increase the likelihood that even brief reports would contain the information and data detailed enough to afford a qualitative summary at the very least might be to revisit the use of structured abstracts. It is unclear why conference (and journal) abstracts need to be limited to 250 words, especially if a longer version would improve their validity and usefulness. The 250-word limit seems arbitrary and could be extended. The precise content of an extended abstract requires further consideration. Its format could be adapted from the CONSORT statement.9
Better dialogue between investigators and trial sponsors is needed so that the required content material can be obtained, although attempts to get information have not always been successful.3,10 We do recognize, however, that sudden requests and large numbers of requests for data will overburden some recipients, particularly those with limited resources to supply the information. One way to minimize the burden would be for reviewers to request essential data only. Requests could be sent to a sponsor-appointed “data liaison officer.” Perhaps groups such as the Cochrane Collaboration could help to develop a standard template for such requests.
Often companies express their unwillingness to share data, citing their proprietary nature, requiring protection of intellectual property first or stating that the data are part of a report currently being submitted for publication. However, notwithstanding the ethical and scientific shortcomings of this stance,11 it is hard to fathom how, or why, sharing limited, albeit crucial, information from a study would preclude publication in a reputable journal. Moreover, the temporal relation between publication of a primary report and any subsequent systematic review diminishes justification for withholding information from systematic reviewers.
These points may be more pertinent to the evaluation of drugs, which traditionally have had a more rigorous approval process. We do not know whether similar concerns exist for systematic reviews of other new technologies, such as devices.
Abstracts and other grey literature are key ingredients in the systematic review of a new technology. In addition to the solutions described earlier of expanding the length and content of brief reports (e.g., conference abstracts) and improving the accessibilty of trial information and trial data, it might also be helpful to have reputable journals become more vocal about the importance of grey literature. They could assert, for example, that the release of such information would not preclude consideration of a primary study for publication in their journal. In a similar vein drug company sponsors need to inform the larger scientific and consumer community of the existence of studies under their sponsorship and help ensure timely access to this information.
We believe that opportunities exist to remedy these problems. Systematic reviewers would then find it easier to provide health care providers, policy analysts and consumers alike with a clear and comprehensive picture of a new health technology's efficacy.
Footnotes
-
This article has been peer reviewed.
Contributors: Both authors conceived of the paper and participated in writing and revising it.
Acknowledgements: We thank Dr. Drummond Rennie and Ms. Margaret Sampson for their comments on an earlier draft of this article.
Competing interests: None declared.