|
| Abstract |
|---|
|
|
|---|
Methods: We used neutral search terms to identify articles about gene discoveries and associated technologies hosted on the Dow Jones Interactive and Canadian NewsDisk databases from January 1995 to June 2001. We compared the contents, claims and conclusions of the scientific journal article with those of the associated newspaper article. Coders subjectively assigned the newspaper articles to 1 of 3 categories: moderately to highly exaggerated claims, slightly exaggerated claims or no exaggerated claims. We used classification tree software to identify the variables that contributed to the assignment of each newspaper article to 1 of the 3 categories: attention structure (positioning in the newspaper and length of the article), authorship, research topic, source of information other than the scientific paper, type and likelihood of risks and benefits, discussion of controversy, valuation tone (positive or negative), framing (e.g., description of research, celebration of progress, report of economic prospects or ethical perspective), technical accuracy (either omissions or errors that changed the description of the methods or interpretation of the results) and use of metaphors.
Results: We examined 627 newspaper articles reporting on 111 papers published in 24 scientific and medical journals. Only 11% of the newspaper articles were categorized as having moderately to highly exaggerated claims; the majority were categorized as having no claims (63%) or slightly exaggerated claims (26%). The classification analysis ranked the reporting of risks as the most important variable in determining the categorization of newspaper articles. Only 15% of the newspaper articles and 5% of the scientific journal articles discussed costs or risks, whereas 97% of the newspaper articles and 98% of the scientific journal articles discussed the likelihood of benefits of the research.
Interpretation: Our data suggest that the majority of newspaper articles accurately convey the results of and reflect the claims made in scientific journal articles. Our study also highlights an overemphasis on benefits and under-representation of risks in both scientific and newspaper articles. The cause and nature of this trend is uncertain.
Several studies and commentaries have suggested various degrees and sources of reporting inaccuracies. For example, institutional press releases may be incomplete (with risks and limits being underplayed),10 media stories about medications often include inadequate or incomplete information about risks and costs,11 and abstracts from scientific meetings often receive a significant amount of media attention even though the validity and importance of the research has not "been established in the scientific community." 12 Some have speculated about the general weaknesses of the popular media's coverage of medical breakthroughs13,14 and their preference for positive over negative results.15
Surprisingly few systematic studies have examined the accuracy of media reporting in the context of genetics, most focusing on coverage of a single issue, such as sexual orientation16,17 or the discovery of susceptibility to breast and prostate cancer.18,19 The available data are instructive but not definitive.14,16,17,18,19,20
We examined broadsheet newspaper coverage of genetic research in Canada, the United States, Great Britain and Australia, countries with a scientific community heavily involved in genetic research. We attempted to measure media "hype" and the factors that contribute to inaccuracy and exaggeration. In addition, we attempted to gain insight into the possible source (researchers, scientific journals or the media) and nature of any "hype."
| Methods |
|---|
|
|
|---|
We then identified all newspaper stories that had as their source a paper in a peer-reviewed scientific journal. We used specific search terms to locate the scientific papers in MEDLINE or PubMed and then to identify all associated print media articles in the newspaper databases. The specific search terms included the name of the journal, the name of one of the researchers, the name of the institution where the research had been conducted, and the name of the gene, disease or genetic technology. We limited our study to all articles appearing in 26 newspapers from 4 countries. The broadsheet newspapers selected from the United States, Great Britain and Australia were chosen because they were national in distribution, had a large circulation, were of high quality, printed articles from one of the major newswire services and were roughly equivalent to the Canadian newspapers. We included most major Canadian local and national papers to ensure regional representation and thus captured most articles on gene discoveries and related technologies that were printed in that country. If an article appeared in more than one newspaper, it was coded only once; in such cases, priority was given to articles published in 1 of the 2 national newspapers, the National Post and the Globe and Mail.
We developed a coding frame, based on a previously published European media study,21 to compare the content of the scientific paper with the newspaper articles it generated. The coding frame consisted of a series of questions with standardized categorical responses on identifying information for the article; attention structure (positioning in the newspaper and length of the article); authorship; content; source of information other than the scientific paper; assessment of risk, benefits or controversy; valuation tone; framing; accuracy; and use of metaphors (see Appendix 1). All questions except those on framing, authorship and attention structure were then repeated for the scientific paper.
The coders, all of whom had scientific backgrounds, were asked to subjectively assess both the technical accuracy of the newspaper article and whether the claims made in the newspaper article reflected those in the scientific journal article. For the latter, the coders assigned each newspaper article to 1 of 3 categories: moderately to highly exaggerated claims, slightly exaggerated claims and no exaggerated claims. We assumed that there were no technical errors in the scientific journal articles. A newspaper article was considered not to have been exaggerated if its claims had first been made in the corresponding scientific journal article.
We conducted a pilot study with 2 independent coders to test the coding frame. For the study itself, we used 3 independent coders who were not familiar with the project and who had a strong background in genetics and biotechnology. We held 2 moderated meetings early in the study so that the coders could discuss their interpretation of the coding frame.
To assess intercoder reliability, we asked each of the 3 coders to assess the same random selection of 84 (13% of the total) newspaper articles and their associated scientific papers. We calculated intraclass correlation coefficients (model 2)22 for questions in the coding frame with mutually exclusive response. In all cases the coefficient was greater than 0.75 (Table 1), which indicates good agreement.22
|
We performed a classification tree analysis, using CART classification and regression tree software, version 4.0,23,24,25 to determine which variables from the coding frame contributed to the assignment of a newspaper article to 1 of the 3 categories of claims. This nonparametric statistical analysis has been used extensively for clinical risk assessment.25,26,27,28 The CART software uses specific criteria to split the data into 2 groups with greater homogeneity than the original group.24,25 Each group is then split into increasingly homogeneous groups until stopping criteria are reached and a set of terminal groups is produced. The variables are then ranked according to their importance as "splitting variables" (i.e., the relative contribution of each variable in assignment of the newspaper articles to the 3 categories).
| Results |
|---|
|
|
|---|
Twenty percent (127) of the newspaper articles were written by clearly identifiable, in-house specialist science and technology or medical journalists, whereas 22% (138) were identified as being from a newswire service; the latter were almost exclusively brief announcements of a gene discovery with little or no analysis.
In 95% of the newspaper articles (591/625), the main source of information cited was the scientific paper or its authors (see Appendix 1). This is not surprising, given our search strategy. Only 2 newspaper articles cited press releases as their source. Public sector research scientists (those perceived to be associated with a public sector employer, such as a university) were interviewed or cited in 90% of the newspaper articles (563/626), and private sector scientists in 5% (33/626). Opinions outside the research community or biotechnology sector were sought in only 8% (50/626) of newspaper articles, and these opinions were considered the main "voice" in only 2% (14/626) of the newspaper articles. Ninety percent of the newspaper articles (564/627) were framed as either a description of the basic research or as a celebration of progress or new research developments (Appendix 1). Most of the articles (483 [77%]) were not framed as a controversy; only 14% (85) were framed as a balanced controversy and 9% (59) as an imbalanced controversy.
The majority of newspaper articles were categorized as having no exaggerated claims (393 [63%]) or slightly exaggerated claims (165 [26%]). Only 11% (69) of the newspaper articles were categorized as having moderately to highly exaggerated claims (Table 1). Similarly, 82% (513) of the articles had no significant technical or scientific errors, and 18% (112) had 1 to 3 such errors. Only 2 of the newspaper articles contained more than 3 significant errors. Canadian newspapers, both in general and in comparison with those in other countries, had low levels of exaggerated claims in their reporting (Fig. 1).
|
Forty-six (41%) of the 111 scientific papers reported research on human genetics, 40 (36%) used a nonhuman model, and 25 (23%) used both (Table 1). However, 87% (97/111) of the scientific papers and 98% (612/627) of the newspaper articles extrapolated the results to humans, regardless of whether the research model was human.
Beginning with the entire sample, CART identified the variable that best split the data into 2 subgroups with greater homogeneity than the whole and then repeated the process to create a classification tree (see Appendix 2). The relative contribution of the variables (their importance) as primary splitting variables was used in determining the category of exaggerated claims for the newspaper articles (Table 2). The misclassification cost for the learn data indicated a good fit of the model to the data: 0.25 for moderately to highly exaggerated claims, 0.24 for slightly exaggerated claims and 0.13 for no exaggerated claims.
|
The most important variable in determining the categorization of newspaper articles was the likelihood of risks or costs being reported in the newspaper article, and the fourth most important variable was whether risks or costs were reported in the scientific paper (Table 2). Appendix 1 shows why mention of risk may be an important variable. Only 15% of the newspaper articles and just 5% of the scientific journal articles discussed costs or risks. Of the 94 newspaper articles that reported risks, 67 (71%) quantified the risk as very unlikely to somewhat likely, and 25 (27%) quantified the risk as very likely to already present. In the 6 scientific journal articles that mentioned risk, the assessment of risk ranged from very unlikely to very likely. In contrast, 97% of the newspaper articles and 98% of the scientific journal articles discussed the likelihood of benefits arising from the research (Appendix 1); the type and likelihood of benefit were therefore identified as less important in the CART analysis (Table 2). About 70% of the newspaper and scientific journal articles quantified these benefits as very likely or certain.
Also important were the themes of the scientific paper and the associated newspaper articles (Table 2). Newspaper articles on behavioural genetics (e.g., genes for sexual orientation, alcoholism, mental illness or criminality), genetically modified organisms (e.g., "glow-in-the-dark" rhesus monkeys or genetically modified killer pox virus), longevity (e.g., clock genes) or reproductive technologies (e.g., ooplasmic transplantation) were more likely to include moderately to highly exaggerated claims. Newspaper articles on diseases such as obesity were also more likely to include exaggerated claims, whereas articles on life-threatening and prevalent diseases such as cancer, stroke and heart disease did not include exaggerated claims (data not shown).
Our data may also indicate a more subtle form of media hype, in terms of what research newspapers choose to cover. Behavioural genetics and neurogenetics were the subject of 16% of the newspaper articles. A search of PubMed on May 30, 2003, with the term "genetics" yielded 1 175 855 hits, and searches with the terms "behavioural genetics" and "neurogenetics" yielded a total of 3587 hits (less than 1% of the hits for "genetics"). Although the latter is probably an underrepresentation of primary research papers in these fields and the former an overrepresentation of all research in genetics, the magnitude of the difference suggests that behavioural genetics is receiving a disproportionately large amount of coverage in the media.
| Interpretation |
|---|
|
|
|---|
In our study the 2 most important variables in determining the category of exaggeration were the reporting of risks and costs and the topic of the research. Our study also highlights an underreporting of risks and an overemphasis on benefits. A small proportion of the newspaper articles (15%) and an even smaller proportion of the scientific journal articles (5%) dealt with risks. Again, this finding is consistent with existing data. For example, Moynihan and associates11 found that only 15% of 207 news stories on drugs used for disease prevention presented both relative and absolute benefits. Similarly, in a study of Canadian newspaper coverage of new prescription drugs, all articles mentioned some benefit, but the majority made no mention of side effects or harms.30 Our study corroborates concern about the lack of reporting of risk, in both the scientific literature and newspaper coverage. In the rare instances that risk is reported in either type of article, there is a greater likelihood of exaggeration. In contrast, benefits are emphasized in both scientific and newspaper articles. This trend may contribute to a general hyping of genetic research, potentially inflating the expectations of the general public and special interest groups such as patient groups and investors.
Our study had a number of limitations. First, we surveyed only the print media and limited our analysis to the more respected newspaper publications, which do not necessarily have circulation levels as high as some tabloids. Second, our study was limited to newspaper stories that were directly related to peer-reviewed articles and published abstracts. As a result, stories flowing from other sources, such as abstracts from scientific meetings, which might never be published,12 may be underrepresented. Also, all of our coders had strong scientific backgrounds, which undoubtedly brought a certain perspective to the interpretation of both the newspaper stories and the scientific articles.
Nevertheless, our data indicate that journalists may not always be the primary source of exaggerated claims. Although more research is required to confirm the nature and cause of this trend (e.g., investigating the editorial practices of the top journals and analyzing other media such as television and the Internet), a reasonable interpretation is that the media, scientific journals and the scientific community at large may be inadvertent "complicit collaborators" in the subtle hyping of science stories.8 Our data also raise interesting questions about how stories are selected and edited by the top scientific journals (e.g., the possibility of selection bias) and how the research community may "sell" science to the public, to scientific and lay publications, and to research funders.31,32,33,34
ß See related article page 1415
| Appendix 1 |
|---|
|
|
|---|
|
| Appendix 2 |
|---|
|
|
|---|
|
| Footnotes |
|---|
Contributors: Both authors contributed to the study conception and design and the data analysis and interpretation. Both drafted and revised the article and approved the final version.
Acknowledgements: We thank Edna Einsiedel and Usher Fleising for study design; Tim Karels for statistical advice; David Hik for statistical advice and editorial comments; Jordan Ward, Christine Caulfield and Qusair Mohammedbai for coding; and Nina Hawkins at the Health Law Institute for administrative assistance.
This study was supported by Genome Prairie, the Social Sciences and Humanities Research Council of Canada, the Alberta Heritage Foundation for Medical Research and the Stem Cell Network.
Competing interests: None declared.
Correspondence to: Timothy A. Caulfield, Health Law Institute, Law Centre, University of Alberta, 91 University Campus NW, Edmonton AB T6G 2H5; fax 780 492-9575; tcaulfld{at}law.ualberta.ca
| References |
|---|
|
|
|---|
Related Article
This article has been cited by other articles:
![]() |
H. Liu and S. Priest Understanding public support for stem cell research: media communication, interpersonal communication and trust in key actors Public Understanding of Science, November 1, 2009; 18(6): 704 - 718. [Abstract] [PDF] |
||||
![]() |
J. Brechman, C.-j. Lee, and J. N. Cappella Lost in Translation?: A Comparison of Cancer-Genetics Reporting in the Press Release and Its Subsequent Coverage in the Press Science Communication, June 1, 2009; 30(4): 453 - 474. [Abstract] [PDF] |
||||
![]() |
M Legge and L M Jones Stem cell spinal cord regeneration: first do no harm J. Med. Ethics, December 1, 2008; 34(12): 838 - 839. [Abstract] [Full Text] [PDF] |
||||
![]() |
T. Caulfield and S. Harry Popular Representations of Race: The News Coverage of BiDil. J. Law Med. Ethics, September 1, 2008; 36(3): 485 - 490. [PDF] |
||||
![]() |
R. Reis How Brazilian and North American Newspapers Frame the Stem Cell Research Debate Science Communication, March 1, 2008; 29(3): 316 - 334. [Abstract] [PDF] |
||||
![]() |
J. Nisker and A. S. Daar Moral presentation of genetics-based narratives for public understanding of genetic science and its implications Public Understanding of Science, January 1, 2006; 15(1): 113 - 123. [Abstract] [PDF] |
||||
![]() |
R. L. Kravitz Doing Things Better vs Doing Better Things Ann. Fam. Med, November 1, 2005; 3(6): 483 - 485. [Full Text] [PDF] |
||||
![]() |
D. J. Bowen, K. M. Battuello, and M. Raats Marketing Genetic Tests: Empowerment or Snake Oil? Health Educ Behav, October 1, 2005; 32(5): 676 - 685. [Abstract] [PDF] |
||||
![]() |
C. Wang, D. J. Bowen, and S. L. R. Kardia Research and Practice Opportunities at the Intersection of Health Education, Health Behavior, and Genomics Health Educ Behav, October 1, 2005; 32(5): 686 - 701. [Abstract] [PDF] |
||||
![]() |
A. Miah Genetics, cyberspace and bioethics: why not a public engagement with ethics? Public Understanding of Science, October 1, 2005; 14(4): 409 - 421. [Abstract] [PDF] |
||||
![]() |
J. Nisker and A. White The CMA Code of Ethics and the donation of fresh embryos for stem cell research Can. Med. Assoc. J., September 13, 2005; 173(6): 621 - 622. [Full Text] [PDF] |
||||
![]() |
P. Sankar, M. K. Cho, C. M. Condit, L. M. Hunt, B. Koenig, P. Marshall, S. S.-J. Lee, and P. Spicer Genetic Research and Health Disparities JAMA, June 23, 2004; 291(24): 2985 - 2989. [Abstract] [Full Text] [PDF] |
||||
![]() |
D. Spurgeon Media hype? It's not as bad as it seems BMJ, May 8, 2004; 328(7448): 1140 - 1140. [Full Text] |
||||
![]() |
C. Condit Science reporting to the public: Does the message get twisted? Can. Med. Assoc. J., April 27, 2004; 170(9): 1415 - 1416. [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||