Evidence of Methodologic Bias in the Derivation of the Science Citation Index Impact Factor☆,☆☆,★
Introduction
Over the past 20 years, emergency medicine has grown significantly as an academic discipline. Currently there are more than 50 allopathic academic departments of emergency medicine in the United States,1 a steadily growing number of abstracts are presented at scientific meetings,2, 3 and an increasing number of scholarly contributions are submitted to an expanding number of indexed, peer-reviewed journals.4, 5 In spite of this evidence of growth, the “impact factor” for emergency medicine journals, as published in Science Citation Index (SCI) by the Institute for Scientific Information (ISI),6 has reflected little of this development. The overall mean impact factor of the emergency medicine literature has remained below 1.0 since it was first calculated in 1981 (Figure). We wished to explore the basis for this disparity.
The SCI impact factor for a journal is defined mathematically as the number of times that journal's articles are cited during a specified period (the numerator) divided by the total number of articles published by that journal over the same period (the denominator). By convention, impact factors for a journal in a given year are based on calculations from the preceding 24 months. For example, if Journal A published a total of 200 articles in 1992 and 1993, and during that 2-year period articles from journal A were cited 100 times, Journal A's 1994 impact factor would be 100/200 or .5. It is important to note that only papers both published and cited during 1992 and 1993 are eligible to be counted as citations for calculation of the 1994 impact factor.
Citation counts (the numerator of the impact factor) are not obtained by tabulation of references from all indexed medical journals. Rather, only references from a predefined subset of all journals, referred to as “source” journals, are examined. Another group of indexed journals, referred to as “cited-only” journals, receive a citation count each time a reference to them appears in one of the source journals but are not themselves examined for citations. Finally, there is a third group of journals that are recognized by Index Medicus whose impact factors are not reported by SCI.
Although ISI reports impact factors for three of the eight emergency medicine journals listed in Index Medicus, no emergency medicine journal is included in its set of source journals. Therefore, all citations originating from emergency medicine journals are excluded when the count of citations referencing each of the three SCI-indexed emergency medicine journals is determined. We sought to examine the effect of this on emergency medicine impact factors.
Section snippets
Methods
To approximate the effect on emergency medicine impact factors, we arbitrarily chose calendar-year 1994 and recalculated the impact factors for the three indexed emergency medicine journals recognized by the SCI. We followed the standard SCI methodology, drawing citations from only the 2 preceding years (1992 and 1993), but treating all 3 journals as if they were source journals during this period. We then compared the recalculated impact factors with those published in SCI, for which all three
Results
Recalculation of the 1994 impact factors resulted in a quantitatively and statistically significant increase for two of the three emergency medicine journals (Table).
*Of eight emergency medicineEmpty Cell Journal A Journal B Journal C Citation count SCI 324 88 11 SCI + Study 501 146 17 Difference 177 58 6 Impact factor
SCI .62 .39 .09 Study .96 .64 .14 Difference (95% CI) .34 (.29 to .38) .25 (.17 to .34)
.05 (-.03 to .13)
Discussion
Consistently low and unchanging impact factors for emergency medicine journals are puzzling in light of evidence of academic growth in virtually every other aspect of the field.1, 2, 3, 4, 5 In attempting to determine the reason for this discrepancy, we identified a methodologic bias inherent in the derivation of the impact factor that adversely affects emergency medicine journals.
There are at least two implicit assumptions underlying the putative utility of SCI's impact factors.7 The first is
References (10)
- et al.
Changing status of academic emergency medicine (1991–1996)
Acad Emerg Med
(1997) 1995 Annual Meeting Abstracts
Acad Emerg Med
(1995)1996 Annual Meeting Abstracts
Acad Emerg Med
(1996)- et al.
State of the journal
Ann Emerg Med
(1995) The maturation of emergency medicine
Acad Emerg Med
(1995)
Cited by (59)
Will the controversy about publishing in Spanish or in English be solved on the basis of academic merits?
2010, Revista Colombiana de AnestesiologiaCitations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals
2009, Journal of Clinical EpidemiologyCitation Excerpt :The list of source journals used to compute impact factors contains only a small proportion of all scientific journals [29], which explains why one-third of the articles analyzed in this study were published in journals that did not have an impact factor at the time of their publication. The Institute for Scientific Information has been criticized because some fields are almost entirely excluded from the list of source journals, and for the Anglo-American bias in the selection of source journals [29]. Several limitations of citation counts and impact factors have been identified [6,7,9,17,19].
The Outcome of Manuscripts Submitted to the American Journal of Ophthalmology Between 2002 and 2003
2007, American Journal of OphthalmologyCitation Excerpt :Impact factor is the most recognized term used and also perhaps a misnamed, misleading, and misused measure derived from the citation curve of a journal.5,17–19 The interested reader is directed to numerous articles that debate the impact factor and its usage, usually critical of the assumed objective nature of the number.5,17,20–31 However, other alternatives to this objective measure have not emerged.
A bibliometric analysis of global trends of research productivity in tropical medicine
2006, Acta TropicaCitation Excerpt :These include the focus of a scientific journal as well as language issues. It should be noted that the impact factor has often been criticized as a tool for measuring quality of scientific research (Gallagher and Barnaby, 1998; Gensini and Conti, 1999; Whitehouse, 2002). Yet, thus far, it has not been replaced by any other worldwide-accepted method (Garfield, 1955; Luukkonen, 1990).
- ☆
From the Departments of Emergency Medicine, Medicine, and Epidemiology and Social Medicine, Albert Einstein College of Medicine, Bronx, NY.
- ☆☆
Address for reprints: E John Gallagher, MD, Department of Emergency Medicine, Albert Einstein College of Medicine, Montefiore Medical Center, Bronx, NY 10467, 718-920-7459, Fax 718-798-6084
- ★
Reprint no. 47/1/86809