A quantitative ranking of Canada's research output of original human studies for the decade 1989 to 1998
========================================================================================================

* Roy E. Gagnon
* Andrew J. Macnab
* Faith A. Gagnon

## Abstract

**Background:** Since 1987 research articles have been catalogued with the author's affiliation address in the 40 databases of the Medical Literature Analysis and Retrieval System (MEDLARS) of the National Library of Medicine, Bethesda, Md. The present study was conducted to examine the Canadian entries in MEDLARS to interpret past and future trends and to combine the MEDLARS demographic data with data from other sources to rank Canadian research output of human studies both nationally and internationally.

**Methods:** The PubMed Web site of the National Library of Medicine was used to count medical articles archived in MEDLARS and published from Jan. 1, 1989, through Dec. 31, 1998. The articles attributed to Canadian authors were compared by country, province, city, medical school, hospital, article type, journal and medical specialty.

**Results:** During the study period Canadian authors contributed on average 3% (standard deviation [SD] 0.2%) of the worldwide MEDLARS content each year, which translated to a mean of 11 067 (SD 1037) articles per year; 49% were human studies, of which 13% were clinical or controlled trials, and 55% involved people aged 18 years or less. In total, 68% of the articles were by authors affiliated with Canadian medical schools; those affiliated with the University of Toronto accounted for the greatest number (8604), whereas authors affiliated with McGill University had the greatest rate of annual increase in the quantity published (8%). Over one-third (38%) of the articles appeared in Canadian journals. When counted by specialty, 17% of the articles were by authors with clinical specialties, 5% by those with surgical specialties and 3% by those with laboratory specialties.

**Interpretation:** The annual rate of increase in research output for Canada was more than 3 times higher than that seen world wide. Canada is now ranked seventh among countries contributing human studies to MEDLARS. The increase indicates that Canada's medical schools are productive, competitive in making contributions to medical science and are supporting Canadian journals.

As of October 1998 there were over 18 million entries in the 40 databases of the Medical Literature Analysis and Retrieval System (MEDLARS) of the National Library of Medicine, Bethesda, Md. Since 1987 research articles have been catalogued with the author's affiliation address. We wished to examine the Canadian entries to interpret past and future trends and to combine the MEDLARS demographic data with data from other sources to rank Canadian research output of human studies both nationally and internationally.

## Methods

The "Advanced Search" utility of the National Library of Medicine's PubMed Web site ([www.ncbi.nlm.nih.gov/PubMed/medline.html](http://www.ncbi.nlm.nih.gov/PubMed/medline.html)) was used to search MEDLARS. Search terms were accompanied with MEDLINE field codes such as "[AD]" for address, "[DP]" for date published, "[LA]" for language, "[jour]" for journal name, "[MH]" for subject content and "[PT]" for publication type. For example, "human[MH] 1998[DP] anesthesia [MH] journal article [PT] AND (child, preschool[MH] OR aged, 80 and over [MH] AND (1998[DP] OR 1997[DP] OR 1996[DP]) AND (university[AD] OR universite[AD]) AND (ontario[AD] OR ont[AD]) NOT (california[AD] OR united states[AD] OR usa[AD]) NOT (review[PT] OR editorial[PT] OR letter[PT])" was entered to find all recent Canadian entries of human anesthesia studies involving young girls and elderly women that were published as original journal articles (specifically excluding review articles, editorials and letters) and whose authors were affiliated with universities in Ontario having either English or French name spellings regardless of whether the name "Canada" was included in the affiliation address (7 entries found).

Entries with authors from multiple centres were found by grouping place names in the [AD] field code. For example, with "toronto[AD] AND edmonton[AD]" an article of a mammal study was found with 20 authors from 11 US centres, 5 Canadian centres and 1 Swedish centre.

Original human studies categorized as "journal article," "clinical trial" or "randomized controlled trial" were counted. Articles categorized as "review," "review tutorial," "comment," "letter," "editorial," "meta-analysis" or "practice guide" were excluded.

MEDLARS receives data from articles published in 70 countries. We selected 16 countries that, when combined, would account for over 62% (31% United States, 31% others) of the database. To evaluate Canadian human studies further, we examined publications from universities having medical schools. However, these counts would also include human studies from departments such as biology, computing and engineering.

MEDLARS classifies articles by subject content (using medical subject headings [MeSH]) rather than the authors' specialties or disciplines. The specialties we counted were those listed by the Royal College of Physicians and Surgeons of Canada, and the disciplines were those commonly listed in institution directories.

Data were compiled for entries published between Jan. 1, 1989, through to Dec. 31, 1998.

## Results

The following results were extracted from a full report of the study (available from the corresponding author upon request).

During the study period Canadian authors contributed on average 3% (standard deviation [SD] 0.2%) of the worldwide MEDLARS content each year, which translated to a mean of 11 067 (SD 1037) articles per year. Almost all (89%) of Canada's 110 675 publications were original articles, the remainder being mainly review articles or tutorials. Over half (55%) of the original articles were human studies, of which 13% were clinical or controlled trials; 33% of the human studies involved males, 23% females and 44% both males and females. Over half (55%) involved subjects 18 years of age or less. There were almost as many studies involving children aged 6 to 12 years as there were involving middle-aged adults (45-64 years) (13 173 and 13 870 respectively).

Canada's original research output ranked fourth at the beginning of the study period. It began to be exceeded first by France and Italy in 1991 and then by Germany in 1993 (Table 1). Given the current volumes and rates of change, Canada's output will be surpassed by Spain in 2010, Australia in 2014, the Netherlands in 2023 and Switzerland in 2035. Canada's pattern of change in annual productivity most closely resembled that of the Netherlands and Italy (regression analysis), whereas Canada ranked ninth in having a stable pattern of change progressing in equal increments (i.e., annually). Canada's annual rate of increase in productivity (4%) was over 3 times that for MEDLARS as a whole and 1.5 times that for MEDLARS' greatest contributor, the United States.

View this table:
[Table1](http://www.cmaj.ca/content/162/1/37/T1)

Table 1: Distribution of articles of human studies published between Jan. 1, 1989, and Dec. 31, 1998, in MEDLARS, by country 

Among the Canadian universities with a medical school the average output per school was 267 original articles of human studies per year (Table 2). Of the Canadian human studies 28% came from Ontario medical schools and 25% from medical schools in the western provinces. Combined, medical schools in Quebec and the Atlantic provinces accounted for 16% of the human studies. In 1995/96 (the only year for which data were available) each faculty of medicine produced on average 6 articles for every $1 million of research funding expended.

View this table:
[Table2](http://www.cmaj.ca/content/162/1/37/T2)

Table 2: Distribution of Canadian articles of human studies by medical school 

MEDLARS contains articles from more than 3900 journals, of which 97 (2.5%) are Canadian. Of the 57 Canadian journals currently active and indexed, 21 are published in Ottawa, 18 in Toronto, 6 in Montreal, 5 in Hamilton, 2 in Edmonton and 1 each in Vancouver, Calgary, Regina, Halifax and Champaign (United States). Of the 110 675 Canadian articles, 38% were published in Canadian journals. The *Canadian Medical Association Journal* accounted for the greatest number (5441 [13%]) of articles among Canadian journals. This was followed by the *Journal of Rheumatology* (4928 [12%]), the *Canadian Journal of Anaesthesia* (2718 [6%]) and the *Canadian Journal of Physiology and Pharmacology* (2094 [5%]). Among the remainder, 9 journals accounted for 1000 to 2000 articles each.

Authors with surgical specialties contributed 5607 articles (5% of the Canadian total), laboratory specialties 3842 articles (3% of the Canadian total) and clinical specialties 18 644 articles (17% of the Canadian total). Pharmacology affiliates produced the greatest number of articles (4451), followed by pathology (3728), psychiatry (2472), immunology (2011) and medical genetics (1671). Among the disciplines, without regard for affiliation, those labelled "cell" contributed 28 483 articles, followed by "genetics" (23 197), "chemistry" (19 268) and "medicine" (16 020).

To determine the number of articles resulting from multicentre projects, all combinations of Toronto, Montreal, Vancouver, Winnipeg, Edmonton, Calgary and Halifax were tested, but only 5 projects from centres in Toronto and Montreal were found. However, there were 2 shared projects between Canadian and US centres and 1 between Canadian and United Kingdom centres; none of the shared projects were from centres in Japan, France or Germany (the only other countries tested).

During the survey we found 22 instances of imperfectly addressed affiliations (e.g., "McGill University, Montreal USA," or "Dept of Preventive Medicine, University of Toronto, Vancouver") found among 45 913 affiliations sampled. This is an error ratio of 1:2100 (0.05%).

## Interpretation

Research output can be measured by quantity, quality or impact factor (ratio of quality to quantity). Paradoxically, quality has been expressed as the total number of bibliographic citations, a quantity.[1–3] Each method is flawed. Quantities can be inflated by subdividing or substantively duplicating reports. Citation counts can be inflated by self-citation, author bias and counting methodology.[4, 5] Both quantity and quality counts can suffer from the continued citation of retracted articles.6

Measurements of research output have been either praised, deplored or misused.[7–9] They have been used to rank journals, countries, universities, departments and authors, and to justify journal de-selection by university libraries.[10–12] Decisions have had to be made based on survey counts simply because there is too much literature to review for assessment based on merit, even though methods of assessing quality and merit have been developed.13 Is Canada's annual average contribution of 11 067 human and animal studies to MEDLARS relevant? In this regard, we have to trust the eye of the reviewer and the future.[14, 15]

In our analysis we relied on the authors' declaration of affiliation(s) and MEDLARS' classification of "human" studies. We considered all human studies affiliated with nonmedical institutions to be relevant without testing for relevance. Also, despite use of the search routine's Boolean operators, under some combinations of extraction codes, an entry may have inadvertently been counted twice. Affiliation may have referred to the institution holding the principal investigator's funding, the institution providing the facilities, the principal author's employer or the corresponding author's postal address. MEDLARS does not distinguish whether coauthor's affiliations are absent, nor does it categorize affiliations as academic, clinical, institutional or corporate.

Although we found MEDLARS to be useful for extracting demographic data, there are few other annual databases to combine with it to enhance interpretations in the context of Canadian medical research.

Our analysis indicates that Canada's increase in research output during the 10-year study period (4%) has kept pace with the trend seen in other countries and corresponds with the average annual increase in publications among Canadian universities. This increased rate of publication, Canada's seventh place ranking, the average cost and the ratio of articles in Canadian journals versus all journals imply that the country's medical schools are productive, competitive in making contributions to medical science and are supporting Canadian journals.

Competing interests: None declared.

## Acknowledgments

The Canadian Cancer Society (National) will award $1000 for the best paper on a topic in oncology or cancer control written by a student enrolled in an undergraduate medical program in Canada. Essays should be no longer than 3000 - words and will be judged on relevance, originality and scientific merit. The winning paper(s) will be considered for publication in *CMAJ.*

For more information and submission forms contact Ms. Monika Dixon, Junior Administrator, Essay Prize for Oncology or Cancer Control, Canadian Cancer Society, 10 Alcorn Ave., Suite 200, Toronto, ON M4V 3B1; tel 416 961-7223; fax 416 961-4189; mdixon{at}cancer.ca OR visit their Web site ([www.cancer.ca](http://www.cancer.ca)).

## Footnotes

*   This article has been peer reviewed.
    
    Correspondence to: Dr. Andrew J. Macnab, c/o Leetha Semrick, Rm. 2L10, Critical Care Unit, Children's & Women's Health Centre of British Columbia, 4480 Oak St., Vancouver BC V6H 3V4; fax 604 875-2728

## References

1.  1. Garfield E. Which medical journals have the greatest impact? Ann Intern Med 1986;105:313-20.
    
    

2.  2. Garfield E. How can impact factors be improved? BMJ 1996;313:411-3.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTMvNzA1NC80MTEiO3M6NDoiYXRvbSI7czoxOToiL2NtYWovMTYyLzEvMzcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  3. Larson JS, Kershaw R. Rating journals in health care administration by the textbook citation method. Med Care 1993;31(11):1057-61.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-199311000-00007&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8231336&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 

4.  4. Campbell FM. National bias: a comparison of citation practices by health professionals. Bull Med Libr Assoc 1990;78(4):376-82.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2224301&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1990EB82200008&link_type=ISI) 

5.  5. Gallagher EJ, Barnaby DP. Evidence of methodologic bias in the derivation of the Science Citation Index Impact factor. Ann Emerg Med 1998;31(1):83-6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/S0196-0644(98)70286-0&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9437347&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000071363400014&link_type=ISI) 

6.  6. Judd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations to the publications. JAMA 1998;280(3):296-7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.3.296&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9676689&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000074706000034&link_type=ISI) 

7.  7. News Editor. Reviews of UK university research 'have helped to raise standards'. Nature 1997;385:3.
    
    

8.  8. Meenen NM. [The impact factor - A reliable sciento-metric parameter?] [German] Unfallchirurgie 1997;23(4):128-34; discussion 135-6.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9381604&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 

9.  9. Funding cuts put pressure on peer review. Nature 1996;383:567.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8857526&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 

10. 10. Sittig DF, Kaalaas-Sittig J. A quantitative ranking of the Biomedical Informatics serials. Methods Inf Med 1995;34(4):397-400.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7476471&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995RY83200011&link_type=ISI) 

11. 11. De Jong JW, Schaper W. The international rank order of clinical cardiology. Eur Heart J 1996;17(1):35-42.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZWhqIjtzOjU6InJlc2lkIjtzOjc6IjE3LzEvMzUiO3M6NDoiYXRvbSI7czoxOToiL2NtYWovMTYyLzEvMzcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

12. 12. Deurenberg R. Journal deselection in a medical university library by ranking periodicals based on multiple factors. Bull Med Libr Assoc 1993;81(3):316-9.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8374589&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 

13. 13. Taddio A, Pain T, Fassos FF, Boon H, Hersich AL, Einarson TR. Quality of nonstructured and structured abstracts of original research articles in the *British Medical Journal,* the *Canadian Medical Association Journal* and the *Journal of the American Medical Association. CMAJ* 1994;150(10):1611-5.
    
    

14. 14. Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med 1992;67(12):856-9.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1457023&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom) 

15. 15. Ramsey SD, Hillman AL, Renshaw LR, Kimberly JR, Pauly MV, Schwartz JS. How important is the scientific literature in guiding clinical decisions? The case of magnetic resonance imaging. Int J Technol Assess Health Care 1993;9(2):253-62.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8458704&link_type=MED&atom=%2Fcmaj%2F162%2F1%2F37.atom)