The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care
===========================================================================================================================================================

* Robyn Tamblyn
* Allen Huang
* Robert Perreault
* André Jacques
* Denis Roy
* James Hanley
* Peter McLeod
* Réjean Laprise

## Abstract

**Background:** Adverse drug-related events are common in the elderly, and inappropriate prescribing is a preventable risk factor. Our objective was to determine whether inappropriate prescribing could be reduced when primary care physicians had computer-based access to information on all prescriptions dispensed and automated alerts for potential prescribing problems.

**Methods:** We randomly assigned 107 primary care physicians with at least 100 patients aged 66 years and older (total 12 560) to a group receiving computerized decision-making support (CDS) or a control group. Physicians in the CDS group had access to information on current and past prescriptions through a dedicated computer link to the provincial seniors' drug-insurance program. When any of 159 clinically relevant prescribing problems were identified by the CDS software, the physician received an alert that identified the nature of the problem, possible consequences and alternative therapy. The rate of initiation and discontinuation of potentially inappropriate prescriptions was assessed over a 13-month period.

**Results:** In the 2 months before the study, 31.8% of the patients in the CDS group and 33.3% of those in the control group had at least 1 potentially inappropriate prescription. During the study the number of new potentially inappropriate prescriptions per 1000 visits was significantly lower (18%) in the CDS group than in the control group (relative rate [RR] 0.82, 95% confidence interval [CI] 0.69–0.98), but differences between the groups in the rate of discontinuation of potentially inappropriate prescriptions were significant only for therapeutic duplication by the study physician and another physician (RR 1.66, 95% CI 0.99–2.79) and drug interactions caused by prescriptions written by the study physician (RR 2.15, 95% CI 0.98–4.70).

**Interpretation:** Computer-based access to complete drug profiles and alerts about potential prescribing problems reduces the rate of initiation of potentially inappropriate prescriptions but has a more selective effect on the discontinuation of such prescriptions. 

Drug-related adverse events are reported to be the sixth leading cause of death1,2 and contribute to substantial morbidity, particularly in the elderly.2,3,4,5,6,7,8,9 Inappropriate prescribing has been identified as a preventable cause of at least 20% of drug-related adverse events.10,11,12,13,14,15,16 Elderly patients are at greatest risk of receiving inappropriate prescriptions.17 Because primary care physicians write approximately 80% of prescriptions for people 65 years of age and older,18 effective interventions to optimize prescribing in primary care are a priority.

Computerized decision-making support (CDS) for drug management may be an effective method of reducing inappropriate prescribing. Automated surveillance of a patient's drug and disease profile can alert a physician to potentially problematic prescriptions when treatment decisions are being made. There is evidence that CDS in hospital can reduce the incidence of drug-related adverse events,19,20,21,22 improve the cost-effectiveness of drug selection23,24,25,26,27 and optimize drug–dose calculations.28,29,30,31,32

Evaluation of CDS for prescription drug management in primary care settings has been limited.20 One of the challenges in community-based practice is that there is no central pharmacy to track all drugs prescribed. This is a substantial problem because 40% of elderly patients use more than 1 pharmacy, and 70% have more than 1 prescribing physician.18 In this study we assessed whether inappropriate prescribing would be reduced when primary care physicians had access to information on all prescriptions dispensed to their elderly patients.

## Methods

### Context

The study was conducted in Quebec, where a universal health insurance program provides complete coverage of medical and hospital services for all residents, as well as comprehensive drug insurance for the elderly. Beneficiary, medical-service and prescription-claims databases maintained by the Régie de l'assurance maladie du Québec (RAMQ)33 and previously validated34 were used to assemble the eligible study population, provide information on prescriptions dispensed, and evaluate the use of both medical services and drugs before and after the implementation of CDS.

### Study design and participants

To test whether CDS would reduce inappropriate prescribing, we conducted a 13-month cluster-randomized controlled trial between January 1997 and February 1998. Sample size was estimated for the cluster trial35 with a relative reduction in inappropriate prescribing of 30%, type 1 and 2 errors of 1% and 20% respectively and estimates of variation in rates among patients and among physicians.36 The Collège des medicines du Québec used annual licensure-renewal data to identify eligible physicians: general practitioners 30 years of age or older who had practices in Montreal, spent at least 70% of the week in private fee-for-service practice and had a minimum of 100 elderly patients. Letters of invitation and information sessions were used to recruit physicians. To minimize the possibility of contamination, only 1 physician per group practice was included. Differences in characteristics and prescribing habits of participating and non-participating physicians were assessed with the use of non-identifiable data from the Collège and the RAMQ prescription-claims files.

Patients of participating physicians were eligible if they were 66 years of age or older, had been seen on 2 or more occasions by the study physician in the past year, and were living in the community at the start of the study. The RAMQ provided a list of eligible patients to each physician and a total count of patients per practice to the investigators. With the consent of the patient, personal information was provided to the RAMQ and the researchers.

### Randomization and blinding

Physicians were stratified by age (3 categories), sex, language (French, English), location of medical school of graduation (foreign, Canada or the United States) and number of elderly patients (less than 118, 118 or more).

Two months before CDS was implemented, after more than 90% of patients had been recruited, half of the physicians within each stratum were randomly assigned to the CDS group and the other half to the control group. Physicians and patients were not told the specific outcomes of the study but were aware of which group they had been assigned to.

### Basic intervention

Each physician was given a computer, a printer, health-record software and dial-up access to the Internet. The health-record software documented health problems and medications prescribed. For each patient, trained personnel developed a health-problem list by abstracting, coding and entering data from the primary care physician's chart, using a standardized form that documented the 26 health problems related to the targeted drug–disease contraindications, as well as other chronic health problems. Concordance in identification of key target problems between the chief abstractor and the abstraction team was 86.1% (κ = 0.56) in independent audits of a systematic sample of 1138 charts.

### CDS group

Physicians in the CDS group obtained information on each patient by downloading updates of dispensed prescriptions from the RAMQ drug-insurance program. All retail pharmacies have a data link to the RAMQ for online prescription adjudication, which provided a daily update of all prescriptions dispensed for each patient. These data were integrated into the patient's health record and categorized as having been prescribed by the study physician or by another physician. Alerts were instituted to identify 159 clinically relevant prescribing problems in the elderly, a list established previously by expert consensus:37 26 problems were related to drug–disease contraindications, 23 to drug interactions, 17 to drug–age contraindications, 3 to duration of therapy and 90 to therapeutic duplication. The alerts appeared when the electronic chart was opened, when prescription-record updates were downloaded from the RAMQ, and when current health problems and prescriptions were recorded by the physician in the chart. Each alert message identified the nature of the problem and possible consequences and suggested alternative therapy in accordance with the expert consensus.

### Outcomes

The primary outcome measures were initiation and discontinuation rates of the 159 prescription-related problems. Records of prescriptions dispensed and medical visits (from the RAMQ prescription-claims and medical-service-claims files and from the abstracted office-chart data) were used to assess outcomes to ensure that the same measures were used for the 2 groups of physicians. Discontinuation rates were calculated for patients who had been given at least 1 inappropriate prescription in the 2 months before the study began. An inappropriate prescription was considered to have been discontinued by the study physician if it had not been refilled within 2 months after the prescription end date and if there had been a visit to the study physician before or during the month of the prescription end date. Initiation rates were calculated for the remaining patients from the prescriptions written by the study physician for 1 or more of the 159 prescription-related problems during the 13-month study period. The denominator for each rate, measured by medical-service claims, was the number of patient visits to the study physician during the study period; this number provided an accurate assessment of differences in opportunity to initiate or discontinue inappropriate prescriptions. Follow-up was terminated after an inappropriate prescription had been initiated or discontinued. Secondary outcomes were initiation and discontinuation rates by type of prescribing problem and discontinuation rates by source of prescription.

### Analysis

Descriptive statistics were used to summarize the characteristics of the physicians and patients in the 2 groups. The association between the weekly frequency of prescription downloads and the number of weeks of computer problems was estimated with Pearson correlation. Poisson regression, within the framework of a generalized estimating equation, was used to determine if there were differences between the 2 groups of physicians in the rates of initiation and discontinuation of inappropriate prescriptions, based on an intention-to-treat analysis.38,39 The patient was the unit of analysis. Physicians were identified as the clustering factor within which rates were examined, and an exchangeable correlation structure was used to take into account the dependence of observations for patients of the same physician. Empirical standard errors were used to take into account the overdispersion in estimated rates.

## Results

Of the 440 eligible physicians, 127 (28.9%) agreed to participate, and the first 107 were included in the study (Fig. 1). Participating physicians were slightly younger than those who did not participate (mean age 46.5 v. 49.4 years). However, participating and nonparticipating physicians were similar in the average number of prescriptions per elderly patient (35.6 v. 33.8) and the prevalence of inappropriate prescribing (18.9% v. 18.8%) in the 18 months before the study start date. There were no differences in characteristics between the CDS and control groups (Table 1).

View this table:
[Table1](http://www.cmaj.ca/content/169/6/549/T1)

Table 1. 


![Figure1](http://www.cmaj.ca/https://www.cmaj.ca/content/cmaj/169/6/549/F1.medium.gif)

[Figure1](http://www.cmaj.ca/content/169/6/549/F1)

**Fig. 1: Selection and assignment of study population. List of Montreal general practitioners provided by the Collège des médecins du Québec. Random assignment was within strata defined by physician age (34–44, 45–48, 49–68 years), language (French, English), sex, number of elderly patients (< 118, ≥ 118) and location of medical school of graduation (foreign, Canada or the United States). CDS = computerized decision-making support.** 

Of the 20 109 eligible patients, 12 560 (62.4%) agreed to participate. Those in the CDS group were more likely than those in the control group to be men, to have made fewer visits to their primary care physician and to have received fewer prescriptions from their primary care physician (Table 1).

At the beginning of the study, there was at least 1 prescribing problem for 33.3% of the patients in the control group and 31.8% of those in the CDS group (Table 2). For 20.4% and 18.8%, respectively, the problems were attributable to a study physician, for 3.3% and 3.2% they were attributable to a study physician plus another physician, and for 8.3% and 9.1% they were attributable to another physician. In both groups, drug–disease contraindications were the most common prescribing problems, followed by drug–age contraindications and excessive duration of therapy (Table 2).

View this table:
[Table2](http://www.cmaj.ca/content/169/6/549/T2)

Table 2. 

Two unforeseen factors influenced the effectiveness of the CDS. First, copayments for prescription drugs were increased when the study began, which resulted in a 9% reduction in prescription drug use by the elderly.40 Second, 22% of the physicians experienced frequent hardware or software failure in the early months of the study; the proportion declined to 4% by month 6. Physicians in the CDS group downloaded prescription information in 81% of the study weeks; however, those who had more computer problems downloaded information less often (*r* = –0.31).

During the study, the rate of initiation of an inappropriate prescription was significantly lower (18%) in the CDS group than in the control group (Table 3). This trend was evident for drug–disease contraindications, drug–age contraindications, excessive duration of therapy and therapeutic duplication and was significant for drug–age contraindications and excessive duration of therapy.

View this table:
[Table3](http://www.cmaj.ca/content/169/6/549/T3)

Table 3. 

CDS had no significant impact on the discontinuation of pre-existing inappropriate prescriptions (Table 4). Although more patients in the CDS group than in the control group had all inappropriate prescriptions discontinued (47.5% v. 44.5%; or 35.5 v. 32.1 per 1000 visits; relative rate [RR] 1.14; 95% confidence interval [CI] 0.98–1.33), the 14% difference was not statistically significant. The only substantially higher discontinuation rate for a specific prescribing problem was for drug interactions: 68.6 v. 51.5 per 1000 visits in the CDS and control groups respectively.

View this table:
[Table4](http://www.cmaj.ca/content/169/6/549/T4)

Table 4. 

Physicians in the CDS group were able to identify excessive duration of therapy, therapeutic duplication and drug interaction resulting from more than one source of prescribing for the same patient. Most of the therapeutic duplications and drug interactions occurred because prescriptions were written by both the study physician and another physician or another physician alone (Table 5). Discontinuation rates in the CDS group were systematically higher for problems created by the combination of prescriptions from study physicians and other physicians than for the other types of prescription problems. An exception was with drug interactions: the relative difference in discontinuation rates between CDS and control physicians was highest for problematic prescriptions written by the study physician, followed by problematic prescriptions written by both the study physician and another physician.

View this table:
[Table5](http://www.cmaj.ca/content/169/6/549/T5)

Table 5. 

Adjusting for patient characteristics (Table 1) did not modify differences in initiation and discontinuation rates between the CDS and control groups. However, a physician's previous computer experience influenced the effectiveness of CDS. Among experienced computer users the rate of initiation of inappropriate prescriptions was 30% lower in the CDS group than in the control group (RR 0.70, 95% CI 0.55–0.89). Among the computer beginners the rate of initiation of inappropriate prescriptions was virtually identical in the 2 groups (RR 1.03, 95% CI 0.82–1.29). The same trend was evident for discontinuation rates (RR for experienced users 1.17 and for beginners 0.93), but this apparent modification of the effectiveness of CDS by computer experience was not significant (interaction term: study group*computer experience, *p* = 0.32).

## Interpretation

This study illustrated the magnitude of the challenge of coordinating health care for elderly patients in an urban setting. Primary care physicians provided only half of all medical services to their elderly patients, who, on average, received prescriptions from at least 3 other physicians and filled those prescriptions at several pharmacies. We addressed the problem of incomplete information on current drug use by using existing prescription-claims information to provide a complete drug profile for each patient. This was a lower-cost solution than using pharmacy-information networks41,42 or smart cards.43

The study also addressed one of the chief criticisms of software screening for drug interactions: clinical relevance.44 We limited alerts to interactions judged by a consensus panel to produce clinically important adverse effects, and we expanded surveillance to include clinically relevant drug–disease contraindications, drug–age contraindications, excessive duration of therapy and therapeutic duplication.37 The alert system was limited, however, by the absence of treatment indications (needed to assess prescription appropriateness) and the absence of weight, height and data on renal function (needed to assess dosage appropriateness). Further, because lower levels of evidence are used to identify potentially problematic prescriptions, the effect of reducing inappropriate prescribing on health outcome remains unknown.

The selectively greater impact of CDS on the initiation of inappropriate prescriptions than on the discontinuation of existing ones could be the result of inaccurate measurement of discontinuation or type 1 errors from multiple comparisons. However, the same pattern was observed in a drug review trial,45 in which physicians were reluctant to stop drug therapy, even when they agreed with the consulting pharmacist's recommendation, because of concerns for patient resistance or discomfort in discontinuing therapy prescribed by another physician. Physicians in the CDS group expressed similar concerns, particularly in relation to drugs prescribed by other physicians. As with a Dutch study,46 we found that the perception of responsibility for patients' treatment varied among the physicians. This lack of clarity in responsibility likely had an impact on the action taken when physicians identified problematic prescriptions.

Poor technical performance is a known deterrent to the use of computer-based systems.47,48,49 Hardware and software failures reduced the frequency of computer use and likely the potential benefits of the CDS. An extensive infrastructure was required to resolve numerous technical problems with the computers and local patient databases. This “heavy client model” is not a viable solution for community-based computer networks. Handheld “personal digital assistants” and wireless technologies, coupled with architectures that provide centralized services for applications and data,50 will provide community-based physicians with less labour-intensive technologic solutions in patient care.

Future research should assess the role of more robust information technologies in primary care, as well as the impact of inappropriate prescriptions on health outcomes.

## Footnotes

*   This article has been peer reviewed.
    
    *Contributors:* Dr. Tamblyn, as principal investigator and author, was involved in all aspects of the study and article preparation. Drs. Huang, Perreault, Jacques, Roy and Hanley were involved in the study's conception and design and revised the article for important intellectual content. In addition, Dr. Hanley was involved in the analysis and interpretation of the data. Dr. McLeod was involved in the study's conception and design. Dr. Laprise was involved in the acquisition, analysis and interpretation of the data. All authors approved the final version of the article.
    
    *Acknowledgements:* Funding was provided by the Fonds de recherche en santé du Québec, the Fond d'autoroute à l'information, the Medical Research Council, the National Health Research and Development Program and Clinidata Inc. In addition, Dr. Tamblyn was supported as a health scholar by the National Health Research and Development Program. This study was made possible by support provided by the Régie de l'assurance maladie du Québec, which developed the computerized interface for the drug insurance-claims database of the seniors drug-insurance program, and by Clinidata, which developed the software to record disease and drug profiles and to conduct automated surveillance for investigator-defined prescribing problems. Catherine Marquis, Jimmy Fragos and Teresa Moraga provided expert assistance in study coordination, implementation and analysis.
    
    *Competing interests:* None declared.

## References

1.  1. Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998;279:1200-5.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.279.15.1200&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9555760&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000072969400034&link_type=ISI) 

2.  2. Hallas J, Harvald B, Gram LF, Grodum E, Brosen K, Haghfelt T. Drug related hospital admissions: the role of definitions and intensity of data collection, and the possibility of prevention. J Intern Med 1990;228:83-90.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1111/j.1365-2796.1990.tb00199.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2394974&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1990DU68600003&link_type=ISI) 

3.  3. Colt HG, Shapiro AP. Drug-induced illness as a cause for admission to a community hospital. J Am Geriatr Soc 1989;37:323-6.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2921453&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1989T832300005&link_type=ISI) 

4.  4. Ives TJ, Bentz EJ, Gwyther RE. Drug-related admissions to a family medicine inpatient service. Arch Intern Med 1987;147:1117-20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/archinte.1987.00370060113019&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3592877&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1987H624200017&link_type=ISI) 

5.  5. Jha A, Kuperman GJ, Rittenberg E, Teich JM, Bates DW. Identifying hospital admissions due to adverse drug events using a computer-based monitor. Pharmacoepidemiol Drug Saf 2001;10:113-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1002/pds.568&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11499849&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000169594800004&link_type=ISI) 

6.  6. Chan M, Nicklason F, Vial J. Adverse drug events as a cause of hospital admissions in the elderly. Intern Med J 2001;31:199-205.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1046/j.1445-5994.2001.00044.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11456032&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000169751300002&link_type=ISI) 

7.  7. Cooper J. Adverse drug reaction-related hospitalizations of nursing facility patients: a 4-year study. South Med J 1999;92:485-90.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00007611-199905000-00007&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10342894&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000080354100007&link_type=ISI) 

8.  8. Raschetti R, Morgutti M, Menniti-Ippolito F, Belisari A, Rossignoli A, Longhini P, et al. Suspected adverse drug events requiring emergency department visits or hospital admissions. Eur J Clin Pharmacol 1999;54:959-63.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/s002280050582&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10192758&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000078997200011&link_type=ISI) 

9.  9. Stanton L, Peterson G, Rumble R, Cooper G, Polack A. Drug-related admissions to an Australian hospital. J Clin Pharm Ther 1994;19:341-7.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7876364&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994PW14900003&link_type=ISI) 

10. 10. Hallas J, Worm J, Beck-Nielsen J, Gram LF, Grodum E, Damsbo N, et al. Drug related events and drug utilization in patients admitted to a geriatric hospital department. Dan Med Bull 1991;38:417-20.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=1802630&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 

11. 11. Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med 1993;8:289-94.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1007/BF02600138&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8320571&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1993LG74300001&link_type=ISI) 

12. 12. Leape LL, Bates DW, Cullen DJ, Cooper J, Demonaco HJ, Gallivan T*,* et al. Systems analysis of adverse drug events. JAMA 1995;274:35-43.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1995.03530010049034&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7791256&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995RG23300028&link_type=ISI) 

13. 13. Bates DW, Cullen DJ, Laird N, Peterson JA, Small SD, Servi D, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995;274:29-34.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1995.03530010043033&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7791255&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995RG23300027&link_type=ISI) 

14. 14. Lindley CM, Tully MP, Paramsothy V, Tallis RC. Inappropriate medication is a major cause of adverse drug reactions in elderly patients. Age Ageing 1992; 21: 294-300.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiYWdlaW5nIjtzOjU6InJlc2lkIjtzOjg6IjIxLzQvMjk0IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE2OS82LzU0OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

15. 15. Schmader KE, Hanlon JT, Landsman PB, Samsa GP, Lewis IK, Weinberger M. Inappropriate prescribing and health outcomes in elderly veteran outpatients. Ann Pharmacother 1997;31:529-33.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9161643&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WY24400001&link_type=ISI) 

16. 16. Lesar TS, Briceland L, Stein DS. Factors related to errors in medication prescribing. JAMA 1997;277:312-7.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.1997.03540280050033&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9002494&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WC45800031&link_type=ISI) 

17. 17. Ferguson JA. Patient age as a factor in drug prescribing practices. Can J Aging 1990; 9:278-95.
    
    

18. 18. Tamblyn RM, McLeod PJ, Abrahamowicz M, Laprise R. Do too many cooks spoil the broth? Multiple physician involvement in medical management and inappropriate prescribing in the elderly. CMAJ 1996;154:1177-84.
    
    [Abstract](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTU0LzgvMTE3NyI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xNjkvNi81NDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

19. 19. Bates DW, Leape L, Cullen DJ, Laird N, Peterson LA, Teich JM, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311-6.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.15.1311&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9794308&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076472500021&link_type=ISI) 

20. 20. Hunt DL, Haynes B, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes. A systematic review. JAMA 1998;280:1339-46.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.15.1339&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9794315&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076472500028&link_type=ISI) 

21. 21. Raschke RA, Gollihare B, Wunderlich TA, Guidry JR, Leibowitz AI, Peirce JC, et al. A computer alert system to prevent injury from adverse drug events. Development and evaluation in a community teaching hospital. JAMA 1998;280:1317-20.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.15.1317&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9794309&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000076472500022&link_type=ISI) 

22. 22. Bates DW. Using information technology to reduce rates of medication errors in hospitals. BMJ 2000;320:788-91.
    
    [FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjAvNzIzNy83ODgiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTY5LzYvNTQ5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

23. 23. Pestotnik SL, Classen DC, Evans S, Burke JP. Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes. Ann Intern Med 1996;124:884-90.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.7326/0003-4819-124-10-199605150-00004&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8610917&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1996UJ80100004&link_type=ISI) 

24. 24. Hershey CO, Porter DK, Breslau D, Cohen DI. Influence of simple computerized feedback on prescription charges in an ambulatory clinic. Med Care 1986; 24:472-81.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-198606000-00002&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3520177&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1986C811700001&link_type=ISI) 

25. 25. Gehlbach SH, Wilkinson WE, Hammond WE, Clapp NE, Finn AL, Taylor WJ, et al. Improving drug prescribing in a primary care practice. Med Care 1984; 22:193-201.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1097/00005650-198403000-00002&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=6700282&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1984SG35400002&link_type=ISI) 

26. 26. Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med 1997;12:672-8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1046/j.1525-1497.1997.07140.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9383135&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997YF24600003&link_type=ISI) 

27. 27. Evans RS, Pestotnik SL, Classen DC, Clemmer TP, Weaver LK, Orme JF*,* et al. A computer-assisted management program for antibiotics and other anti-infective agents. N Engl J Med 1998;338:232-8.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1056/NEJM199801223380406&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9435330&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000071594100006&link_type=ISI) 

28. 28. Poller L, Wright D, Rowlands M. Prospective comparative study of computer programs used for management of warfarin. J Clin Pathol 1993;46:299-303.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiamNsaW5wYXRoIjtzOjU6InJlc2lkIjtzOjg6IjQ2LzQvMjk5IjtzOjQ6ImF0b20iO3M6MjA6Ii9jbWFqLzE2OS82LzU0OS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

29. 29. Casner PR, Reilly R, Ho H. A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing. Clin Pharmacol Ther 1993;53:684-90.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8513660&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1993LJ93800010&link_type=ISI) 

30. 30. Mungall DR, Anbe D, Forrester PL, Luoma T, Genovese R, Mahan J, et al. A prospective randomized comparison of the accuracy of computer-assisted versus GUSTO nomogram-directed heparin therapy. Clin Pharmacol Ther 1994;55:591-6.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=8181203&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1994NM88000014&link_type=ISI) 

31. 31. Walton R. Computer support for determining drug dose: systematic review and meta-analysis. BMJ 1999;318:984-90.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTgvNzE4OS85ODQiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTY5LzYvNTQ5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

32. 32. Fitzmaurice DA, Hobbs FD, Delaney BC, Wilson S, McManus R. Review of computerized decision support systems for oral anticoagulant management. Br J Haematol 1998;102:907-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1046/j.1365-2141.1998.00858.x&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9734638&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000075793300005&link_type=ISI) 

33. 33. Régie de l'assurance-maladie du Québec. Statistiques annuelles. Quebec: The Régie; 1995. p. 46-8.
    
    

34. 34. Tamblyn RM, Lavoie G, Petrella L, Monette J. The use of prescription claims databases in pharmacoepidemiological research: the accuracy and comprehensiveness of the prescription claims database in Quebec. J Clin Epidemiol 1995; 48:999-1009.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1016/0895-4356(94)00234-H&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7775999&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1995RE38200002&link_type=ISI) 

35. 35. Hsieh FY. Sample size formulae for intervention studies with the cluster as unit of randomization. Stat Med 1988;8:1195-201.
    
    

36. 36. Tamblyn R, Abrahamowicz M, Brailovsky C, Grand'Maison P, Lescop J, Norcini J, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA 1998;280:989-96.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.280.11.989&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=9749481&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000075891400032&link_type=ISI) 

37. 37. McLeod PJ, Huang AR, Tamblyn RM, Gayton DC. Defining inappropriate practices in prescribing for elderly people: a national consensus panel. CMAJ 1997;156:385-91.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxNTYvMy8zODUiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTY5LzYvNTQ5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

38. 38. Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika 1986;73:13-22.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiYmlvbWV0IjtzOjU6InJlc2lkIjtzOjc6IjczLzEvMTMiO3M6NDoiYXRvbSI7czoyMDoiL2NtYWovMTY5LzYvNTQ5LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

39. 39. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics 1986;42:121-30.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.2307/2531248&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=3719049&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1986C904700010&link_type=ISI) 

40. 40. Tamblyn R, Laprise R, Hanley JA, Abrahamowicz M, Scott S, Mayo N*,* et al. Adverse events associated with prescription drug cost-sharing among poor and elderly persons. JAMA 2001;285:421-9.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1001/jama.285.4.421&link_type=DOI) 
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11242426&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000166506000029&link_type=ISI) 

41. 41. Demkjaer K, Johansen I, Bernstein K. Third generation electronic pharmacy communication. Recommendations based on ten years' experience. Stud Health Technol Inform 1999;68:278-82.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=10724888&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 

42. 42. Papshev D, Peterson AM. Electronic prescribing in ambulatory practice: promises, pitfalls, and potential solutions. Am J Managed Care 2001;7:725-36.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=11464430&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=000169784600006&link_type=ISI) 

43. 43. Auber BA, Hamel G. Adoption of smart cards in the medical sector: the Canadian experience. Soc Sci Med 2001;53:879-94.
    
    

44. 44. Soumerai SB, Lipton HL. Computer-based drug-utilization review — risk, benefit, or boondoggle? *N Engl J Med* 19950;322:1641-4.
    
    

45. 45. Kroenke K, Pinholt EM. Reducing polypharmacy in the elderly. A controlled trial of physician feedback. J Am Geriatr Soc 1990;38:31-6.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=2404053&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1990CJ86900006&link_type=ISI) 

46. 46. Hulscher ME, van Drenth BB, Mokkink HG, van der Wouden JC, Grol RP. Barriers to preventive care in general practice: the role of organizational and attitudinal factors. Br J Gen Pract 1997;47:711-4.
    
    [Abstract/FREE Full Text](http://www.cmaj.ca/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNDcvNDI0LzcxMSI7czo0OiJhdG9tIjtzOjIwOiIvY21hai8xNjkvNi81NDkuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

47. 47. Chase CR, Ashikaga T, Mazuzan JE Jr. Measurement of user performance and attitudes assists the initial design of a computer user display and orientation method. J Clin Monit 1994;10:251-163.
    
    [PubMed](http://www.cmaj.ca/lookup/external-ref?access_num=7931456&link_type=MED&atom=%2Fcmaj%2F169%2F6%2F549.atom) 

48. 48. Kohlisch O, Kuhmann W. System response time and readiness for task execution — the optimum duration of inter-task delays. Ergonomics 1997;40:265-80.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1080/001401397188143&link_type=DOI) 
    
    [Web of Science](http://www.cmaj.ca/lookup/external-ref?access_num=A1997WQ12300002&link_type=ISI) 

49. 49. Barber RE, Lucas HC Jr. System response time, operator productivity, and job satisfaction. Commun ACM 1983;26:972-86.
    
    [CrossRef](http://www.cmaj.ca/lookup/external-ref?access_num=10.1145/182.358464&link_type=DOI) 

50. 50. *Scaling the N-Tier architecture: Solaris infrastructure products and architecture*. Software white papers*.* Santa Clara (CA): Sun Microsystems, Inc; 2002. p. 1–16. Available: [www.sun.com/software/whitepapers/wp-ntier](http://www.sun.com/software/whitepapers/wp-ntier) (accessed 2003 Mar 23).