Assessment of physician performance in Alberta

In common with other provincial medical licensing authorities, the College of Physicians and Surgeons of Alberta seeks to maintain high standards of practice that meet the expectations of society. Its stated mission is to guide the medical profession and protect the public.

The provincial medical licensing authorities have established procedures to assess the small minority of physicians who may have problems with loss of competence or unacceptable professional conduct or against whom allegations of deficient performance have been made. [1–3] These programs are expensive, and their focus on questionable performance does not assist or assure the large majority of physicians without such problems.

In 1995 the College of Physicians and Surgeons of Alberta established the Physician Performance Advisory Committee to establish a process to evaluate physician performance; members were drawn from the 2 Alberta medical schools (the Universities of Alberta and Calgary) and the Alberta Medical Association and included a representative of the public.

In 1998 more than half of the 770 complaints to the College of Physicians and Surgeons of Alberta related to issues of physician-patient communication (Donald E. Chadsey, Deputy Registrar: personal communication, 1999). The Physician Performance Advisory Committee anticipated that a process drawing to the attention of physicians important issues such as physician-patient communication might improve the quality of medical practice and avert many such complaints. There is evidence that this strategy can be effective. For example, the peer review program of the College of Physicians and Surgeons of Ontario showed that the office records of randomly selected physicians tended to improve after defects were pointed out. [4] A British study suggested that the routine practice of doctors can be influenced by feedback from patients. [5] Other studies have reported the feasibility and value of physician performance appraisal by patients, [6–9] peers [10, 11] and hospital nurses. [12, 13]

This paper describes the purpose, development and pilot studies of a program that will regularly assess the performance of all licensed physicians in Alberta. The process may trigger further assessment of a limited number of physicians.

Development of a program to assess physician performance

After its initial work, the Physician Performance Advisory Committee concluded that a performance assessment program should conform to some specific concepts and characteristics (Table 1). The committee selected the program name Physician Achievement Review (PAR) to denote a supportive purpose and the goals of describing professional accomplishment and improving practice.

Extensive discussions within the committee generated 2 complementary inventories of medical practice characteristics: first, 6 broad categories of physician performance attributes - medical knowledge and skills, attitudes and behaviour, professional responsibilities, practice improvement activities, administrative skills and personal health - and second, 5 categories of sources of physician assessment - self, patients, medical peers or colleagues, consulting physicians to whom patients are referred and nonphysician coworkers in health care (e.g., secretaries, nurses and pharmacists). The Physician Performance Advisory Committee used these inventories to construct an assessment grid that listed 90 performance attributes and appropriate raters. The committee then classified the attributes in relation to the members' judgement of good medical care as essential, important, desirable or irrelevant. The performance grid categories were used to develop separate questionnaire instruments reflecting the perspectives of self, patients, medical peers, consultants and nonphysician coworkers respectively (Table 2).

Methods for pilot feasibility study

The goals of the pilot study were to evaluate the psychometric properties and the statistical validity of the questionnaires, to compare self-ratings with ratings given by peers and co-workers, and to examine the influence of raters' familiarity with the physicians being assessed. A sample of approximately 300 physicians was required; 874 physicians were contacted to meet this sample size. The inclusion criteria were registration with the College of Physicians and Surgeons of Alberta for more than 5 years and current practice as a family physician, general practitioner, or specialist in obstetrics and gynecology, general internal medicine, or general pediatrics. Participants were recruited by means of a letter sent by the registrar of the College of Physicians and Surgeons of Alberta to a stratified random sample of physicians.

Results of the pilot study

Implementation of the PAR program

On the basis of a recommendation from the Physician Performance Advisory Committee, the Council of the College of Physicians and Surgeons of Alberta implemented the PAR program for all physicians in February 1999. Some practical aspects of the evolving PAR program are described here.

There are 4700 physicians in Alberta, and it is planned that approximately 20% will be assessed each year over a 5-year cycle. For the assessment, a PAR package containing an explanation of the process and questionnaires will be mailed to each physician. The physician will complete the self-evaluation questionnaire (consisting of 26 questions) and will distribute questionnaires to the following assessors: 25 patients (44 questions), 8 medical colleagues (peers or consultants; 26 questions) and 6 nonphysician health care coworkers (17 questions). The completed questionnaires will be returned to the data processing centre in prepaid envelopes.

The PAR responses will be sent to participating physicians in text and graphic formats. Each physician's performance report will be returned only to that individual and the Physician Performance Committee. Performance data will be classified according to the domain of assessment (as in Table 2) and in relation to reference ranges for peer groups. A guide to interpreting the results will be provided. The PAR report will include the names and phone numbers of contact persons for further discussion about the interpretation of PAR profiles, as well as a list of educational resources.

The confidentiality of individual physician data will be assured by amendments to the Medical Professions Act of Alberta, which were enacted in 1998. [17] The legislation states that a physician shall not be required to participate in the PAR program more than once every 5 years.

Review of the PAR results will be the responsibility of a committee (Physician Performance Committee) accountable to the Council of the College of Physicians and Surgeons of Alberta. This committee will examine analyses of the results and, on the basis of operating experience, will consider modifications to PAR procedures, such as changes in the number and content of questionnaires. This committee will evaluate the value and outcomes of the PAR program. Program monitoring will cover the educational activities reported by physicians in their annual license reapplication to the college, the profile of patients' complaints to the college, and practice changes observed or reported in research studies.

Participation in the PAR program will be required unless a physician can explain to the Physician Performance Committee why some or all of the process is inappropriate for his or her practice. Refusal to participate without reasonable grounds may be regarded as conduct subject to the disciplinary processes of the College of Physicians and Surgeons of Alberta.

The PAR has been developed with the majority of Alberta physicians in mind: those who engage in office and hospital practice with substantial direct patient care. Specialty groups, particularly procedural and diagnostic specialties, have been invited to assist in developing modified questionnaires to assess their performance.

The administrative costs of the PAR program will be approximately $200 per physician (in 1999 dollars). The PAR program will be funded through the operating budget of the College of Physicians and Surgeons of Alberta. The necessary increase in annual licence fees of $40 per year for all Alberta physicians is regarded as a cost inherent to the self-regulation function of the profession.

The primary objective of the PAR will be improvement of practice quality. This is the anticipated result of informing physicians of possible performance deficiencies, as in the model of the quality improvement cycle [14] and the educational model of practice reflection. [18] Education related to the PAR will be voluntary for most physicians. If serious performance deficiencies are identified during review by the Physician Performance Committee, the College of Physicians and Surgeons of Alberta, under its existing authority, may require and direct detailed evaluation, remedial education and subsequent reassessment by methods appropriate to the deficiencies.

The PAR office will maintain an inventory of educational resources for professional and nonprofessional help, for example relating to physician-patient communication or practice management. This inventory will be available to all Alberta physicians, not only those participating in the PAR program in a particular year.

The PAR process will not replace existing procedures by which the College of Physicians and Surgeons of Alberta fulfills its statutory obligations under the Medical Profession Act of Alberta, such as investigation of complaints. Now that the program has been implemented, the threshold criteria for considering further assessment will be determined by the Physician Performance Committee from operating experience and its review of aggregate performance data. The results of the PAR pilot study indicate that for approximately 10% of physicians, further assessment or directed education may be appropriate.

Conclusion

The PAR program has been developed by a partnership of the College of Physicians and Surgeons of Alberta, the 2 Alberta medical schools and members of the medical profession, including 336 physicians who participated in 2 pilot projects.

The program has been designed to be relevant to aspects of medical practice judged to be important for most physicians in Alberta. Most of the reported research on assessing physician performance by questionnaires has used the ratings of single groups of raters, such as patients, peers or coworkers. Our findings suggest that there are practical and technical advantages to the multidimensional perspective provided by the different groups.

The PAR program aims to assist the physicians of Alberta to identify aspects of their circumstances and practice that they might wish to review and perhaps change or improve, for the benefit of their patients and their own professional satisfaction.

Further information about the College of Physicians and Surgeons of Alberta process is available at a Web site under development (www.par-program.org). The intent of this site is to provide information to physicians regarding pertinent continuing medical education.

Dr. Anthony Marini assisted with the psychometric analyses. Members of the Physician Performance Advisory Committee, in addition to the authors, were Lynn Malin (member of the lay public) and Drs. Gordon Arnett, Keith Brownell, Betty Flagler, Susan Low, Wolfgang Schneider and Brynn Whittaker. The committee's work was ably supported by Dr. Bryan Ward and John Swiniarski, assistant registrars of the College of Physicians and Surgeons of Alberta. We thank the registrar of the college, Dr. Larry Ohlhauser, for his support throughout this development project, which was funded by the college.

Competing interests: None declared.

This article has been peer reviewed.

Reprint requests to: Dr. William G. Hall, Health Sciences Centre, University of Calgary, 3030 Hospital Dr. NW, Calgary AB T2N 4N1; fax 403 670-1240