Tips for learners of evidence-based medicine: 3. Measures of observer variability (kappa statistic)

First, I would like to thank the Canadian Medical Association Journal and the Evidence-Based Teaching Tips Working Group for a wonderfully useful series for those of us teaching basic concepts to residents and other physicians. Article 3 in the series, "Measures of Observer Variability (Kappa Statistics)"(1) contained a couple of points to which I hope to contribute based upon my experiences teaching at various residency Journal Clubs and lecture series.

1) Whereas the Working Group had students construct the 2 x 2 tables and calculate Kappa from successively higher rates of positive calls (Teaching Tip 3), I have instead given students the raw data from actual small studies (N <25) then asked them to construct the 2 x 2 table and calculate first actual agreement and then chance agreement using the multiplication rule(2). One is able to utilize the multiplication rule to calculate joint probability if two different events are independent of one another. Most situations that consumers of the medical literature will encounter will involve analyzing the numbers provided in various research data forms and determining whether the level of agreement is both acceptable and consistent with the data presented. Rarely will it involve assigning a level of agreement and asking the reader to calculate a Kappa. Therefore, I feel the method described here might make a valuable Teaching Tip 4 as another means to calculate chance agreement and Kappa using more realistic values.

For example, our institution recently implemented the Emergency Severity Index(3) (ESI) for nursing triage. Given a random sampling of 25 patients from Month 1 using the nurse administrator’s ESI score as the second assessment, I had the residents compare Triage scores 1 or 2 with 3, 4, or 5 between the two nurses. The resulting 2 x 2 table and calculation of chance agreement were completed as demonstrated below:

Your 2 x 2 Table should look like this:

Kappa = [(Observed - Expected agreement) / (1 - Expected agreement)]

Nurse Admin said high risk 11/25 = 0.44
Triage said high risk 10/25 = 0.40

Nurse Admin said lower risk 14/25 = 0.56
Triage said lower risk 15/25 = 0.60

Observed agreement = (9 + 13) / 25 = 0.88

Expected agreement = [Chance high risk] + [Chance lower risk]

Chance high risk = (0.44) * (0.40) = 0.176
Chance lower risk = (0.56) * (0.60) = 0.336

So Expected agreement by chance alone = 0.176 + 0.336 = 0.512

Kappa = (0.88 - 0.512) / (1 - 0.512) = 0.368/0.488 = 0.75

2) Table 1 in both the Tips for Teachers and the Tips for Learners papers reference Maclure’s work(4)on κ as a source of the qualitative classification of κ. My review of this paper did not reveal any attempt to qualitatively assess κ, but at least three sources have done so including Altman,(5) Fleiss,(6) and Byrt.(7) In my experience the most widely utilized description for κ is the latter who proposed the following guidelines for interpreting kappa values(7)

0.93-1.0 Excellent Agreement
0.81-0.92 Very good agreement
0.61-0.80 Good agreement
0.41-0.60 Fair Agreement
0.21-0.40 Slight Agreement
0.01-0.20 Poor Agreement
≤ 0 No Agreement

References

1) McGinn T, Wyer PC, Newman TB, Keitz S, Leipzig R, Guyatt G. Tips for teachers of evidence-based medicine: 3. Understanding and calculating kappa. CMAJ 2004; 171: Online 1-8.
2) Dawson B, Trapp RG. Basic and Clinical Biostatistics, 3rd Edition. New York: McGraw-Hill, 2001, 66-67, 115-117.
3) Wuerz RC, Milne LW, Eitel DR, Travers D, Gilboy N. Reliability and validity of a new five-level triage instrument. Acad Emerg Med 2000; 7: 236-242.
4) Maclure M, Willett WC. Misinterpretation and misuse of the kappa statistic. Am J Epidemiol 1987; 126: 161-169.
5) Altman DG. Practical Statistics for Medical Students. London: Chapman and Hall, 1991.
6) Fleiss JL. Statistical Methods for Rates and Proportions, 2nd Edition. New York: John Wilen and Sons, 1981: 218.
7) Byrt T. How good is that agreement? Epidemiology 1996; 7: 561.

Christopher R. Carpenter, MD
Assistant Professor
Division of Emergency Medicine
Washington University School of Medicine in St. Louis
St. Louis, MO