I would like to thank CMAJ and the Evidence-Based Medicine Teaching Tips Working Group for the teaching tips series, which is wonderfully useful to those of us who are teaching these basic concepts to residents and other physicians. Part 3 in the series, discussing the kappa statistic,1 contained a couple of points to which I would like to contribute, on the basis of my own teaching experiences.
Whereas Thomas McGinn and associates1 suggest that students construct 2 х 2 tables and calculate kappa from successively higher rates of positive calls (see tip 3 in the article), I have instead given students the raw data from actual small studies (with fewer than 25 subjects) and then asked them to construct the 2 х 2 table and calculate actual agreement and chance agreement using the multiplication rule.2 The multiplication rule can be used to calculate joint probability if 2 different events are independent of one another. Most situations that consumers of the medical literature will encounter involve analyzing the numbers provided in various data forms and then determining whether the level of agreement is both acceptable and consistent with the data presented. Rarely will readers need to assign a level of agreement and calculate the kappa statistic. Therefore, the method described here might be valuable as another means to calculate chance agreement and kappa on the basis of more realistic values.
For example, our institution recently implemented the emergency severity index3 (ESI) for nursing triage. Given a random sample of 25 patients from month 1 after implementation and using the nurse administrator's ESI score as the second assessment, I asked residents to compare high-risk and lower-risk triage scores between the triage nurse and the nurse administrator. The resulting 2 х 2 table is completed as shown in Fig. 1, and calculation of chance agreement proceeds as follows:
kappa = [(observed agreement – expected agreement)/(1 – expected agreement)]
High-risk assessments by nurse administrator: 11/25 = 0.44
High-risk assessments by triage nurse: 10/25 = 0.40
Lower-risk assessments by nurse administrator: 14/25 = 0.56
Lower-risk assessments by triage nurse: 15/25 = 0.60
Observed agreement = (9 + 13)/25 = 0.88
Expected agreement = (chance of high-risk assessment) + (chance of lower-risk assessment)
Chance of high-risk assessment = 0.44 х 0.40 = 0.176
Chance of lower-risk assessment = 0.56 х 0.60 = 0.336
Expected agreement by chance alone = 0.176 + 0.336 = 0.512
kappa = (0.88 – 0.512)/(1 – 0.512) = 0.368/0.488 = 0.75
Table 1 in both the teachers'1 and learners'4 versions of this article references Maclure and Willett5 as a source of the qualitative classification of kappa. My own review of that paper did not reveal any attempt to qualitatively assess kappa, but at least 3 other sources have done so.6,7,8 In my experience the most widely used classification for kappa is the last of these,8 which proposed the guidelines for interpreting kappa values as outlined in Table 1 in this letter.