Christopher Carpenter suggests another approach to teaching the kappa statistic: giving real data to the students and having them do the requisite calculations. Our approach takes them through the principles of the calculations, step by step. Calculating the kappa score from real data would be an excellent subsequent step for the most enthusiastic students. We encourage readers to consult the teachers' version of our article on the kappa score,1 which has interactive components and data that may be helpful in understanding the concept of kappa.
Regarding the interpretation of kappa scores, Table 1 in both the teachers'1 and learners'2 versions of our article was based on a text by Sackett and colleagues3 and not, as Carpenter correctly points out, the article by Maclure and associates.4 Carpenter is also correct in noting that several different versions of this table are available in the literature. All have 3 basic categories: poor agreement, fair to good agreement, and very good to excellent agreement. In our view, further differentiating within these groups adds little to the practical clinical discussion.
Although we have never seen it in real life, David Juurlink and Allan Detsky correctly state that kappa can theoretically be less than 0 when agreement is poorer than chance. This is most likely to occur when both observers call almost every observation positive or almost every observation negative. In these circumstances, chance agreement would be close to zero and at times could be negative; determining chance-independent agreement (the phi statistic) may represent a better approach.5
Michael Allan asks about chance-corrected agreement when outcomes are categorical or continuous. One useful approach to this problem is the “weighted kappa,” which gives maximal credit for full agreement, partial credit for partial agreement and no credit when disagreement is extreme. For example, in the case of ventilation- perfusion scans for the evaluation of pulmonary embolus, if both people reading a scan interpret the test result as normal (or both say there is intermediate or high probability of embolus), they get full credit for their agreement (weight of 1.0). If one reads the result as normal and the other as high probability, they get no credit (weight of 0). If one assessor classifies the scan as low probability and the other calls the same scan high probability or normal, they get partial credit (a weight of 0.75, for instance). Readers can find a more in-depth explanation and the details of how to calculate weighted kappa from the Web site of MedCalc Software (Mariakerke, Belgium; www.MedCalc.org).
Footnotes
-
Competing interests: None declared.