Validation and diagnostic accuracy of predictive curves for age-associated longitudinal cognitive decline in older adults

Patrick J. Bernier; Christian Gourdeau; Pierre-Hugues Carmichael; Jean-Pierre Beauchemin; René Verreault; Rémi W. Bouchard; Edeltraut Kröger; Robert Laforce

doi:10.1503/cmaj.160792

Article Figures & Tables

Figures

Tables

Download figure
Open in new tab
Download powerpoint
Figure 1:
The cognitive charts. Each solid parallel line represents a percentile: 99th percentile (blue), 85th percentile (light brown), 65th percentile (turquoise), 43rd percentile (purple), 22nd percentile (green) and 10th percentile (red). These colours were chosen from colour palettes for colour-blindness. The grey zone at the bottom represents the cut-off zone. Points on the chart at baseline that fall within the cut-off zone (or any single point within that zone) or, subsequently, that represent a decline greater than the allotted width of 1 percentile zone from the initial measurements, indicate potential cognitive problems. Note: QuoCo = cognitive quotient, S_A = standardized age.
Download figure
Open in new tab
Download powerpoint
Figure 2:
Receiver operating characteristic curve for the MMSE and diagnostic accuracy of QuoCo. These plots are the estimated ROC curves for the MMSE in predicting cognitive status for participants in the databases of CSHA and NACC. The dashed curve is a 95% bootstrap CI for the ROC curve.27 In each plot, A represents diagnostic performance with 95% CI for the QuoCo and B shows the diagnostic performance of the MMSE cut-off (< 24). Lines C and D are the positive and negative likelihood ratios, respectively. The area defined by B–C–D illustrates where any test would be more suitable for confirming the absence of disease than the MMSE cut-off score (see Biggerstaff26), and is the region where the QuoCo lies. Altogether, the graphs show that the QuoCo algorithm is better than the MMSE cut-off for confirming absence of disease, a highly desirable characteristic for a screening test. However, the slightly better overall diagnostic accuracy of the MMSE cut points over the cognitive curves on the graphs is likely related to selection bias (i.e., the CSHA used a cut-off for participant selection). Note: CI = confidence interval, CSHA = Canadian Study of Health and Aging,22 MMSE = Mini-Mental State Examination, NACC = National Alzheimer’s Coordinating Center’s Uniform Data Set, QuoCo = cognitive quotient, ROC = receiver operating characteristic.
Download figure
Open in new tab
Download powerpoint
Figure 3:
Variations in sensitivity and specificity, by population subgroups. Interpretation of MMSE classification (QuoCo v. Cut-off) was accomplished in 3 separate ways: proximity to the external frame, which shows better performance; SE/SP equilibrium, which is most centred as shown in the graph; and stability of performances in subgroups as shown by the regularity of the continuous line. This graph illustrates the stability of the QuoCo and its balanced SE/SP, whereas the cut-off SP remains superior. Note: Educ = years of education, MMSE = Mini-Mental State Examination, QuoCo = Cognitive quotient, SE = sensitivity, SP = specificity.

Tables

Figures

View popup

Table 1:

Characteristics of the study populations

Characteristic	Participants from CSHA,^* mean ± SD		Participants from NACC,^* mean ± SD

	Healthy controls group n = 6411	Dementia group n = 1158^*	Healthy controls group n = 6000	Dementia group n = 501
Age at start of study, yr	76.04 ± 6.90	80.79 ± 6.67	72.93 ± 7.76	79.87 ± 7.07

Education, yr	10.49 ± 3.78	9.35 ± 3.88	15.71 ± 3.00	15.00 ± 3.25

MMSE score at start of study	27.07 ± 2.69	23.71 ± 5.14	28.98 ± 1.34	28.24 ± 1.77

MMSE decline at end of study	−0.54 ± 2.64^†	−6.70 ± 5.94^‡	−0.08 ± 1.45^§	−5.36 ± 5.33^¶

Note: CSHA = Canadian Study of Health and Aging,22 MMSE = Mini-Mental State Examination, NACC = National Alzheimer’s Coordinating Center’s Uniform Data Set,25 SD = standard deviation.
↵* Dementia developed in 1158 participants over the course of the study, including 361 prevalent cases of dementia.
↵† Participants who had normal cognitive function through the course of the study and had at least 1 follow-up MMSE (n = 3951).
↵‡ Participants who had a diagnosis of dementia during the course of the study and had at least 1 follow-up MMSE (n = 695).
↵§ Participants who had normal cognitive function through the course of the study and had at least 1 follow-up MMSE (n = 5864).
↵¶ Participants who had a diagnosis of dementia during the course of the study and had at least 1 follow-up MMSE (n = 486).

View popup

Table 2:

Estimates and confidence intervals for the initial prediction model for Mini-Mental State Examination scores

Parameter	Effect	Estimate (95% CI)
β₀	Intercept	−1.791 (−6.5 to 2.9)
β₁	Age, yr	0.786 (0.67 to 0.90)
β₂	Age₂	−0.00577 (−0.0065 to −0.0051)
β₃	Education, yr	0.0547 (−0.11 to 0.22)
β₄	Age × education	0.00297 (0.00086 to 0.0051)

Note: CI = confidence interval.

View popup

Table 3:

Accuracy of the cognitive charts for identifying participants with dementia in the training sample population (Canadian Study of Health and Aging)22

Parameter	At baseline n = 7569	At 5-yr follow-up n = 4401	At 10-yr follow-up n = 2528	Combined^*
Sensitivity (95% CI)	80 (75 to 84)	84 (79 to 88)	76 (70 to 81)	80
Specificity (95% CI)	89 (88 to 90)	81 (80 to 82)	88 (86 to 89)	87
PPV (95% CI)	26 (24 to 29)	22 (20 to 25)	43 (38 to 47)	29
NPV (95% CI)	99 (99 to 99)	99 (98 to 99)	97 (96 to 98)	99
LR+	7.3	4.4	6.3	6.2
LR−	0.22	0.20	0.27	0.23

Note: CI = confidence interval, LR+ = positive likelihood ratio, LR− = negative likelihood ratio, NPV = negative predictive value, PPV = positive predictive value.
↵* Combined values were obtained by modelling the relation between diagnoses and our classifier using repeated-measures logistic regression analyses. This takes into account the correlation between observations from a single participant; however, estimated sensitivities, specificities and predictive values are the same as if we had taken each individual observation independently. Consequently, we do not present confidence intervals for these combined values.

View popup

Table 4:

Accuracy of the cognitive charts for identifying participants with dementia in the validation sample population (National Alzheimer’s Coordinating Center’s Uniform Data Set25)^*

Parameter	At baseline n = 7846	At 5-yr follow-up n = 2230	At 9-yr follow-up n = 510	Combined^†
Sensitivity (95% CI)	–	65 (53 to 76)	57 (29 to 82)	64
Specificity (95% CI)	98 (98 to 98)	92 (91 to 93)	92 (90 to 95)	93
PPV (95% CI)	–	17 (13 to 22)	19 (8 to 33)	10
NPV (95% CI)	–	99 (99 to 99)	99 (97 to 99)	99
LR+	–	8.1	7.1	9.14
LR−	–	0.38	0.47	0.39

Note: CI = confidence interval, CSHA = Canadian Study of Health and Aging,22 LR+ = positive likelihood ratio, LR− = negative likelihood ratio, NACC = National Alzheimer’s Coordinating Center’s Uniform Data Set,25 NPV = negative predictive value, PPV = positive predictive value.
↵* In contrast with the results in Table 3 for the training sample population (CSHA), only healthy participants at the baseline measurement were included from the NACC’s UDS.
Consequently, we could only estimate specificity for the baseline measurement. In addition, the participants in the NACC’s UDS25 were evaluated every year for a maximum of 10 yr; therefore, the results at 5- and 9-yr follow-up exclude participants with a diagnosis of dementia before those dates. We have not presented results at 10-yr follow-up to mirror the CSHA22 study because attrition was too high by that time. However, the combined results take into account the full follow-up using the same procedure as in Table 3 and are comparable between the 2 data sets.
↵† Combined values were obtained by modelling the relation between diagnoses and our classifier using repeated-measures logistic regression analyses. This takes into account the correlation between observations from a single participant; however, estimated sensitivities, specificities and predictive values are the same as if we had taken each individual observation independently. Consequently, we do not present confidence intervals for these combined values.

View popup

Table 5:

Comparison of the cognitive quotient method and the Mini-Mental State Examination cut-off method for identifying participants with dementia, by sample population^*

Parameter	QuoCo		MMSE < 24

	CSHA participants	NACC participants	CSHA participants	NACC participants
Sensitivity (95% CI)	80 (77 to 83)	68 (63 to 72)	74 (71 to 77)	47 (42 to 51)

Specificity (95% CI)	82 (81 to 84)	90 (90 to 91)	92 (91 to 93)	99 (99 to 99)

PPV (95% CI)	44 (42 to 47)	37 (34 to 40)	63 (60 to 66)	80 (75 to 84)

NPV (95% CI)	96 (95 to 96)	97 (97 to 98)	95 (95 to 96)	96 (95 to 96)

LR+	4.44	6.80	9.25	47

LR−	0.24	0.36	0.28	0.54

Note: CSHA = Canadian Study of Health and Aging,22 LR+ = positive likelihood ratio, LR− = negative likelihood ratio, MMSE = Mini-Mental State Examination, NACC = National Alzheimer’s Coordinating Center’s Uniform Data Set,25 NPV = negative predictive value, PPV = positive predictive value, QuoCo = cognitive quotient.
↵* Reported values are for participants who completed at least 2 visits.

View popup
Table 6:
Net reclassification improvement of the cognitive quotient over the Mini-Mental State Examination cut-off score^* of < 24, by sample population
Parameter Study
CSHA^* NACC
NRI_e 0.0546 0.2099
NRI_ne −0.0990 −0.0863
NRI −0.0444 0.1236
Note: CSHA = Canadian Study of Health and Aging,22 NACC = National Alzheimer’s Coordinating Center’s Uniform Data Set,25 NRI_e = net reclassification improvement for events (participants with dementia), NRI_ne = net reclassification improvement for nonevents (participants with normal cognition), NRI = total net reclassification improvement, 3MS = Modified Mini-Mental State Examination.
↵* In the training sample population (CHSA), most participants with a 3MS score of 78 or more were automatically classified as cognitively normal without further clinical evaluation, thus providing an artificial advantage to the MMSE cut-off for nonevents.