Abstract
Background: The brief Patient Health Questionnaire (PHQ-9) is commonly used to screen for depression with 10 often recommended as the cut-off score. We summarized the psychometric properties of the PHQ-9 across a range of studies and cut-off scores to select the optimal cut-off for detecting depression.
Methods: We searched Embase, MEDLINE and PsycINFO from 1999 to August 2010 for studies that reported the diagnostic accuracy of PHQ-9 to diagnose major depressive disorders. We calculated summary sensitivity, specificity, likelihood ratios and diagnostic odds ratios for detecting major depressive disorder at different cut-off scores and in different settings. We used random-effects bivariate meta-analysis at cutoff points between 7 and 15 to produce summary receiver operating characteristic curves.
Results: We identified 18 validation studies (n = 7180) conducted in various clinical settings. Eleven studies provided details about the diagnostic properties of the questionnaire at more than one cut-off score (including 10), four studies reported a cut-off score of 10, and three studies reported cut-off scores other than 10. The pooled specificity results ranged from 0.73 (95% confidence interval [CI] 0.63–0.82) for a cut-off score of 7 to 0.96 (95% CI 0.94–0.97) for a cut-off score of 15. There was major variability in sensitivity for cut-off scores between 7 and 15. There were no substantial differences in the pooled sensitivity and specificity for a range of cut-off scores (8–11).
Interpretation: The PHQ-9 was found to have acceptable diagnostic properties for detecting major depressive disorder for cut-off scores between 8 and 11. Authors of future validation studies should consistently report the outcomes for different cut-off scores.
See related commentary by Kroenke at www.cmaj.ca/lookup/doi/10.1503/cmaj.112004
Depressive disorders are still under-recognized in medical settings despite major associated disability and costs. The use of short screening questionnaires may improve the recognition of depression in different medical settings.1 The depression module of the Patient Health Questionnaire (PHQ-9) has become increasingly popular in research and practice over the past decade.2 In its initial validation study, a score of 10 or higher had a sensitivity of 88% and a specificity of 88% for detecting major depressive disorders. Thus, a score of 10 has been recommended as the cut-off score for diagnosing this condition.3
In a recent review of the PHQ-9, Kroenke and colleagues argued against inflexible adherence to a single cut-off score.2 A recent analysis of the management of depression in general practice in the United Kingdom showed that the accuracy of predicting major depressive disorder could be improved by using 12 as the cut-off score.4
Given the widespread use of PHQ-9 in screening for depression and that certain cut-off scores are being recommended as part of national strategies to screen for depression (based on initial validation studies, which might not be generalizable),4,5 we attempted to determine whether the cut-off of 10 is optimum for screening for depression. This question could not be answered by two previous systematic reviews6,7 because of the small number of primary studies available at the time. We also aimed to provide greater clarity about the proper use of PHQ-9 given the many settings in which it is used.
Methods
We performed a meta-analysis of the available literature using recently developed bivariate meta-analysis methods.8,9,10 We included all cross-sectional validation studies of PHQ-9 as a screening tool for major depressive disorder that met our inclusion criteria.
Literature search
We searched Embase, MEDLINE and PsycINFO from 1999 (when PHQ-9 was issued) to August 2010 using the terms “PHQ-9” and “patient health questionnaire.” We manually searched the reference lists of each study that met our inclusion criteria, and we performed a reverse citation search in Web of Science to identify additional studies. We contacted the authors of original studies to obtain unpublished data if necessary. We also contacted the authors of unpublished studies and conference abstracts, and we reviewed these in an attempt to minimize publication bias. No publication or language restrictions were applied.
Study selection
We selected studies that reported the accuracy of PHQ-9 for diagnosing major depressive disorder. The studies had to provide sufficient data to allow us to calculate contingency tables. We included studies that defined major depressive disorder according to standard classification systems such as the International Classification of Diseases (ICD) or the Diagnostic and Statistical Manual of Mental Disorders (DSM). We excluded studies in which the diagnoses were not made using a standardized diagnostic interview schedule (e.g., Mini International Neuropsychiatric Interview [MINI], Structured Clinical Interview for DSM Disorders [SCID], Composite International Diagnostic Interview [CIDI], Diagnostic Interview Schedule [DIS] or Revised Clinical Interview Schedule [CIS-R]). We made the final selection after examining the full-text articles. Figure 1 shows selection of studies using this strategy.
Selection of studies for inclusion in the systematic review. Note: PHQ-9 = Patient Health Questionnaire.
Data abstraction
We collected information about study characteristics and quality using a standardized data collection form. We included the following characteristics: settings, year of study, sample size, study design, timing between reference and index tests, training of the rater of the reference test, blinding of the assessor of the reference test, data integrity, cut-off score, and translation and validation of non-English versions of PHQ-9. We recorded accuracy data for the reported cut-off scores in contingency tables.
Quality assessment
We based our assessment of quality on current guidelines for evaluating diagnostic studies.11 We followed the quality ratings used in other studies of the diagnostic characteristics of psychological measures to generate our quality assessment criteria.12 We devised specific questions to evaluate the translation and validation of non-English versions of PHQ-9 and design issues that could influence the outcomes.
Our quality criteria included adequate sample size (n ≥ 250). We considered the single-gate design to be ideal. A single-gate study compares the results from an index test with those from a reference standard used to confirm the diagnosis. In a two-gate design, the disease status is already known. A two-gate study compares the results of an index test for patients with an established diagnosis of the target condition, which are therefore treated as the reference standard, with the results of the same test in healthy people or people with another condition.13
We assessed whether the rater of the test had been trained in the use of the reference test and whether the assessor was blinded to the result of the reference test. We also examined whether withdrawals or drop-outs were explained or accounted for. We considered a rate of less than 20% refusals or drop-outs to be acceptable. It was also important that the time between the index test and the reference test to be two weeks or less. We considered the chosen cut-off score for reporting to be acceptable if it was tested as best trade-off. Finally, for studies that used a translated version of PHQ-9, we examined whether the translation had been validated according to recognized standards.14,15,16
Data synthesis and statistical analysis
We constructed 2 × 2 tables for each cut-off score and computed the sensitivity, specificity and positive and negative predictive values.
We performed a bivariate meta-analysis to obtain pooled estimates of specificity and sensitivity and their associated 95% confidence intervals (CIs).17 We constructed summary receiver operating characteristic curves17 using the bivariate model to produce a 95% confidence ellipse within the receiver operating characteristic curve space.18 Each data score in this space represents a separate study. This is unlike a traditional receiver operating characteristic plot, which explores the effect of varying thresholds on sensitivity and specificity in a single study.
We assessed between-study heterogeneity using the I2 statistic for the pooled diagnostic odds ratio (OR),19 which describes the percentage of total variation across studies that is caused by heterogeneity rather than chance. We considered an I2 value of 25% to be low, 50% to be moderate and 75% to be high. We explored the causes of heterogeneity if there was significant between-study heterogeneity. We identified the studies that were outside of the 95% confidence ellipse by visually inspecting the summary receiver operating characteristic curve plots.
We performed a meta-regression analysis of the logit diagnostic ORs using a priori identified sources of heterogeneity entered as covariates in the meta-regression model.9 We investigated heterogeneity resulting from the characteristics of the sample or study design by exploring the effects of potential predictive variables.8
We examined publication and small study bias using Begg’s funnel plots of log diagnostic ORs versus the inverse of the variance.10,20
Results
We found 18 studies (7180 participants) that met our inclusion criteria (Appendix 1, www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110829/-/DC1). The excluded studies and the reasons for exclusion are described in Appendix 2 (available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110829/-/DC1). Eight of the included studies validated PHQ-9 in primary care;21–28 five in specialized secondary care services (brain injury,29 cardiology,30,31 stroke32 and renal33) and two in samples from the community.34,35 Three studies were conducted in mixed settings (outpatient clinics and family practices).3,36,37 The mean age of participants ranged from 24.8 to 71.4 years.23,34 Within the 18 included studies, the prevalence of depression, as diagnosed by the gold-standard tests, ranged from 2.5% to 37.5%.22,34 The included studies used the English version3,22,28–35 and translated versions (Portuguese,21 German,36,37 Dutch,23,27 Thai,24 Malay25 and Konkani26) of PHQ-9.
Quality assessment
All included studies used the DSM or ICD-10 diagnosis of depression, established using a standardized interview schedule. The interview schedules used were the English or translated versions of the MINI,23,24,30,34 SCID, 3,21,22,27–29,32,33,35–37 CIDI,25 DIS31 or CIS-R.26 In 14 studies, participants were assessed by use of both the PHQ-9 and reference test. 3,21,22,24–26,29,35–37 Four studies used other study designs in which only patients who scored below a certain cut-off on the PHQ-923,27,28 or who showed the core symptoms or at least two symptoms on the PHQ-932 underwent testing using the gold-standard test.
Meta-analysis
Eighteen studies (7180 patients, 927 with major depressive disorder confirmed by DSM or ICD) reported the diagnostic properties of PHQ-9 at different cut-off scores. We pooled studies with different cut-off scores because not all studies reported the same cut-off scores.
We found a high level of between-study heterogeneity for psychometric attributes (I2 = 82.4%). The pooled sensitivity for a cut-off score of 10 was 0.85 (95% CI 0.75–0.91) and the pooled specificity was 0.89 (95% CI 0.83–0.92) (Table 1).
Pooled estimates of the sensitivity, specificity, positive and negative likelihood ratios and diagnostic odds ratios of the brief Patient Health Questionnaire (PHQ-9) for diagnosing major depressive disorder, by cut-off score
When we summarized individual studies within receiver operating characteristic curve space for a cut-off score of 10, we found that most studies gathered within an informative top left corner (Appendix 3, available at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110829/-/DC1).
Two of the three studies with a relatively low sensitivity at a cut-off score of 10 were conducted in cardiology settings.30,31 The other study was conducted in primary care and was the only one that used a diagnostic interview based on ICD criteria.25 The pooled sensitivity for studies performed in hospital settings (0.74, 95% CI 0.55–0.86) was lower than that for primary care (0.89, 95% CI 0.66–0.97); however, the specificity was very similar for these settings (0.89 [95% CI 0.87–0.91] v. 0.88 [95% CI 0.80–0.93])
Because of the between-study heterogeneity, we performed a meta-regression. The criterion of blind application of a diagnostic gold standard was the only a priori source of heterogeneity that was predictive (p = 0.032). Diagnostic performance did not vary according to the percentage of women (p = 0.39), study setting (primary care and community settings v. hospital; p = 0.73), prevalence of depression (p = 0.70), sample size (p = 1.00) or mean age (p = 0.28).
Translation of PHQ-9 (p = 0.33), study design (single gated v. double gated; p = 0.19), timing of index and reference testing (p = 0.61), standard of data integrity reporting (p = 0.40) and training of the rater of the reference test (p = 0.11) did not have a significant impact on diagnostic performance.
We found that the diagnostic OR was lower in hospital settings30–33 (diagnostic OR 25.43, 95% CI 11.35–57.00) than in primary care settings21,22,25–27 (diagnostic OR 65.26, 95% CI 9.17–464.47) (Table 2). Studies in primary care and hospital settings were equally heterogeneous (primary care I2 = 84.7%; hospital I2 = 84.2%).
Pooled estimates of the sensitivity, specificity, positive and negative likelihood ratios and diagnostic odds ratios of the brief Patient Health Questionnaire (PHQ-9) for diagnosing major depressive disorder, by setting
Alternative cut-off scores
Fourteen studies reported the diagnostic properties of PHQ-9 for a cut-off score of 10, and some reported additional cut-off scores (Table 1). A cut-off score greater than 10 was reported as being optimal by only three studies.3,25,34 In 4 studies, a cut-off score of 11 or 12 had better diagnostic properties than a cut-off score of 10.22,29,36,37
The pooled sensitivity and specificity results show no significant differences in the diagnostic properties of PHQ-9 for cut-off scores between 8 and 11. The pooled specificity was between 0.73 (95% CI 0.63–0.82) for a cut-off score of 7 and 0.96 (95% CI 0.94–0.97) for a cut-off score of 15. Overall, the sensitivity did not decrease as the cut-off score increased (Table 2). There was a decrease in sensitivity to 0.77 (95% CI 0.60–0.88) for a cut-off score of 12; however, sensitivity improved to 0.81 for a cut-off score of 13 (95% CI 0.75–0.85). A cut-off score of 11 had the best trade-off between sensitivity and specificity.
Interpretation
We found that PHQ-9 has acceptable diagnostic properties at a range of cut-off scores (8–11). There were no significant differences in sensitivity or specificity at a cut-off score of 10 compared with other cut-off scores within this interval (8–11). However, our results are based on a variable number of studies for each cut-off scores. To provide a more valid comparison between different cut-off scores, more studies that consistently report data for a range of cut-off scores are needed.
This systematic review of the diagnostic properties of the PHQ-9 for different cut-off scores follows previous recommendations to analyze the sensitivity and specificity of the PHQ-9 at various cut-off scores using bivariate meta-analysis.6,7 Our results support previous findings that PHQ-9 has acceptable diagnostic properties for major depressive disorder. Its diagnostic accuracy was reasonably consistent despite clinical heterogeneity of the included studies.
The methodologic quality of the studies varied, and the level of between-study heterogeneity was consistently high. A significant finding is that the reported blind application of a diagnostic gold standard was the only predictive source of heterogeneity. Although no other a priori sources of heterogeneity apart from blinding were able to explain the substantial between-study variation, we recommend that the proposed potential sources of heterogeneity (e.g., single-gated study design, training of the rater of the reference test, blinding of the assessor to the result of the reference test, and the use of validated translations of the index and reference tests) should be included if further primary studies are performed.
We found that there were no significant differences in pooled sensitivity and specificity for cut-off scores between 8 and 11. There was a decrease in sensitivity for a cut-off score of 12; however, sensitivity improved for a cut-off score of 13. The fact that different studies contributed to the calculations of different cut-off scores might be a possible explanation for this unexpected trend.
Limitations
Our study has several limitations. First, we could not rule out publication bias. Study selection was perfomed by one author, and this might have introduced bias. Our quality assessment criteria have not been validated. We were unable to fully explain the large amount of heterogeneity between studies, and caution should be used when interpreting the results.
Four of the studies included in our analysis used designs in which only patients who scored below a certain cut-off on the PHQ-923,27,28 or who showed the core symptoms or at least two symptoms on the PHQ-932 underwent testing using the gold-standard test. These designs may have led to partial verification bias. By selectively including patients with a known or possible diagnosis of depression, the prevalence of depressive disease in the sample could be increased and the sensitivity may have been overestimated.
Conclusions and implications for further research
The PHQ-9 is a popular tool for detecting depression in many settings. Our findings emphasize the importance of using caution when choosing a specific cut-off score, taking into account the characteristics of the population, the settings and the efficacy of screening on outcomes. A cut-off score of 10 may result in many false negatives in hospital settings, while more false-positive results may be seen in primary care. Our results support previous observations that a cut-off score of 10 is not superior to a score of 11 or 12 in terms of sensitivity.4,22,37 However, the included studies reported different cut-off scores, and some of them made a post hoc selection of the best cut-off score. The results of our analysis are based on a different number of studies for each cut-off score, each of which included a different population and had different methodologic characteristics. We recommend that studies of diagnostic accuracy should report the results for all cut-off scores and avoid reporting only the scores that are determined to be best after the study has been performed.
It has previously been suggested that the blinding of the assessor of the reference test could account for some of the between-study variation;6 however, this has not been shown empirically. Blinding is one of the few evidence-based quality criteria that can have a significant effect on estimates of diagnostic accuracy.38 Our study provides further evidence that future studies of diagnostic accuracy should explicitly report blinding.
It is also important that researchers recruit participants who represent the intended spectrum of severity of the target condition and that all participants complete both the index and reference test.
The same cut-off score might not be appropriate in all settings. PHQ-9 is a highly useful screening tool, but it is not a stand-alone diagnostic test. Given the structure of the questionnaire and its intended use as a screening tool, the optimal cutoff score may differ depending on the setting
Footnotes
Competing interests: Dean McMillan has received grants from the UK National Institute for Health Research. No competing interests declared by Laura Manea and Simon Gilbody.
This article has been peer reviewed.
Contributors: Laura Manea contributed to the acquisition, analysis and interpretation of the data and drafted the manuscript. Simon Gilbody and Dean McMillan contributed to the concept and design of the study and the interpretation of data and revised the manuscript for important intellectual content. All of the authors approved the final version of the manuscript submitted for publication.
Funding: No funding was received for this study.
References
In this issue
Article tools
Jump to section
Related Articles
Cited By...
- Mental health condition of college students compared to non-students during COVID-19 lockdown: the CONFINS study
- International survey on fear and childbirth experience in pregnancy and the postpartum period during the COVID-19 pandemic: study protocol
- Acute psychological impact of coronavirus disease 2019 outbreak among psychiatric professionals in China: a multicentre, cross-sectional, web-based study
- Depression and anxiety before and during the COVID-19 lockdown: a longitudinal cohort study with university students
- Examining the association between family status and depression in the UK Biobank
- Health Care Workers Mental Health During the First Weeks of the SARS-CoV-2 Pandemic in Switzerland - A Cross-Sectional Study
- Depression, periodontitis, caries and missing teeth in the USA, NHANES 2009-2014
- Epidemiology of depressive disorders in people living with hypertension in Africa: a systematic review and meta-analysis
- A Randomized Controlled Trial to Increase Cancer Screening and Reduce Depression among Low-Income Women
- Caring about caregivers: the role of paediatricians in supporting the mental health of parents of children with high caregiving needs
- Experience of fatigue and associated factors among adult people living with HIV attending ART clinic: a hospital-based cross-sectional study in Ethiopia
- Psychological impact of COVID-19 pandemic on postgraduate trainees: a cross-sectional survey
- Can and should neurologists screen their patients for depression? Yes, and...
- Association between depressive symptoms and arterial stiffness: a cross-sectional study in the general Chinese population
- Efficacy of a mindful-eating programme to reduce emotional eating in patients suffering from overweight or obesity in primary care settings: a cluster-randomised trial protocol
- A two-stage approach to identifying and validating modifiable factors for the prevention of depression
- Depressive disorder and its associated factors among prisoners in Debre Berhan Town, North Showa, Ethiopia
- Computerised memory specificity training (c-MeST) for the treatment of major depression: a study protocol for a randomised controlled trial
- Examining the Relationships Between Perfectionism and Obsessive-Compulsive Symptom Dimensions Among Rural Veterans
- Obsessive-Compulsive Symptom Dimensions and Insomnia: Associations Among a Treatment-Seeking Veteran Sample
- Exposure and Response Prevention for Obsessive-Compulsive Disorder: A Case Study of a Veteran With Violent Intrusive Thoughts
- Systematic psychiatric assessment of patients with sickle cell disease
- Factors associated with depressive symptoms in pharmacy residents
- Smartphone application for preventing depression: study protocol for a workplace randomised controlled trial
- Rates of depressive symptoms among pharmacy residents
- Ethnic Differences in Prevalence of Post-stroke Depression
- Predictors, help-seeking behaviour and treatment coverage for depression in adults in Sehore district, India
- Telephone-supported computerised cognitive-behavioural therapy: REEACT-2 large-scale pragmatic randomised controlled trial
- Depression in epilepsy, migraine, and multiple sclerosis: Epidemiology and how to screen for it
- Reporting quality in abstracts of meta-analyses of depression screening tool accuracy: a review of systematic reviews and meta-analyses
- Comparative Potential of the 2-Item Versus the 9-Item Patient Health Questionnaire to Predict Death or Rehospitalization in Heart Failure
- Developing an Instrumental Activities of Daily Living Tool as Part of the Low Vision Assessment of Daily Activities Protocol
- Collaborative Care Versus Screening and Follow-up for Patients With Diabetes and Depressive Symptoms: Results of a Primary Care-Based Comparative Effectiveness Trial
- Screening, Assessment, and Care of Anxiety and Depressive Symptoms in Adults With Cancer: An American Society of Clinical Oncology Guideline Adaptation
- Enhancing the clinical utility of depression screening
More in this TOC Section
Similar Articles
Collections
