Development of the AGREE II, part 1: performance, usefulness and areas for improvement

Design

We used an unbalanced mixed factorial design incorporating four factors, which were user type (i.e., clinicians, guideline developers or researchers, and policy-makers), clinical topic (i.e., cancer, cardiovascular medicine and critical care), group (i.e., AGREE and Global Rating Scale versus Global Rating Scale only) and guideline (i.e., 10 reports). Guideline was not a factor of analytical interest. The group factor will not be discussed here. The study design and allocation of participants to the various factors are illustrated in Table 1. Participants who were randomized to group 1 completed guideline assessments with and questionnaires about both the AGREE and the Global Rating Scale, and group 2 did so with the Global Rating Scale only. We report here on the AGREE-specific data from group 1.

Participants

A sample of clinicians, policy-makers, and guideline developers or researchers was recruited to participate in the study (Table 1). For the whole study (i.e., including both the AGREE instrument and the Global Rating Scale), 503 clinicians, 174 policy-makers, and 164 developers or researchers were invited to participate. Three strategies that were unique to each user group were used to allocate participants to factors (Table 1).

Sample size

Our sample-size calculation was based on the interaction of user type x clinical topic x group, which was the primary analysis of interest of the whole study. After an a priori sample-size calculation for our primary outcome, which was domain score, our recruitment target for the total study was 192 participants. For the component of the study in this report, it was 96, comprising 40 clinicians (16 oncologic, 16 cardiovascular and 8 critical care clinicians), 16 policy-makers, and 40 developers or researchers.

Selection of guidelines

We used the US National Guidelines Clearinghouse database to search for eligible guidelines. To ensure a range of quality, a purposeful sample of 10 was chosen based on two independent assessments (M.K., J.M.) for rigour of development using the original AGREE instrument (i.e., the original seven items of this domain were assessed using the original four-point scale).

Administration

After obtaining ethics approval, we sent each participant-candidate a personalized letter of invitation in the mail and an email to ascertain interest. Two reminders were sent by email to each nonresponder. Candidates who did not complete their survey were categorized as non-responders. Consent to participate was implied with receipt of the participant’s data.

Candidates who agreed to participate were assigned a unique identifier code and a confidential username and password to access the Web-based study platform. (We also accepted submissions by mail or fax.) Once logged on to the Web-based platform, participants read the guideline assigned to them and evaluated it. They then completed a series of questionnaires aimed at assessing the usefulness of the AGREE items and domains, ways in which the items could be improved and the feasibility of application. 1 An online portal for web surveys collected the data and saved it on a secure password protected data storage site.

Measures

Analysis of data

A series of analysis of variance tests, multiple regressions, Chonbach α, and intraclass correlations were used to assess the data set (Figure 1) (Appendix 1, available at www.cmaj.ca/cgi/content/full/cmaj.091714/DC1).

Seven-point scale

When we tested the performance of the seven-point scale, we found significant differences in domain scores as a function of user type. With the exception of the applicability domain, where no differences were found, guideline developers and researchers gave lower domain-quality ratings, on average, than did clinicians or policy-makers (Table 3).

An exploratory analysis of the measurement properties of the scale showed that internal consistency ranged from 0.64 (for editorial independence) to 0.89 (for rigour of development). Inter-rater reliability was adequate. The number of appraisers required to reach a level of inter-rater reliability of 0.7 ranged from two to five across domains (Table 3).

Usefulness of the items

All items and domains were rated above the mid-point of the scale as useful by participants (Table 4). Among the items, “updating procedure” (item 14) received the lowest mean score (4.80) and “link between evidence and recommendations” (item 12) was rated highest (6.53). The domain of applicability was rated lowest (4.98) and that of scope and purpose was rated highest (6.32). No significant differences were found across ratings of usefulness of items or domains as a function of user type.

Predicting outcomes

With the exception of the editorial independence domain, each of the remaining five domains was a significant positive predictor for the three outcome measures. The magnitude of effect varied across domains and outcomes, and ranged from a change of 0.1 to 0.8 (on a seven-point scale) in the outcome measure for every 10% change in the domain score (Table 5). For example, every 10% change in the rigour of development domain predicted a 0.8 change in participants’ endorsement score. User type and the interaction of domain and user type did not predict outcomes.

Improving items and domains

Participants provided recommendations for improvement of all items and domains. (Qualitative data is not reported.) Feedback was offered most frequently for the stakeholder involvement domain, with 39.8% of all participants suggesting modifications. Ten items were recommended by at least one user for deletion. The items nominated most frequently for deletion were “pilot testing” (item 7) (9.6% of participants) and “tools for application” (item 18) (10.6%).

Limitations

Our study has limitations. First, although we oversampled by at least four times to achieve our target sample size, we were unable to meet our goal of 96 participants. While this shortfall did not have an impact on the power required for our primary analysis, it does attest to the challenges of conducting research related to health services. 17 Second, owing to the smaller pool from which to recruit participants, policy-makers evaluated cancer-related guidelines only (i.e., in contrast to developers or researchers and to clinicians) (Table 1). Therefore, differences found in performance as a function of type of user may have been confounded by guideline topic. Indeed, in a separate analysis (not shown), cancer-related guidelines tended to be evaluated as being of higher quality than other guideline topics. However, when policy-makers were excluded from the primary analyses, with the exception of one finding, the results continued to show that guideline developers and researchers gave statistically lower ratings of quality than did clinicians. This finding gives us confidence that we are seeing true differences between types of users.

Conclusion

The results presented here serve as an important component of the consortium’s overall program of research. Our study shows a promising introduction of the new seven-point scale. It shows that items in the AGREE have universal value and can predict important outcomes associated with guideline adoption. Finally, we received considerable feedback on the original version of AGREE, which we used to improve and refine the tool. These results, in combination with the results from our second study reported in this series, 12 have led to the release of the AGREE II, the revised standard for guideline development, reporting and evaluation. 13