Development of the AGREE II, part 2: assessment of validity of items and tools to support application

Design and sample size

We used a two-level factorial design. Guideline quality (i.e., high and low) was the between-subjects factor. We sought to recruit 15 participants per group, to enable a two-sided test to have 80% power to detect an advantage of as little as one point on the seven-point scale between the high-quality and low-quality groups.

Participants

A convenience sample of guideline developers, researchers and clinicians was recruited to participate in this study. They were recruited from the Program in Evidence-based Care of Cancer Care Ontario, the Canadian Partnership Against Cancer and international coinvestigators of the research team. We oversampled by 33% to ensure we would receive data for our targeted sample size of 30.

Creating guideline content of varying quality for assessment

An existing cancer-related guideline developed by an established guidelines program 10 was used as the source guideline from which we purposefully designed excerpts of content of varying quality to reflect 21 of the 23 AGREE items. This guideline was chosen because it was of mid-range quality (as determined by two independent appraisers using the original AGREE instrument [MK, JM]), and thus enabled us to easily craft higher-quality content and lower-quality content. The AGREE instrument had not been explicitly used to facilitate its development. We did not test item 16 (i.e., the different options for management of the condition are clearly presented) because the source document only focused on one effective treatment option, and we did not want to introduce a recommendation that was fictitious or not based on evidence. Therefore, this item was excluded. Item 17 (i.e., key recommendatons are easily identifiable) was not manipulated, because we were presenting guideline excerpts related to each of the items one at a time rather than embedding all of the manipulated content to create a whole version of a guideline. Therefore, all participants received the same content as in the original source guideline for item 17.

In crafting guideline content, our objective was to reflect more nuanced differences that might typically be seen between guidelines rather than extreme examples of high and low content (Table 1). For each item, a high-quality version and a low-quality version of the content was pilot-tested, reviewed and refined by three members of the team (MB, MK, ER). From there, two versions of a study package were created. Excerpts of high- and low-quality content were randomly assigned to each version of the study package using a random number generator, such that in each package, only one version (high or low) was included for each item (except item 17 as per above). Version 1 included 14 high-quality items and 7 low-quality items. Version 2 included 7 high-quality items and 14 low-quality items (i.e., the inverse of version 1 in quality).

Administration

After obtaining ethics approval, we distributed personalized letters of invitation and then reminders via email to participant-candidates. Participants were assigned a unique identifier code and were blinded to group and purpose of the study. They were randomly assigned to one of the two versions of the study package and sent a confidential username and password to access the web-based study platform. Once logged on, participants were asked to assess the guideline content, using the β-AGREE II items and user’s manual to guide their assessment. Content relevant to each item was presented sequentially. Participants were then asked to fill out a survey to assess the usefulness of the user’s manual.

Measures

Analysis of data

To assess whether differences in item ratings existed between guideline content designed to be of high and low quality and to correct for multiple comparisons, we conducted a multivariable analysis of variance (MANOVA). This analysis included the 21 manipulated items as dependent measures. We report the results of both the MANOVA and the univariable analysis. A separate analysis of variance (ANOVA) was conducted to compare scores for item 17 between the two groups, where no difference was expected.

Descriptive statistics were calculated for each of the three assessment measures of the user’s manual. For exploratory purposes, total scores were added across the AGREE items for each of the three assessment measures of the user’s manual, and a one-way ANOVA was undertaken to determine if differences in overall assessments existed between version 1 and version 2 of the study packages. Our hypothesis was that no differences would exist between the two groups.

Participants

Of 41 invited participants, we received data from 30 people (for a response rate of 73%), which met our requirement for sample size. One data point was missing for two of these participants. The demographic characteristics of participants are provided in Table 2. Almost three quarters of participants identified themselves as researchers, 28% engaged in clinical practice, and 83% were participants in some aspect of the guideline enterprise.

Assessment of guideline excerpts

Multivariable analysis of variance yielded a significant main effect for guideline quality (p = 0.005). Univariable analyses yielded significant differences in scores for 18 of the 21 manipulated items (Table 3). In all cases, content designed to be of high quality was rated higher than content designed to be of low quality. Whereas the mean scores were in the correct direction, the three items that did not yield significant univariable differences between the high- and low-quality item versions were item 10 (i.e., methods for formulating recommendations are clearly described) , item 11 (i.e., health benefits, side effects and risks have been considered in formulating recommendations); and item 12 (i.e., there is an explicit link between the recommendations and the supporting evidence). As expected, item 17 (i.e., key recommendations are easily identifiable), for which participants in both groups received the same version, did not yield a significant difference between the high- and low-quality versions (p > 0.05) in the separate ANOVA.

Assessment of the user’s manual

The results of the three usability assessments of the β version of the user’s manual across each of the AGREE II items are presented in Table 4. Mean scores were high, with a range of 5.43–6.43 for the measure of appropriateness, 5.33–6.33 for that of ease of application, and 5.21–6.27 for that of ability to discriminate. No differences in total assessment scores were found between study package 1 and study package 2 (p > 0.05).

Final refinements

We received considerable written feedback from participants, including specific suggestions for improvements to the instrument (not presented). All feedback, in combination with feedback received in part 1, was formally discussed by the AGREE Next Steps Consortium, and final modifications were made to create the AGREE II. 8

Limitations

Our study has limitations. First, in testing of the β-AGREE II, participants were presented with excerpts of a guideline that reflected each item’s concept and not an entire guideline. Whether the items would be sensitive in discriminating between differences in quality when users are presented with an entire guideline is a question for future research. Second, we chose a convenience sample of participants that was comprised primarily of guideline developers and researchers rather than a full range of potential users of AGREE II. As such, the generalizability of the findings may be limited. However, given that most of our participants (83%) were experienced in guideline development or research, they were uniquely situated as consumers who could be critical of the value of the user’s manual. Third, although we met our sample-size goal of 30 participants for analytical purposes, this study is modestly sized. This fact may raise questions regarding the generalizability of our findings to a larger group of stakeholders. Fourth, by using a specialist (oncologic) guideline focused on one procedure as our source document, extrapolation of our findings to other clinical areas is contestable. Finally, for many of the items, the word count for high-quality items was larger than the word count for low-quality items. Thus, word count may be confounded with quality when interpreting the differences that emerged. This fact too may limit the generalizability of our findings.

Conclusion

Our study represents the first systematic assessment of the construct validity of the AGREE. Future research is warranted to reproduce these findings using a larger sample of stakeholders and including manipulated guideline content within the context of a whole report.

In combination with part 1, 9 our results led to the final refinements and release of the AGREE II, the revised standard for guideline development, reporting and evaluation. 8 The AGREE II is available at the website of the AGREE Research Trust (www.agreetrust.org).