Monitoring use of knowledge and evaluating outcomes

How can use of knowledge be measured?

Many tools exist for assessing use of knowledge. Dunn 3 completed an inventory of tools for conducting research on use of knowledge and identified 65 strategies, but most have unknown validity or reliability. Most frequently, tools for the utilization of knowledge measure instrumental use of knowledge. 9 Often these measures rely on self-report and are subject to recall bias. For example, a case study described adoption by call centre nurses of a protocol for decision-making support. 10 Eleven of 25 nurses who were surveyed said they used the tool in practice. Potential limitations to this study include recall bias and a short period of follow-up (i.e., one month) without repeated observation. 10 In a more valid assessment of instrumental use of knowledge, participants underwent a quality-based assessment of their coaching skills during simulated calls to determine how often the protocol for decision-making support was used. 11

Assessing instrumental use of knowledge can also be done by measuring adherence to recommendations or quality indicators. Grol 12 completed a series of studies involving family physicians in the Netherlands who recorded their adherence to 30 national guidelines. Three hundred forty-two indicators of adherence were constructed and physicians received educational sessions on how to record their performance on these indicators. Computer software was developed to relate performance to clinical conditions to assess adherence. 12 More simply, we could look at how often we prescribe β-blockers in appropriate patients with heart failure through a chart-based audit.

We also need to consider who the targets are for use of knowledge (i.e., the public, health care professionals, policy-makers), because they may require different strategies for monitoring use of knowledge. Assessing use of knowledge by policy-makers may require strategies such as interviews and analysis of documents (e.g., reviewing policies to assess use of evidence). 13 When assessing use of knowledge by physicians, we could measure use of paths of care or ordering of relevant medications, which are often measured through use of administrative or clinical databases. Also, when measuring use of knowledge by the public, we could measure attitudes of patients through surveys or use of resources through administrative databases.

What is the target level of use of knowledge that we are aiming for? This target is based on discussions with stakeholders and includes consideration of what is acceptable and feasible and whether a ceiling effect may exist. 14 If the degree of use of knowledge is found to be adequate, strategies for monitoring sustained use of knowledge should be considered. If the degree of use of knowledge is less than expected or desired, reassessment of barriers to uptake may be necessary.

When should we measure use of knowledge versus the impact of use of knowledge? If the implementation-related intervention targets a behaviour for which a strong evidence of benefit exists, measuring the impact of the intervention in terms of whether the behaviour has occurred, rather than whether a change in clinical outcomes has occurred, may be appropriate. 15 A strategy to implement the guidelines of Osteoporosis Canada in a community setting was recently studied. 16 The primary outcome of this randomized trial was appropriate use of medications for osteoporosis (i.e., instrumental knowledge) rather than fractures in patients (i.e., clinical outcome). The researchers felt that, because sufficient evidence exists to support use of medication for osteoporosis to prevent fragility fractures, they did not need to measure fractures as the primary outcome. In such instances, measurement of outcomes at the patient level could be prohibitively expensive, but failure to measure at the patient level does not address whether the intervention improves relevant clinical outcomes.

Methods for evaluating interventions

The question should be matched to the appropriate study design. When developing an evaluation, we need to consider rigour and feasibility. By rigour, we mean that the strategy for evaluation should use explicit and valid methods. Both qualitative and quantitative methods could be used. By feasible, we mean that the strategy for evaluation should be realistic and appropriate given the setting.

Selection of a strategy for evaluation also depends on whether we want to enhance local knowledge or provide generalizable information on the validity of the intervention related to knowledge translation. Those interested in local applicability of knowledge (i.e., whether an intervention worked or not in the context in which it was implemented) should use the most rigorous study designs feasible. These designs may include observational evaluations, in which the researcher does not have control over allocation of study participants to the intervention or a comparable control. Those interested in generalizable knowledge (i.e., whether an intervention is likely to work in comparable settings) should use the most rigorous design for evaluation-specific research that they can afford, such as randomized trials or experimental evaluation. A third form of evaluation to consider is process-related evaluation. This form of evaluation may involve determining the extent to which decision-makers were exposed to the intervention. Additionally, it may include a description of the experience of those exposed to the intervention and potential barriers to the intervention.

For example, a study evaluating the effectiveness of an educational intervention on the use of radiography for diagnosis of acute ankle injuries showed that the dissemination of the Ottawa ankle rules had no impact. However, less than a third of those receiving the intervention were actually physicians who had authority to order x-rays. This fact raises questions about whether the intervention was not effective or simply not directed to the appropriate decision-makers. 20 This type of evaluation is also useful because it allows corrections to be made to the intervention.

Qualitative methods of evaluation can be helpful in exploring the “active ingredients” of an intervention related to knowledge translation and thus they are particularly useful in process-specific evaluation. In a randomized trial of a comprehensive, multifaceted strategy for implementation of guidelines for family physicians, no changes in testing of cholesterol were noted after a one-year intervention. 21 This finding led to interviews with family physicians who expressed concern about the extra workload associated with implementation of the guidelines and suggested revisions to the diagnostic algorithm. 22

Quantitative evaluation methods included randomized and quasi-experimental studies. Randomized trials are more logistically demanding but provide more reliable results than non-randomized studies. Nonrandomized studies often can be implemented more easily and are appropriate when randomization is not possible.

Framework for evaluating complex interventions

Mixed methods can be used to evaluate complex interventions. To some extent, all interventions can be seen as complex. The relatively simple act of prescribing a pill is accompanied by a series of steps to ensure adherence and check for adverse effects and drug interactions. The key active ingredient, the pill, is readily identified. For more complex interventions, identifying the precise mechanism that may contribute to outcome is difficult because these interventions contain a number of different elements that act independently or interdependently. 23 An example is systems of care to optimize health outcomes for patients recovering from a stroke. Stroke units, compared with less organized forms of inpatient care, are effective in improving the survival of patients who have had a stroke and reducing their level of dependency. 24 The elements of a stroke unit that are associated with a beneficial outcome are not obvious from the trials included in this systematic review.

Recently, complex interventions have been a focus of debate because evidence has shown a beneficial effect for some complex interventions and not others. This discrepancy has led decision-makers to question which elements of an intervention are essential, and whether, when a trial has shown no effect, the cause is related to problems with the design or conduct of the study. One of the most influential initiatives to address this challenge is the Medical Research Council framework for the evaluation of complex interventions. 25 This framework provides researchers with an iterative, step-wise approach to evaluating a complex intervention.

The first step in this framework is defining the intervention, which involves identifying the existing evidence and any theoretical basis for the intervention so that the components of the intervention can be described. The second step is an exploratory phase in which the acceptability and feasibility of delivering the intervention and the comparison intervention are assessed and the study design is piloted. The third step is an explanatory phase, during which the final design of the trial is implemented in a relevant setting with appropriate criteria for eligibility, taking into account statistical power and relevant measures of outcome. Finally, the fourth step is a pragmatic phase in which the implementation of the intervention is examined with attention to the fidelity of the intervention, participants eligible for the intervention and any possible adverse effects. 23

Knowledge translation, complex interventions and the iterative loop

The framework of the Medical Research Council can be used to facilitate the translation of evidence by providing a mechanism for integrating additional forms of evidence relevant to decision-makers, such as qualitative or survey-derived data. In a survey of trialists contributing data to the systematic review of stroke units, 25 stroke units appeared to act as a focal point for the organization and coordination of services rather than a centre for intensive rehabilitation. A common feature of stroke units in the survey was that care was organized and coordinated by a multidisciplinary team of staff who were interested or knowledgeable about stroke. The stroke units also encouraged the involvement of caregivers. 26

A qualitative study 27 was conducted in parallel with a trial of intensive case management for people with severe mental illness. The study investigated the active ingredients of the intervention with attention to the roles of staff, practices and organizational features. Providing a comprehensive assessment and needs-led service were regarded as the key mechanisms of this intervention. Organizational features, such as an absence of team-management, limited the extent to which case managers could make an impact. Finally, the degree to which an intervention has been sustained outside the trial can be explored, such as by assessing the volume and type of patients using an admission-avoidance hospital-at-home program after the completion of a randomized trial. 28

At each phase of research on interventions for knowledge translation, input should be obtained from policy-makers, clinicians and managers in health care. Involving decision-makers in shaping the question and defining the intervention can help to ensure the relevance of research. Input from decision-makers has the potential to strengthen the generalizability of the research. Local applicability is a key factor influencing the use of evidence, and identifying the variables that define the context of the findings of research can help decision-makers address this factor. 29

The importance of the generalizability of complex interventions has recently received attention, with the development of standards to improve the quality and relevance of research. 30^,31 These standards focus on the contextual variables affecting the delivery of an intervention. The link between knowledge translation and generalizability should be further explored to ensure that attributes identified as important by decision-makers in health care are considered by researchers. These factors include data on accessibility, the risk of adverse events, 32 cost-effectiveness and the sustainability of interventions. Relatively little attention has been paid to the sustainability of interventions in contrast with the initial implementation of a strategy for knowledge translation.