Impact Evaluation

Introduction

The principles of evidence-based medicine dictate that the best available evidence should be used to guide clinical choices (Dawes et al. 1999). It has been argued that the same standard should be applied to educational interventions to improve practice (Oxman et al. 1995, Grimshaw et al. 2004). In this regard there have been numerous studies evaluating the impact of various educational interventions, including academic detailing. A recent review of meta-analyses and systematic reviews found 26 reviews of educational interventions to improve physician practice (Bloom 2005). A dozen of these reviews dealt specifically with academic detailing/outreach. In terms of impact on physician behaviour, the majority of the reviews concluded that the impact of academic detailing could be considered moderate or high (Figure 1).

The impact on actual patient outcomes was measured less frequently but was most commonly associated with moderate impact. This review of reviews can be criticised for duplication/overlap of the same primary literature in multiple review papers. The designation of impact (high, moderate, low) is also not well defined and relies on the description given in the original review. Despite these limitations, this review provides us with insight into the overall theme in the scientific literature. Academic detailing is considered to be one of a few interventions that are capable of producing physician behaviour change and influencing patient health outcomes (Bloom 2005). In this assessment of impact evaluation of academic detailing, we will begin to look at the complexity that underlies this simple message and the challenges of impact assessment in the real world.

Method

The review of impact evaluation was informed by two main sources. The first source was a brief review of the scientific literature. The second sources was a series of surveys and follow-up interviews with eight academic detailing organizations (5 in Canada, 2 in the United States and 1 in Australia). These programs were:

Academic Detailing Service (ADS) – Nova Scotia, Canada
Alberta Drug Utilization Program (ADUP) – Alberta, Canada
BC Community Drug Utilization Program (BC CDUP) – British Columbia, Canada
Prescription Information Services of Manitoba (PrISM) – Manitoba, Canada
RxFiles – Saskatchewan, Canada
Drug and Therapeutic Information Service (DATIS) – Kentucky, USA
Independent Drug Information Service (iDiS) – Massachusetts, USA
National Prescribing Service (NPS) – Sydney, Australia

Results Part 1: Exploration of the Scientific Literature on Impact Evaluation of Academic Detailing

Outcome Measures

A variety of outcome measures can be used to quantify the impact of academic detailing interventions. These outcome measures can be based on the physician, the patient or the process.

Physician Based

Academic detailing is an educational intervention with the ultimate target of creating a change in behaviour. A change in knowledge, however, may form an important antecedent required to support change in behaviour. In some cases, this change in knowledge has been the outcome assessed by researchers. This has taken the form of direct assessment of knowledge with pre and post-intervention questionnaires (Geriatrics General Knowledge Test – Wong and Lee 2004, Evidence-based Medicine – Marke et al. 2001, PSA testing – Weller et al. 2003). At times, attitudes towards the topic or the required behaviour change are assessed (Marke et al. 2001). Physicians have also been asked to use visual analogue scales to self-assess their own competence of an area prior to the academic detailing intervention (Young et al. 2002).

Most studies have recognised that knowledge does not equate to behaviour change and have made efforts to (more) directly measure physician behaviour. In many cases, the target behaviour is the prescribing of medications. Early efforts made use of paper duplicate prescriptions to track physician prescribing patterns (Gray 2006). In one case, physicians were asked to select and recruit eligible patients and collect relevant medical and non-medical data on patients (Coenen et al. 2004). It seems unlikely that this method would be universally acceptable to physicians even if the challenges of blinding and the Hawthorne effect were ignored. Audit of the medication records have also been used to gather information on physician behaviour change (Young et al. 2002). Increasingly, prescription claims databases are used to track prescribing. In some cases, this may be the accumulated records of individual retail pharmacies or collected data from a variety of retail pharmacies that is assembled by a private data company (Doyne et al. 2003). Ideally, the jurisdiction may have a comprehensive data base for prescriptions (Witt et al. 2004) that may be even be linked to an electronic health record (Siegel et al. 2003). The challenge then becomes the proper interpretation of the database to accurately reflect the complexity of prescribing variations (Maclure et al. 2006).

The target behaviour change may also involve non-prescribing behaviours like diagnostics (PSA testing – Weller et al. 2003, colorectal cancer screening – Walsh et al. 2005, Helicobacter pylori testing – Majumdar et al. 2005). Smoking cessation interventions have made use of patient waiting room questionnaires to assess physician behaviour (asking about quitting, arranging quit dates, follow up) while others have used medical chart audit to determine this information (Young et al. 2002, Goldstein et al. 2003).

Patient Based

Academic detailing seeks to provide education that provides knowledge and addresses barriers to change behaviour. Physician behaviour represents only one part of the equation, with the behaviour of patients and other healthcare professions also influencing final health outcomes. As with clinical trials for medication, surrogate markers are often used as outcomes representing the actual desired change in health status.

The DOVE (Diabetes Outreach Van Enhancement) intervention in Alberta measured HbA_1c,blood pressure and total cholesterol as surrogates for complications of diabetes (Majumdar et al. 2003). They were also able to measure increased patient satisfaction in patients whose caregivers received academic detailing. The DOVE approach, however, does move one-step closer to health outcomes than many studies.

Surrogates for health outcomes can include prescription medication records. Adherence with post-myocardial infarction therapy (ACE inhibitor, beta-blockers, statins) or anti-thrombotic use in atrial fibrillation represent examples of this approach (Van der Elst et al. 2006, Jackson et al. 2006). When databases of pharmacy prescription records are used, they reflect both physician prescribing behaviour and patient adherence in filling (and presumably taking) the prescription. Physiological or medication prescribing surrogate measures are often necessary in the evaluation of academic detailing because of the longer time course required to measure health outcomes directly.

In some case, there have been attempts to measure the actual outcome of interest for shorter-term events. Symptom diaries were used by patients in an intervention to reduce antibiotic use in acute cough (Coenen 2004). These symptoms were used as supporting evidence to indicate comparable health outcomes with less antibiotic use. While this may demonstrate the lack of harm, the reduction of antibiotic resistance with interventions to decrease inappropriate antibiotic use has been more difficult to demonstrate (Finch et al. 2004). Likewise, interventions to reduce the use of antidepressant with high anticholinergic side effects have failed to demonstrate population level reduction in these types of side effects (Van Eijk et al. 2002). Self-report symptoms, satisfaction with care, OTC use and health related quality of life (Medical Outcome Study 12 - Item short from Health Survey - http://www.sf-36.org/tools/sf12.shtml) could represent an important supplement to other hard outcome measures (Ofman et al. 2003). While it may difficult to demonstrate patient health outcomes improvements with academic detailing interventions, this remains the ultimate evaluation for these interventions.

Process Based

The process of academic detailing lends itself to a number of quantitative and qualitative assessments. The length of the visit, the travel time and waiting time have been recorded in some studies (Moser et al. 2002, Polinski et al. 2005). The cost of producing academic materials has also been measured (Moser et al. 2002). When a variety of academic detailing materials are available the actual materials used in the visits can be recorded. Requests for further materials, follow-up calls and questions can also be tracked as potential measures of the utility of the visits (Polinski et al. 2005). Many of these measures can be used to develop costing of the academic detailing intervention. Combined with a costing of the resulting health outcomes can give a more complete assessment of the cost effectiveness of academic detailing interventions (Mason et al. 2001, Ofman et al. 2002, Stone et al. 2005).

Detailer effectiveness can also be tracked by measuring the percentage of target physicians they are able to see. Physician impressions of the visit have also been assessed using satisfaction surveys using a visual analogue scale (Jackson et al. 2004-see Appendix 33). Qualitative methods can also be used to solicit the opinion of physicians or their patients with relative quotations and themes assembled for consideration (Moser et al. 2002).

Evaluation Design

It is commonly considered that blinded randomised controlled trials represent the pinnacle of the evidence-based hierarchy (Figure 3). Evaluation of academic detailing interventions is held up against this standard. Most would agree, however, that it is not appropriate to consider only evaluations by randomised controlled trials given the complex nature of educational interventions (Eccles et al. 2003). A variety of other research methods have been used in the evaluation of academic detailing. This section provides an overview of research designs used in the published literature of academic detailing evaluation.

Randomised Controlled Trials

Randomised controlled trials (RCT) have been used in the evaluation of academic detailing. The complexity of educational interventions complicates the research design and simple RCT design is only possible when the level of randomisation and the level of analysis are the same. This can occur when physician knowledge is the outcome measure. Wong and Lee (2004) randomised physicians and assessed changes in geriatric knowledge in pre and post-test using a RCT design. Similar methods were use in Australia when assessing changes in physician knowledge of evidence-based medicine (Markey and Shattner 2001).

Cluster Randomised Controlled Trials

Cluster randomisation involves randomisation at one level of an organisation (e.g. physician, clinic, region) and evaluation of outcomes at a lower level (e.g. patients). This scenario is very common in the evaluation of academic detailing interventions. With clustering, patient outcomes cannot be considered statistically independent measures and failure to consider this in the analysis results in over-estimating the precision of the study outcomes (Campbell and Grimshaw 1998, Klar and Donner 2001). Randomisation at higher organisational levels is necessary to avoid the potential contamination between clusters. Since individual physicians see multiple patients and the educational interventions typically occur at the physician level, patient level randomisation is obviously impossible. Physicians also tend to work in clinic groups and sharing of information within the clinic make educational contamination a threat to study integrity. Physicians also have a preference to provide the same level of service within a clinic group (Maclure 2006).

For these reasons, randomisation often occurs at the level of the clinic (Young et al. 2002, Ofman et al. 2003, Awad et al. 2006). For practical reasons higher level randomisation may sometimes be necessary. Witt et al. 2004 randomised by CME group (clinics grouped together to receive CME where the intervention was administered). The decision is not without cost since the higher the level of randomisation, the larger the cluster size and the greater the number of patients required in the sample size for a given intra-cluster correlation (ICC). It is therefore important to consider the ICC in the calculation of the required sample size and in the analysis of study outcomes. The use of balanced incomplete block design (Young et al. 2002) perhaps with staggered randomisation delays (MacClure 2006) may enhance the efficiency of cluster RCTs by allowing for the testing of multiple interventions.

Quasi-experimental Designs

Before and After Studies

Some studies simply monitor outcome measures before and after an intervention. Given the potential changes over time in the outcome measure unrelated to the intervention, interpretations of these studies are problematic. Without a control group, the data is essentially meaningless (Finch et al. 2004). Beyond the uncontrolled, before and after study, there are a broad range of studies that use the before and after methodology. The intervention may be non-randomised with controls from other geographic areas (Siegel et al. 2003, Jackson et al. 2004). Randomisation may be used to select one of two health regions for control or intervention (Majumdar 2003). With this high level randomisation, examination of baseline characteristics of the region’s populations is essential as randomisation at this level cannot be expected to control pre-existing population differences. Provincial or state programs may choose to divide the entire state/province into control and intervention areas. Design delays may be used to ensure that control areas eventually receive the intervention education (Goldstein et al. 2003).

Interrupted Time Series Analysis

Interrupted time series analysis is a variation on the before and after design where the outcome measure is tracked over time. As with before and after studies, randomisation may not be present when larger program-based geographic areas are used (Finch et al. 2004, Majumdar et al. 2005). Many studies however, do make use of randomisation and controls to enhance study quality (Witt et al. 2004, Doyne et al. 2004). It is possible to analyse the effects of the intervention over time while taking into account other correlations in the data at different points in time. A major problem with interrupted time series analysis is the lack of an adequate data set with pre-intervention data typically lacking.

Qualitative Techniques

Qualitative research techniques may be useful for ascertaining the determinants of practice, barriers to practice change and gathering information on the experience of academic detailing (Finch et al. 2004, Moser et al. 2004).

Continuous Quality Improvement Models

While rigorous randomised evaluation will remain an important part of the evaluation of educational interventions, it may be necessary to incorporate other models for evaluating impact and change. Continuous quality improvement models may be a more practical approach to system-based implementation of education initiatives (Finch et al. 2004). The “Plan, Develop, Implement, Assess” approach may allow for system-wide movement towards established quality targets while assessing the impact of educational interventions.

Results Part 2: “Real World” Evaluation Methods for Academic Detailing

Academic detailing services have to balance the requirements of service delivery with the need to conduct impact evaluation. It can be difficult to meet the needs for material development and the co-ordination of academic detailing delivery with the incorporation of rigorous impact assessment. The result is an inversion of the focus of the data seen in the published data. The published data focussed on outcome measures associated with patients and physicians with much less focus on process. Our survey of Academic Detailing organisations has shown a heavy focus on impact evaluation of the process and much less evaluation of physician or patient outcomes. In this environmental scan of eight academic detailing organisations, we will look at how evaluation is conducted in the “real world”.

Academic Detailing Process Level

Evaluation can be focused at the level of the academic detailer. Basic information collected around academic detailing visits can be used to address a variety of questions.

Typical questions include:

What percentages of physicians are visited by detailers in a given region, territory or province?
How many physicians does a detailer see in an average day/week/month?
What does the program cost to run per month, per visit?
How many contacts are made with physician offices between academic detailing visits (scheduling appointments, answering follow up questions, providing additional materials)?
How much time is spent in developing and producing academic detailing materials?
How much time is spent in training per detailer per visit?
How much time is spent on booking appointments, travel, preparation, and actual visit time?
What is the cost of each academic detailing visit?
Usefulness of specific program resources?

Some organisations with multiple detailers spread over a wide geographic area have plans to use some of these measures as indicators of detailer performance or to guide spending levels of detailers (Appendix 34). Many organisations have a tracking form to allow detailers to collect activity and educational encounter information while in the field. Examples of these tracking forms can be found in Appendix 35.

The Alberta Drug Utilization Program (ADUP) and the National Prescribing Service (NPS) in Australia use this type of information as part of a more comprehensive assessment of their programs (NPS, Appendix 36) (ADUP, Appendices 10 and 37).

The NPS, as a national program, brings some unique perspectives and approaches to evaluation. They have developed activities and participation databases, which track their activities and the level of participation of the countries healthcare professionals. This information is supplemented with readership surveys for each of their individual publications. In many ways, much of this is just a larger scale version of what (Canadian) provincial or local programs are performing.

NPS also supplements their survey-based data with externally performed in-depth interviews with stakeholders. This in-depth interview format is not currently part of the evaluation of academic detailing in Canada.

Physicians Level

As the primary target of academic detailing initiatives, physicians represent an important impact target for academic detailers. Many organisations provide physicians with some form of satisfaction survey to evaluate their material and their program. In general, these surveys are considered relatively easy to perform but some organisations caution against excessive physician burden.

Return rates for these surveys vary and some offer prize incentives for completion (Appendix 38). Some organisations evaluate their programs after each topic (ADUP, ADS), while others do periodic surveys every one (RxFiles) to two years (BC CDUP).

Surveys have also been used to assess written materials provided to other healthcare professional (Appendix 39).

These surveys serve several purposes. They may be helpful in maintaining the requirements for CME accreditation (Appendices 11 and 40). For programs that cover a wide geographical area, they can give an indication of the performance of the program and the detailer in specific regions (Appendices 35 and 40). They can also gauge opinion on the usefulness of the overall program (Appendices 38 and 40). These surveys have been used to evaluate the acceptability of a new service or potential alternative delivery mechanisms (Appendices 11, 38, 40). Several examples of these surveys are included in Appendices 10, 11, 33, 34, 37, 38, 39, 40, and 41. Surveys have occasionally been used to assess physicians’ post-intervention intentions to change behaviours (Appendices 10 and 37). Most surveys combine quantitative Likert scales with an opportunity to provide descriptive qualitative comment. While most of the programs did not directly test physician knowledge, one program provided CME post-tests for physicians and kept records of the number of physicians who completed these tests (Appendix 34). ADUP evaluates some of its interventions with aggregate data following the provision of individualised comparative prescribing feedback to physicians (Appendices 10 and 37).

The NPS is the only national program evaluated (Appendix 36). It conducts national surveys of the physician assessing not only the credibility of their organisation and products but also the knowledge and attitudes of physicians on prescribing behaviours. They are tracking key message-related prescribing behaviour outcomes in the overall population of physicians. They have also conducted pre and post-intervention assessment of physicians’ knowledge and attitudes. In addition to their work with physicians, NPS also survey pharmacists for their perceptions and utilisation of NPS product.

Patients Level

The goal of academic detailing is often to influence physician behaviour to optimise medication use and improve patient outcomes. Frequently medication usage is used as a surrogate for more direct patient outcomes. Actual medication use results both from physician prescribing behaviour and from patient participation in medication use.

Some programs have made use of their provincial drug databases to evaluate the impact of their services. Quasi-experimental designs with pre and post comparisons of drug use and cost have been used a number of times in RxFiles evaluations (Appendices 11 and 40). RxFiles makes comparisons to non-random concurrent controls from areas without detailing. Province-wide distribution of RxFiles written materials presents a challenge in obtaining controls and interpretation of this data.

BC CDUP has made use of pragmatic non-random concurrent comparison with a matched health region to gauge the impact of their detailing program (Appendix 42). They have gone further in designing a randomised intervention with internal delays/cross-over and external concurrent comparisons. This greater degree of rigour in evaluation has come at a cost. This cost has been in terms of financial expenses and human resources, but also in the long delays required to get ethics approval and access to clean data. Preliminary data also suggest that sample size was a challenge in assessing the effect size seen with the intervention (Appendix 43).

ADS in Nova Scotia, in association with the Drug Evaluation Alliance of Nova Scotia (DEANS) program, has evaluated the impact of their osteoarthritis academic detailing program using propensity scores/regression analysis. They are currently evaluating the impact of providing physicians with a list of patients that might benefit from review related to academic detailing material provided on statins. Nova Scotia has attempted design delays and will consider design delays to support future evaluation. These design delays (i.e. providing academic detailing to physicians in two distinct waves) allow for a randomised evaluative template to be superimposed over detailing practice.

The ADUP has assembled comprehensive documents outlining the evaluation of their multifaceted academic detailing interventions (Appendices 10 and 37).

At its core, aggregates of Alberta Blue Cross data claims associated with test physicians in the intervention region were compared with control physicians in another region. This quasi-experimental non-randomised pragmatic design is similar to the basic comparisons used by BC CDUP.

PrISM’s efforts at evaluation have been limited to a cluster randomised controlled trial in the Rural Evaluation of Academic Detailing (READ) Study. Like B.C., Manitoba will have to deal with ethics, consent, privacy, and database issues to extract the data from this intervention.

The Australian NPS program does direct assessment of consumers’ attitudes and knowledge around prescription drug use. They focus on issues they have addressed in their interventions that have included public components. In addition to knowledge and attitudes, they make use of the Medicare Australia database to monitor the effect and cost savings from their interventions. Database and economic modelling are used to project the saving of the program. The evaluation work is done by a separate unit of the organisation and much of the research studies are funded separately from the service work.

The American based organisations in Kentucky and Pennsylvania also have plans for evaluation. The iDiS (Harvard group) plans to use interrupted time series analysis with historical and concurrent controls to evaluate their efforts in Pennsylvania (Appendix 34). The DATIS group in Kentucky avoids direct means of evaluation and places service and physicians relationships at the forefront of their planning. They rely primarily on indirect evaluation of the service by recording GP retention and observations and anecdotal information gathered by detailers.

Evaluation and Resources

Part of a “real world” discussion on evaluation of academic detailing must include an assessment of the resources needed to support evaluation. For many programs the conflict between program delivery and evaluation was evident. Some programs, like DATIS, have chosen to focus their evaluation primarily on areas related to service delivery. Many other programs also perform this type of evaluation. It is customer centred and relatively inexpensive to deliver. A strengths of some Canadian jurisdictions is the availability of comprehensive pharmaceutical databases. A number of programs have made pragmatic use of these databases for their evaluations through strategic partnerships with government agencies. More rigorous evaluations have usually required ethics and privacy approval. In some cases, this type of evaluation has taken years to complete. There is also often a cost associated with data access and programming. Both the time invested, lag time to obtain data and the cost of data access have been identified as barriers to evaluation.

Results Part 3: Data Sources

Process related data collection of operational field information is an important source of data in evaluating academic detailing interventions. Combined with objective measure of the level of coverage of physicians, this data can provide a relatively comprehensive measure of academic detailing cost and program participation. Participation rates are the foundation of any further assessment at the level of the physician or in terms of patient outcomes. The majority of jurisdictions collect this kind of data. It is often collected using simple spreadsheets completed by the detailer in the field (Appendices 34 and 35)

then collated and analysed centrally by program co-ordinators.

In most jurisdictions, general surveys or post-interventions surveys are viewed as reasonable sources of evaluative data. These surveys can provide information on the perceptions of the academic detailing visit and an assessment of the relevance and usefulness of the materials provided. More detailed questionnaires can also assess physicians’ knowledge or attitudes toward prescribing behaviour. Ideally, pre and post-intervention surveys would provide the most information but all organisations cautioned against overburdening time strapped primary care physicians.

No Canadian program has surveyed patients but most do not routinely provide patient information. If patient information were provided, some mechanism to have patients evaluate this material would also be desirable. NPS does national consumer surveys but their patient interventions are on a much larger scale. Collection of patient level data may mean a requirement to obtain patient level consent (Majumdar 2003, Young et al. 2002, Coenen 2004).

Databases for Evaluation of Academic Detailing

Patient level prescribing data measures the actual change in the target prescribing behaviour. Actual patient outcomes can in turn measure the impact of changes in prescribing. The collection of this data is specific to each jurisdiction.

BC CDUP - British Columbia

Like Saskatchewan, the BC CDUP program in British Columbia has made use of retrospective Pharmacare data to provide evaluative information for their program. This higher-level data was only used to compare a region with academic detailing service with another comparable region without the service. B.C. has attempted more rigorous evaluation using a more experimental design and obtaining outcome data from a variety of linked databases (Appendix 43). These databases include the B.C. PharmaNet for medications, B.C. Medical Services Plan Fee for Service Billings for physician visits, Discharge Abstract Database for hospitalisations, and B.C. Vital Statistics for deaths. This process although fruitful, has taken a great deal of time to complete analysis and get through all of the required committee structures to access the data.

ADUP - Alberta

The ADUP in Alberta has made extensive use of its available drug databases. The Alberta Blue Cross Group (1, 66 and 66A) houses the provincial drug program. Alberta Health and Wellness has archive tapes that contain the assessment of drug claims. Links to physician diagnosis, lab test requests and hospitalisations are possible. The Alberta database does not contain a comprehensive listing of all prescriptions products used by all Alberta residents.

RxFiles - Saskatchewan

With the assistance of Saskatchewan Health and the University of Saskatchewan, College of Pharmacy, the RxFiles group submits a memo requesting specific information from Saskatchewan Health (Appendix 44). The Saskatchewan database, used extensively in pharmacoepidemiological study, contains a relatively complete record of prescriptions filled by the residents of Saskatchewan. Data is available province-wide, by region, or by physician but is provided in aggregated form. The aggregation must be such that all areas contain at least five physicians. This aggregation has prevented the need for individual physician consent. The data used by RxFiles from provincial drug database and has not been linked with other patient outcome data. Linked data is available in Saskatchewan but only with greater financial and time resource.

PrISM - Manitoba

Manitoba’s Pharmacare system captures all prescriptions filled by Manitoba residents. It may be possible to receive rough high-level data via direct requests through the Pharmacare program. This was done in B.C. and Saskatchewan but the process for this is not yet well established in Manitoba. More sophisticated data or linked data would have to be obtained through the Manitoba Centre for Health Policy. This process (as described by B.C.) requires a great deal of time and the approval of both ethics and privacy committees.

ADS - Nova Scotia

The ADS in Nova Scotia has made use of Pharmacare data both in operations and evaluation. Physicians have used fax back forms to get Pharmacare to release the names of patients relevant to a particular intervention. The Drug Evaluation Alliance of Nova Scotia (DEANS) is helpful in bringing various players to the table to facilitate access to Pharmacare data. The access to linked data is more difficult (as it is in most jurisdictions) and must be accessed through the Population Health Research Unit at Dalhousie University. Longer time delays are also expected with this process in Nova Scotia.

iDiS – Massachusetts (Harvard)

The iDiS program is a contracted service for the Pharmaceutical Assistance Contract for the Elderly (PACE) program in Pennsylvania. This program tracks expenditures for patients with-in the program. The iDiS group will use this claims data to perform interrupted time trend analysis before and after their academic detailing intervention. There is no current indication that this data is linked to other health outcome data.

NPS - Australia

The National Prescribing Service is able to use the Medicare database to evaluate its interventions. They use scrambled aggregated data to track changes in prescribing on an ongoing basis. Unfortunately, their system does not track lower cost prescription products for which the government does not provide a subsidy (i.e. below co-pay). The Pharmacy Guild provides an ongoing annual survey of the data from a representative sample of community pharmacies to access data for some of these lower cost products. By combining this data with the Medicare database information a relatively complete picture is obtained. NPS also makes use of data from a General Practice Research Network to obtain data at the level of primary care physician. They also use data from the Bettering the Evaluation and Care of Health (BEACH) that has 1000 physician each year provide detailed reporting on 100 consecutive patient encounters.

Results Part 4: Relationship, Evaluation and Intervention

Evaluation should be planned during the development of the intervention and can even trace its routes back to the needs assessment. The gap analysis and the analysis of the required behaviour change will provide important insight into the required evaluation components.

Process data related to academic detailing visits can provide basic encounter information (time of visit, etc.) and may allow for detailers to provide some impressions of process. This data will help with program management, costing and general comparative information on the academic detailing encounter (short visits – not enough detail, very long visits – too much material for one visit). This information should be collected for all interventions because it is the least intrusive and does not add to the time burden for physician targets of the intervention. Ultimately, the best measure of service satisfaction may not be a survey but the percentage of physician coverage and the continuance of physician participation.

Evaluation feedback forms following academic detailing visit are an important source of information on the impression of physician of the program and materials. They can also include some questions to assess physicians’ knowledge, attitudes and intentions related to the target behaviour. Almost all academic detailing organisations do some type of survey with their physicians. The frequency of these surveys varies from; after each intervention, to once every one to two years. Some caution that evaluation surveys are an imposition that may act as a barrier to the rapid and harmonious development of detailer-physician relationships. This may be a particular problem early in the academic detailing service. There may also be real limits in relating the value of program to satisfaction surveys, which tends to be unimodal and skewed to the positive. It would, however, seem reasonable to include this type of evaluation with some interventions depending on the maturity of the academic detailing service (Appendices 10, 11, 33, 34, 37, 38, 40, 41 provide examples).

If patients’ decisions are an important part of the health outcome, the materials should be developed to support these decisions. The effectiveness of these materials can be evaluated directly with patients or indirectly by an assessment of prescribing or health outcomes. The NPS in Australia has provided a model for public involvement in the process of appropriate drug use. The assessment of patient intentions, knowledge and attitude would be important in many interventions but particularly those with a great deal of patient involvement in the process (OTC component to care, antibiotics, product with high level of direct to consumer advertising).

Some interventions will have tangible outcomes that can be assessed from prescription databases alone (Appendices 11 and 40). In other circumstance, we may wish to measure actual health outcomes or their surrogates (Majumdar et al. 2003). In many cases there are multiple indications for a medication and linkages to diagnosis is required to assess the impact of academic detailing. In many circumstances, a variety of clinical information is required to establish the appropriateness of prescribing. For example in determining the appropriateness of statin prescribing, one must consider whether the patient is using the medication for primary or secondary prevention, what drug should be chosen, what dose should be chosen, what the patient’s lipid levels are and what the target levels should be. If linked data are required it is likely there will be increased cost and significant delays in obtaining the data.

Topic selection and key message selection should provide some consideration of potential methods for outcome assessment. All organisations have indicated a need for balance between the rigours of experimental design and the practicality of service delivery. It may be the strength of detailer-physicians relationships that determine the efficacy of detailing efforts. It may be difficult to randomise these relationships in a trial. There continues to be a place for rigorous academic trials of detailing services but pragmatic trials may have an important role. Quasi-experimental designs with analysis techniques like interrupted time trend analysis may be easier for programs to achieve. Continuous quality improvement models may also be worthy of consideration. The repeated monitoring used by the NPS in Australia provides an example of this type of approach.

Discussion

Importance

Evaluation is an important part of any intervention to affect change. The literature provides strong support for the effectiveness of academic detailing (Bloom 2005) but not all interventions have been successful (Walsh et al. 2005). There is also the possibility of publication bias with primarily positive trials being published (Witt et al. 2004). There are still important questions related to the magnitude of the effect (Grimshaw et al. 2004). Evaluation should be part of the consideration in the initial planning of an intervention.

Interviews with detailing organisations have made it clear that they recognise the need for evaluation but have to balance that against the need for service delivery.

Content and Format

Evaluation can occur at many levels. The toolkit for academic detailing should contain potential tools to provide evaluation at a variety of levels. Tools to monitor and collect field related measures by academic detailers (Appendix 35)

could be an important first step in evaluation. In addition to providing the foundation for detailer evaluation and the economic assessment of the program, they can provide an objective assessment of the program impact. Physician coverage (% of physicians participating) and physicians continuing with the program provide the most important objective assessment of physician satisfaction. If impact is related to physician relationships then this is an indirect measure of this relationship. Physicians will not continue to participate if the program is not seen as valuable.

Physician surveys represent another potential tool for evaluation. They represent an added burden on physicians and must be used judiciously. They can provide a direct assessment of physicians’ opinions of the service and academic detailing materials. In some cases they can also be used to assess physicians’ knowledge, intentions and attitudes. Most organisations have used this type of assessment (Appendix 10, 11, 33, 34, 37, 38, 40 and 41).

If patient level materials are developed then direct assessment of patient opinions regarding these materials is possible. As with physicians, patient knowledge, intentions and attitudes could also be assessed. Relatively few of the organisations studied regularly use patient materials and most have not assessed patients directly. The NPS in Australia is the only organisation to do this regularly (Appendix 36).

Outcome data in terms of surrogates (prescribing data, lab data) or actual health outcomes (reduced MI rate) can be considered as part of an evaluation program. There are a wide variety of measures and approaches to this type of evaluation. They range from non-experimental through quasi-experimental designs to full cluster randomised controlled trials. Outcome data may be overall prescribing data from a particular area or may be de-identified aggregated data linked to the physicians served by the academic detailing services. In some cases, the outcome data maybe linked to a variety of health databases to provide a more complete and accurate picture of the interventions effect (Appendix 43).

There is an increasing level of cost in resources and time with more sophisticated experimental designs and linked database analysis. While there have been attempts to impose pragmatic quasi-experimental designs (paired randomisation with design delays) on the Canadian academic detailing programs there have been some operational challenges in achieving this approach. There is no standard content or format for this level of evaluation. Consideration to this level of outcome evaluation should be considered in the planning stages of all interventions. Individual adaptation to any outcome evaluation plan will be required to ensure its feasibility to a particular academic detailing service and the data sources they have available.

Process

Evaluation should be a consideration as COMPUS assembles its toolkit to support academic detailing interventions. There should be some evaluative tracking implicit in the release of the COMPUS produced academic detailing toolkit. Organisations could track the what, where, how, and who for the toolkit materials used. An assessment of low impact (passive diffusion) and high impact (academic detailing) distribution would also be helpful. This utilisation data would provide basic information on which components to the tool kit where considered helpful and some assessment regarding the level of national physician coverage achieved with the program.

It may also be important to prepare a survey for detailers to assess the value of the toolkit materials. Much of what happens in the field is captured by the general impression of detailers. A COMPUS prepared survey for detailers containing basic demographics (# of physicians seen, region, etc.) and their impressions of the value of the materials would be of value. Detailer feedback could represent an important source of information for continuous quality improvement of COMPUS products.

It may not be possible to impose a physician level survey on the academic detailing programs. Existing organization will have an established survey format and frequency. New organisation may not wish to burden physicians as they seek to establish relationships. Some organisations, however, do provide a feedback survey after each intervention. A physician survey containing a variety of sample questions assessing the value of detailing materials, physician knowledge, intentions and attitudes could be a useful resource for detailing organisations. The questions would likely be treated as a bank with questions pulled or adapted to reflect the final local format of the detailing intervention. As a courtesy, academic detailing organizations could be requested to share the aggregated final results of their surveys with COMPUS.

Patient materials may represent a substantial portion of the academic detailing toolkit provided by COMPUS. Consideration could be given to the development of a survey to provide feedback from patients on the value of the materials. Patient knowledge, intentions and attitudes could also be assessed. Although this type of survey has not been part of the academic detailing programs in Canada, it may be helpful as a resource to move organisation towards this type of assessment.

Outcome assessment in terms of linked or un-linked prescribing data will be an important component of the evaluation of academic detailing efforts. The heterogeneity of academic detailing programs and provincial databases make it difficult to impose a single approach to outcome assessment. Outcome measures, however, should be a consideration in the designing of academic detailing interventions. At a minimum, COMPUS could consider producing a menu of potentially measurable outcomes for the consideration of provincial jurisdictions. In time, it may be possible for COMPUS to track national trends in prescribing to assess the impact of their interventions. This will only be important if there is significant national uptake of these interventions. This will only be possible if there is a national collection of provincial prescribing data (National Prescription Drug Utilisation Information System – NPDUIS). A wide variety of examples of experimental design for evaluation have been presented in this document. Local programs, structures, databases and resources will need to be considered in the selection of a suitable design for evaluation.

Conclusion

Consideration for Evaluative Components in an Academic Detailing Toolkit

Evaluation can occur at many levels and will often need to be tailored to the final format of the detailing intervention. Outcome measurement must also consider the database landscape of the individual jurisdiction. Despite these constraints, several evaluative components could be considered as part of an academic detailing toolkit.

A basic utilization form that captures the usage of the individual toolkit components. It should also capture the basic demographics of the number of physicians and other healthcare professionals that have participated in academic detailing sessions.
A survey for detailers to complete on their impressions of the acceptability and usefulness of the academic detailing materials.
An optional survey for physicians assessing the value of the academic detailing materials. While self-contained, this survey could also function as a bank of possible questions for organizations with an existing physician survey format.
A survey for patients assessing knowledge, acceptability and satisfaction with patient information materials included in the toolkit.
A menu of potentially measurable outcomes related to the target of the academic detailing intervention. These suggestions could be considered in the rollout of the intervention and adapted to the databases available in the jurisdiction.

Literature Cited

Awad, A.I., Eltayeb, I.B., Baraka, O.Z. 2006. Changing antibiotic prescribing practices in health centers of Khartoum State, Sudan. 62:135-142.
Bloom, B.S. 2005. Effects of continuing medical education on improving physician clinical care and patient health: A review of systematic reviews. International Journal of Technology Assessment in Health Care 21:380-385.
Campbell, M.K., Grimshaw, J.M. 1998. Cluster randomised trials: time for improvement. The implications of adopting a cluster design are still largely being ignored. British Medical Journal 317:1171-1172.
Coenen, S, Dirven, K., Michiels, B., et al. 2005. Implementing a clinical practice guideline on acute cough in general practice: a Belgian experience with academic detailing. Medecine et maladies infectieuses 35:S97-S99.
Coenen, S., Van Royen, P., Michiels, B., et al. 2004. Optimizing antibiotic prescribing for acute cough in general practice: a cluster-randomized controlled trial. Journal of Antimicrobial Chemotherapy. 54:661-672.
Dawes, M., Davies, P., Gray, A., et al. 1999. Evidence-based practice: A primer for health care professionals. Churchill Livingstone, London.
Doyne, E.O., Alfaro, M.P., Siegel, R.M. et al. 2004. A randomized controlled trial to change antibiotic prescribing patterns in a community. Arch Pediatr Adolesc Med. 158:577-583.
Eccles, M., Grimshaw, J., Campbell, M., et al. 2003. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 12:47-52.
Finch, R.G., Metlay, J.P., Davey, P.G., et al. 2004. Educational interventions to improve antibiotic use in the community: report from the International Forum on Antibiotic Resistance (IFAR) colloquium, 2002. The Lancet Infectious Diseases 4:44-53.
Goldstein, M.G., Niaura, R., Willey, C., et al. 2003. An academic detailing intervention to disseminate physician-delivered smoking cessation counselling: smoking cessation outcomes of the Physicians Counselling Smokers Project. Preventative Medicine 36:185-196.
Gray, J. 2006. Changing physician prescribing behaviour. Can J Clin Pharmacol 13(1):e81-e84.
Grimshaw, J.M., Thomas, R.E., MacLennan, G., et al. 2004. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technology Assessment 8(6): 1-72.
Jackson, S.L., Peterson, G.M., and Vial, J.H. 2004. A community-based educational intervention to improve antithrombotic drug use in atrial fibrillation. The Annals of Pharmacotherapy 38:1794-1799.
Klar, N., Donner, A. 2001. Current and future challenges in the design and analysis of cluster randomized trials . Statistics in Medicine 30:3729-3740.
Maclure, M., Nguyen, A., Carney, G., et al. 2006. Measuring prescribing improvements in pragmatic trials of educational tools for general practitioners. Basic and Clinical Pharmacology & Toxicology 98:243-252.
Majumdar, S.R., Guirguis, L.M., Toth. E.E., et al. 2003. Controlled trial of a multifaceted intervention for improving quality of care for rural patients with type 2 diabetes. Diabetes Care 26:3061-3066.
Majumdar, S.R., Ross-Degnan, D., Farraye, F.A., et al. 2005. Controlled trial of interventions to increase testing and treatment for Helicobacter pylori and reduce medication use in patients with chronic acid-related symptoms. Aliment Pharmacol Ther. 21:1029-1039.
Markey, P. and Schattner, P. 2001. Promoting evidence-based medicine in general practice – the impact of academic detailing. Family Practice 18:364-366.
Mason, J., Freemantle, N., Nazreth, I., et al. 2001. When is it cost-effective to change the behaviour of health professionals? Journal of the American Medical Association 281:46-52.
Moser, S.E., Dorsch, J.N., and Kellerman, R. 2004. The RAFT approach to academic detailing with preceptors. Family Medicine 36:316-318.
Ofman, J.J., Segal, R., Russel, W.L., et al. 2003. A randomized trial of an acid-peptic disease management program in a managed care environment. The American Journal of Managed Care 9:425-433.
Polinski, J.M., Brookhart, M.A., Katz, J.N., et al. 2005. Educational outreach (academic detailing) regarding osteoporosis in primary care. Pharmacoepidemiology and Drug Safety 14:843-850.
Siegel, D., Lopez, J., Meier, J., et al. 2003. Academic detailing to improve antihypertensive prescribing patterns. American Journal of Hypertension 16:508-511.
Van Eijk, M.E.C., Belitser, S.V., Porsius, A.J., et al. 2002. Evaluation of patient outcomes in an area where prescribing of anticholinergic antidepressants was influence by academic detailing. Pharm World Sci 24:144-148.
Walsh, J.M., Salazar, R., Terdiman, J.P., et al. 2005. Promoting use of colorectal cancer screening tests. Can we change physician behavior? J Gen Intern Med. 20:1097-1101.
Weller, D., May, F., Rowett, D., et al. 2003. Promoting better use of the PSA test in general practice: randomized controlled trial of educational strategies based on outreach visits and mailout. Family Practice 30:655-661.
Witt, K., Knudsen, E., Ditlevsen, S., et al. 2004. Academic detailing has no effect on prescribing of asthma medication in Danish general practice: a 3-year randomized controlled trail with 12-monthly follow-ups. Family Practice 21:248-253.
Wong, R.Y., and Lee, P.E. 2004. Teaching physicians geriatric principles: a randomized control trial on academic detailing plus printed materials vs printed materials only. Journal of Gerontology 59A:1036-1040
Young, J.M., D’Este, C., and Ward, J.E. 2002. Improving family physicians use of evidence-based smoking cessation strategies: a cluster randomized trail. Preventative Medicine 35:572-583.