Appendix 6 : iDiS Questionnaire & Follow-Up

** Notes and follow-up questions from interview in italics **

Section I: Needs Assessment and Key Messages

1. Describe how Academic Detailing topics are chosen within or for your organization?

The Independent Drug Information Service (iDiS) is funded by the Pennsylvania PACE Program (Pharmaceutical Assistance Contract for the Elderly), a drug-benefit program run by that state’s Department of Aging. Clinical content is provided by Harvard Medical School physician-researchers. PACE and the Boston team work together to choose academic detailing topics, taking into consideration (1) potential topics’ relevance to PACE program goals and beneficiaries, and (2) available research data on the topic.

Topics are chosen according to their relevance to PACE Program objectives, which are to optimize medication therapy for the beneficiaries of Pennsylvania's state-supported drug programs, improve the quality of prescribing, and contain costs by providing the most appropriate medications to PA beneficiaries. PACE and the Boston team analyze PACE enrollment, claims, and claims expenditures to identify high volume claims, and assess therapeutic classes to determine areas where educational opportunities may exist to help improve appropriateness of prescribing.

Can they provide an example of an analysis of claims expenditure used to assess the needs for an intervention.

They will provide a pdf of the PACE Annual report (2004 or 2005). This report contains their expenditure per year by drug class and indicates the portion paid by the state and the cardholder.

High volume/value claims would be appropriate targets for potential interventions. The potential for alternative agents is also an important consideration in topic selection.

The availability of good clinical trial data and other research is crucial in the development of educational materials and helps inform decisions about which therapeutic topics to address, particularly in the event of a new finding that may create safety or quality concerns (e.g., withdrawal of Vioxx from the market and the resulting cox-2 inhibitors safety concerns). Our materials are designed to provide summaries of the relevant data and practice-relevant guidelines for physicians. In choosing a topic, we consider whether or not data exists, and to what extent we can assist the busy physician by providing a practice-relevant synthesis of the data.

It’s important to note that PACE and the Boston team choose the topics with an emphasis on quality of care, not just cost. So, in some instances, optimal prescribing may include recommending a more expensive drug; in other cases, it may mean recommending less expensive generics or older drugs. PACE's overarching goal is to improve appropriateness of prescribing by providing unbiased, non-commercial, evidence-based, and timely information to prescribers.

How important are cost savings in the assessment of the academic detailing program?

Too early to determine how important this will be and initial cost-benefit analysis has not yet been performed. The director of the PACE program is expecting to see some savings. There was also an expectation from PACE that the benefits would be relatively immediate. It is more likely that using time trend analysis that the changes in prescribing will occur gradually over time.

One has to take account of the existing refills patients may have available.

There is, however, some recognition that in some cases having patients continue to take a brand name product is essential and quality of care is the most important. For example Plavix is needed for certain patient populations.

With regard to the time trend analysis and the timing of the interventions – are these one time visits or are there follow-up visits or reinforcement of previous topics on next visit? What information do we have about the longevity of effect from academic detailing interventions?

Industry reps are doing just that – repeat visits discussing the same product to encourage use and have a prolonged effect on physician behavior. They are encouraging detailers to build relationship with physicians by stopping back between official visits. Even short visits to confirm an appointment time, drop off the latest materials or new articles. They would expect the greatest change in the physicians where these multiple visits foster this relationship. They try to roll out all materials at the time of the intervention but sometime experience in the field suggests additional material that may be helpful. It can be another excuse to drop by with this new material. Also drop by to see if restocking of materials is necessary. No official guideline for these visits but they are encouraged by the program.

2. Does your organization employ any formal “Needs Assessment” when choosing a topic or to identify issues within a topic that could be detailed? Please describe your process.

Our topics are chosen using an informal needs assessment process, as discussed above. This process usually begins with a discussion among key PACE officials and the Boston team. They evaluate prescribing patterns by analyzing PACE enrollment, claims, and claims expenditures to identify high expenditure and high volume claims, and assess therapeutic classes to determine areas where educational opportunities may exist to help improve appropriateness of prescribing (i.e., prevent misuse and overuse of certain medications). The Boston-based physician-researchers analyze the evidence-based data in the medical literature to review medication options, develop case studies, determine therapies that are clinically appropriate, and develop guidelines based on this evidence. Topics may also be chosen in response to a sentinel event, such as detection of a quality or safety issue, withdrawal of product from the market, or publication of new practice guidelines.

Is there any involvement of end-target (i.e. prescribing physicians) involved in the needs assessment process?

Have a budget established for physician focus groups but have not done this yet. It is part of the planning but has not been included in the initial roll out.

They have a list of high volume prescribers but may not use them for the focus group because of logistical problems. Will likely involve physicians from the Boston area. These focus groups will help to address some of the barriers and enablers for a particular topic. Hopefully Boston area physicians (despite heavy academic environment in Boston) would be similar enough to able to accurately provide this information (applicable for rural Pennsylvanian physician?). Of not Pennsylvania is about 6-10 hours from Boston – travel distance for focus groups and detailer training.

3. Describe the process used for development and refinement of “Key Messages” used in your Academic Detailing material.

Key messages are identified by the Boston team, including the project director, project managers, and physician-researchers who analyze the evidence-based data in the medical literature. They use this evidence to provide a synthesis of medication options, determine therapies that are clinically appropriate, and develop guidelines based on this evidence. When key messages are identified, they must: (1) be evidence-based, (2) imply a change in behavior/knowledge, (3) address a gap between actual and ideal practice (i.e., room for improvement), and (4), when possible, allow for some measurement of success. We have recently begun to delve more fully into the features, benefits, barriers, and enablers of the key messages (a technique we adopted after discussions with Frank May, of the DATIS program).

This further evaluation of the key messages allows our academic detailers to tailor their consultations to the individual physicians’ needs, which will result in more significant, positive behavior change. We do not yet have formal processes in place to identify key messages. But these processes are being developed in an effort to more effectively focus our messages and facilitate the desired changes in behavior.

Section II: Academic Detailer Training and Background Material

1. Who is responsible for training Academic Detailers on an AD topic and how is this training conducted?

Academic detailers are trained by the Boston team, including the project managers and the physician-researchers, led by Jerry Avorn, MD, professor of medicine at Harvard Medical School, and chief of the Division of Pharmacoepidemiology and Pharmacoeconomics at Brigham and Women’s Hospital. Academic detailers travel to Boston for the 2-3 day training sessions. The training session is held in Boston for several reasons: (1) the team has access to the physicians who develop the detail materials, (2) the setting allows for team-building opportunities among the detailers who can feel isolated from their colleagues due to the independent nature of the job and the wide geographic area they cover, and (3) the team returns home newly energized after being immersed in a high-energy, intellectually-stimulating environment.

The initial training session (at the inception of the program) included the following topics:

the history and rationale of academic detailing
healthcare landscape in Pennsylvania
program expectations (the medication educator’s role, administrative details, etc.)
factors that drive decision-making (physician behavior)
building relationships with health care professionals
“un-sales” training
sharing of best practices from experiences in the field
clinical content

introduction to clinical topic, including background of disease state, symptoms, medication pharmacology, etc.
medication use related to topic: current, ideal
overview of patient and physician materials

partner role playing
practice physician visit (taped)

The initial training only occurred once. How is it provided to subsequently hired detailers?

Detailers are flown into Boston for training – generally only one overnight to limit the cost. The ability to connect with the larger team in an academically stimulating environment is important.

Did bring on 2 new detailers after initial training – they provided some one-on-one training in Pennsylvania to compensate for missed sessions. The binders also contain all of the information and PowerPoint presentations. The presentations are all videotaped and available on disc. Increased support, follow-up and availability of Michelle for new detailers. This has also meant that some physicians have not been seen while other are on their third topic.

They are encouraging catch up but are only doing one topic per detailing session. Time trend analysis is affected but will be using 6 months prior and 6 months post intervention.

About ½ pharmacists and ½ nurses work as detailers for this program.

Subsequent training sessions have provided a review of some of the field techniques listed above, discussions on performance measurements, program evaluation, and identification of key messages. A large portion of these training sessions is devoted to training on the clinical content, including a summary of the evidence, overview of recommendations and patient scenarios, and practice visits.

2. What materials are routinely involved in the training of Academic Detailers (e.g. power point presentations on topics)? Please provide examples of training materials employed by your organization in detailer training.

Training materials include power point presentations, detailed evidence-summary documents and related handouts, patient scenarios, key research articles and other relevant clinical information. Generally, we send key research articles to the academic detailers prior to the training session and ask them to thoroughly review the articles so that they have a basic understanding of the key evidence to provide a foundation for the clinical discussions that will be provided during the training session. We also ask them to indicate if they feel that they lack information, expertise, or knowledge about a particular disease state, therapeutic class, or other clinical information so that we can prepare any needed background information.

During training, detailers are provided with binders that include power point presentations and background/summary documents that will be presented throughout the multi-day training session. Relevant key reference articles and drug information are also included in the training binder. During training, the Boston-based physician-researchers present clinical background, review the most relevant papers in the medical literature, summarize the evidence, answer clinical questions, and lead role-playing and practice patient scenarios and educational encounters. Academic detailers are encouraged to actively participate in the training session by asking questions, providing examples of comments/questions/objections they anticipate receiving in the field, providing insight on the topic and materials based on their own background and experience, and practicing educational encounters.

3. What background materials or support materials on an AD topic are provided to detailers to ensure they are well enough versed in the topic to conduct physician visits (e.g. detailed summary documents, primary literature, etc.)? Please provide examples of materials developed by your program that act as background or support materials for detailers.

See answer above and materials that have been sent to PrISM. Training materials include:

Key reference articles
Drug information
Evidence summary documents
Power point presentations that describe relevant data and evidence
Patient scenarios to be used as teaching tools in the physician encounter
Patient education materials for physicians to distribute
Samples of materials to be used during the educational encounters
Role-playing and practice sessions

4. Do you conduct any type of knowledge assessment or testing of detailers to assess their understanding of an AD topic?

Currently, we do not require detailers to complete a knowledge assessment post-test in written format; however, we do require their participation in role-playing, patient scenario, and Q-and-A opportunities, which we feel provides an adequate assessment of their understanding of the topics.

We are considering developing a more quantitative assessment of their clinical knowledge for future topics, which may encourage the detailers to delve into the provided clinical materials with more energy and a greater sense of purpose.

Where is your group in the process of quantitative assessment of detailer knowledge? Would quantitative knowledge assessment be a motivating or discouraging factor for your detailers?

Don’t want people to feel like being preached to or that their judgement is not trusted. Informal survey of detailers suggested that quantitative assessment makes sense and would be expected if you were a pharmaceutical representative. They want to be sure they have the knowledge that they think they have on a particular topic. The professional environment ensures that people know that they are not be degraded, or that their job is on the line. They will be including a quantitative post-test/assessment with their next topic.

5. Do you conduct any evaluation or assessment of the training that detailers are provided by your organization (ex. a training evaluation survey)?

To date, we have not included evaluation surveys of the training sessions, although we have considered the possibility and will likely do so following future sessions. Most of the evaluative information to date has been gathered in one-on-one conversations and/or email correspondence with the detailers following the training sessions, as well as physician survey responses concerning the program, their answers on Continuing Medical Education post-visit exams, and our own in-the-field observations of detailers’ visits with physicians (i.e., do they have enough information to effectively communicate the key messages, are they comfortable with the materials and/or message, etc).

Section III: Academic Detailing Material for Primary Care Practitioners

1. How are Academic Detailing materials (newsletters, patient information, treatment algorithms, etc.) produced by your organization?

Physician-researchers in Boston develop the academic detailing materials used by the Independent Drug Information Service. These individuals are chosen to develop educational material on topics in which they have particular expertise or interest.

These physicians begin by performing a thorough review of the medical literature to identify relevant, quality evidence, and use this evidence to develop a variety of materials as noted above. This review also provides a foundation for the development of key messages. These key messages must be evidence-based, must recommend a desired change in knowledge or behavior, and must address a gap between actual and ideal prescribing practice. In addition, it is useful to identify key messages that allow some measure of successful change in behavior (when possible).

Following the review of the literature, the physicians develop a draft document summarizing the evidence (called the “evidence document”). From this lengthy document, patient scenarios and a shorter, eye-catching educational brochure are produced (called the “un-ad”). These draft materials are reviewed by the project team in Boston, and are also distributed to the detailers and PACE officials for their review.

During training sessions, the project leaders, PACE representatives, and academic detailers discuss the materials and identify other collateral materials that may be helpful as visits are conducted (e.g., physician reference cards, patient brochures). Following these discussions, additional materials may be produced by the Boston team and reviewed by the group as well. A continuing medical education (CME) post-test is also produced.

When content is approved, the Boston team sends the materials to an independently contracted graphic designer for formatting, illustrating, etc. After multiple reviews by the team, and revisions by the graphic designer, materials are sent electronically to print vendors for printing.

Content is developed by the clinical team in Boston and then sent to the graphic designed for layout, formatting etc -- Is it possible to get an example of material in the pre-graphic design state?

Pretty simple after the initial template was developed – but making small changes in the format as they learn more about what is effective. They are looking at adding more white space and spreading out the text a little.

2. Describe the types of Academic Detailing materials produced by your organization, and provide examples of the materials.

Evidence document/monograph – This document provides an overview of the most current literature relevant to the clinical topic. Past evidence documents have been from 8 to 16 pages, depending on the topic (e.g., its complexity, controversiality, difficulty in translating the evidence into practice-relevant recommendations, etc). The evidence document begins with an introductory section, which provides a brief overview of the drug class/topic being addressed. The document also contains a summary of the relevant data with a focus on safety and efficacy, and when therapeutic classes are equivalent, on cost. The evidence document may contain graphs and figures that provide summaries of the evidence and/or that have been previously published in the medical literature (in which case, appropriate permissions are requested/granted). In addition, this document contains a comprehensive list of relevant references.

Is the evidence document provided to (or offered to) prescribers?

Everyone is providing the longer detail document during academic detailing visits. All materials are usually grouped in a folder that can be provided during the visit. Detailers are purchasing folders or using the un-ad as a folder and placing all materials and even key references in the folder. Will also include the post-test CME exam in the folder. Physicians are getting an hour of CME so the expectation is that they will spend time reading the materials – including the larger evidence document.

Where is the base for your detailer?

The detailers are independent contractors and are spread all over the state. They work out of home offices. This is another reason to bring them together for training sessions. In larger centres with multiple detailers – they may get together occasionally between training sessions.

4-page educational brochure (“un-ad”) – This document provides a more concise, eye-catching summary of the information provided in the evidence document, and has been designed as a branding (recognition) piece, utilizing a consistent format, graphics, and color scheme from topic to topic. The key messages that are honed from the evidence and described in the evidence document are highlighted as headlines in the un-ad. Illustrations, diagrams, and figures are also used to emphasize important concepts. The back page of the un-ad generally includes a summary of the recommendations, as well as a somewhat more concise list of references and a section that describes the program and instructions on how to find additional information (e.g., in the evidence document and on the Independent Drug Information Service website: www.RxFacts.org).

Continuing medical education (CME) post-test – A CME post-test is produced in an effort to reiterate the key concepts described in the academic detailing materials. This post-test must follow Harvard Medical School guidelines for an accredited CME program, and therefore, remain fairly consistent from topic to topic. Generally, the test is 10 to 12 questions in length. Questions are multiple choice, and are designed to require the health care provider to participate/engage in the educational visit.

Can you provide an electronic version of an example post-test?

Will provide an electronic copy of CME post-test.

Physician reference cards – Depending on the topic, a reference card may be created as a handy “leave behind” for the health care provider. This card may provide general dosing guidelines for the particular drug class of interest, a review of practice recommendations, and/or a short summary of the relevant research supporting the recommendations. These reference cards are relatively colorful and eye-catching, and are formatted so that, when printed, they can be kept in physicians’ lab coat pockets, on a clipboard, or in a drawer in a clinical exam room.

Patient brochures – These materials are developed to provide unbiased, noncommercial drug information in a format similar to that used by the pharmaceutical industry (i.e., visually engaging, compelling, clearly written, and easy to use). The information in these brochures is designed to aid the physician in educating their patients about the treatment decisions they make, particularly in situations where prescribing decisions may be contrary to what they patient expected or desired (i.e., as a means to counter the effects of direct-to-consumer advertising).

“Prescription” pads – The patient-focused prescription pads are similar to the patient brochures described above, but are designed to resemble a physician’s prescription tablet. These tear-off pads are intended to help physicians educate their patients about treatment decisions, lifestyle changes, or in helping explain why a patient may not be given the prescription that a patient may have expected or even asked for. For instance, we have developed prescription pads that provide general information about acetaminophen and NSAIDs, which physicians may use when they do not write a prescription for an analgesic. (Patients often ask for particular prescriptions - due to direct-to-consumer advertising - and also often expect to leave with a prescription whenever they visit their health care provider.) We have also developed a prescription pad that describes heartburn prevention and non-prescription treatment strategies, as well as a pad that describes a PPI step-down regimen. These are particularly useful in helping patients feel comfortable about leaving the office without a prescription and with instructions to not take a medication they’ve been taking previously (e.g., PPIs). These particular prescription pads include areas where the physician may fill in information while the patient is present (such as patient name, physician’s signature, schedule of steps for step-down therapy, etc), making them more “interactive” and generally more personalized for the patients whose prescription expectations may not have been met.

Has there been any consideration to the use of PDA formats or other electronic tools?

This has been discussed and they are looking into the potential of this. There is a diverse range of physicians and a broad range of acceptance of technology. Not immediate plans to introduce this format to their academic detailing materials.

3. Estimate the length of time required to produce each type of Academic Detailing material (including research, production and peer-review if applicable).

Production of materials to date has varied greatly depending on the topic and can be a relatively lengthy process, particularly because the individuals producing and reviewing the materials must balance this activity with a number of other competing priorities and time demands. If a physician is currently conducting research in a particular area, they may have already conducted a relatively thorough literature review, and may be very knowledgeable about the topic, recommended treatment guidelines, opportunities for improvements in prescribing, etc. This will cut back on preparation time. If the topic is controversial or has been chosen because of a general lack of consensus on treatment (perhaps due to lack of quality evidence, etc.), then development of materials takes more time. In general, the entire process takes from 3 to 4 months. The evidence document is the most time-consuming of the materials to produce, particularly because of the time needed to conduct a thorough review of the literature.

4. Is there any standard formatting or branding used in the AD materials your organization produces?

Branding – The Independent Drug Information Service logo is included on all materials produced by our organization. The logo includes a stylized version of a balance with a pill on one pan and a capsule on the other (color: medium and dark blue). The acronym “iDiS” is placed below the balance (color: black), and the words “Independent Drug Information Service” (color: blue) are included below the acronym. In addition, the tagline “Balanced data about medications” (color: blue) and our website address (www.RxFacts.org) are also included on all materials. We also use a consistent color scheme and format throughout (described below). In addition, the PACE program contact information and Pennsylvania Department of Aging logo are included in a number of materials to encourage co-branding and recognition of their role as the supporting/funding agency.

Formatting – The format of materials also remains consistent from topic to topic.

The evidence document is printed to resemble an academic monograph or similar publication. Its format is relatively simple: black and white print, double-sided, on white paper, finished booklet-style. The font is large enough to be easily read, but is not conspicuous or elaborate.
The un-ad is formatted to fit on a 4-page pre-printed template brochure. The brochure includes a mosaic of photos of individuals of all ages, races, and genders, designed to evoke the types of patients that target physicians regularly see. These photos are full-color, and are distributed throughout the brochure. Editable regions of the template contain the topic-specific content, in easy-to-read font weight and size (color: black). Headlines are formatted consistently, with a light-blue triangle at the beginning of each headline, and text in bold, blue, large font. Bulleted lists are included as much as possible (for ease of reading), and important phrases may be highlighted with a heavily weighted (bold) font. All figures and graphs are created using various shades of the same blue color scheme.

Could we get an electronic copy of the 4-page template of the brochure?

How did the template brochure get developed – when did the designers become involved?

The Boston team actually created the template before they had any content or involved the designers. Dr. Avorn actually found a brochure with a suitable template and it was adapted with picture boarder. They printed 20000 blank brochures for the template. Wanted it to be visually appealing but not over the top. They hired a Pennsylvania based graphic designer who used to work for industry but is now independent. The process is a compromise between the design/artistic suggestions and the requirements of the clinical content team. May be adding some more white space to the template but will not be cutting content, text or words – so will have to add pages.

They already try to boil down the message as much as possible. Bulletin created to accommodate new information that becomes available after the brochures created. This allows for additional information and gives the added impression of being very timely and current.

They will provide an electronic blank version of the existing template.

The patient scenario documents include a photo of a model representing a typical patient (top of front page) with a scenario/case description included beside the photo. The bottom 2/3 of the page includes various questions about treatment options for the particular scenario (to serve as talking points for the academic detailer). A description of ideal, evidence-based treatment decisions is included on the top half of the back page, and key references are listed at the bottom of this page. The patient scenario documents are relatively simple, although they are formatted to be eye-catching and easy to read, and are printed in full-color.

Can you provide an electronic copy of a patient scenario?

They will provide an electronic copy of the patient scenarios.

Patient brochures are formatted as tri-fold brochures (with a gated fold). The front of the brochure includes a colorful, attention-grabbing image and a general introduction of the topic. The inside flaps contain specific educational information, and the middle back flap contains branding information and images, a recommendation to talk with their physician about the content of the brochure, a warning that the material is a general overview only and that specific drug-use decisions must be made by the prescriber based on a given patient’s unique situation, and instructions on how to find additional information.
Prescription pads are formatted to resemble a physician’s prescription tablet. They are gummed tear-off pads, usually about 4” x 5”, black and white print, single-sided. The pads contain concise information, and may include check boxes or blanks for physicians to fill-in with patient information, dates of treatment, etc.

5. Are the Academic Detailing materials you produce pre-viewed by a test audience to determine acceptability and clarity to the end-user? Please describe your process.

During development, program materials are reviewed by the Boston team, which includes physicians, program managers, and research support staff. The review and revision process is described in question 1 above. In addition, we have allocated funds for focus groups, which we plan to utilize in the future.

6. How are AD materials utilized during an AD intervention (e.g. are all materials brought by detailer or are some materials mailed out ahead of time)?

All materials developed for a particular topic are used by the academic detailer during the educational visit. Detailers generally begin the encounter by presenting a folder that contains all of the materials, or they may use the un-ad as a folder. Inside, the detailer will have placed the evidence document, patient scenario documents, CME post-test, any key references, and/or patient brochures. In some instances, key references and patient brochures are presented near the end of the educational encounter, after the individual has “detailed” the health care provider.

After cordialities, the encounter generally begins with a brief discussion between the detailer and the health care provider to present an introduction to the topic at hand. The academic detailer may initiate this conversation by using the patient scenario documents, the un-ad, or nothing at all (methods vary depending on the academic detailer and the health care provider). The majority of the encounter is conducted using the un-ad as the primary educational tool. During the encounter, the academic detailer methodically introduces the topic’s key messages, pointing to the corresponding text in the un-ad. The detailer encourages the health care provider to engage in the encounter by asking questions about particular scenarios that (s)he may face (referring to the patient scenario documents), or by asking if he would like additional information about a particular concept (referring to the evidence document or key references). The patient scenario documents have also been used subsequently by some physicians as teaching tools for residents or in other educational settings.

Before concluding the encounter, the academic detailer brings attention to the evidence document (if he has not already done so) and asks the health care provider to spend some time reviewing the document at a later time. The detailer then asks the provider if he is interested in completing the CME post-test, which they can do while the detailer is present or at a later time (instructions for submission are included with the exam). After the discussion of the key messages (near the end of the encounter or possibly earlier, depending on how the discussion unfolds), the academic detailer presents the health care provider with the reference cards, patient brochures, and prescription pads. Detailers are encouraged to make return trips to the physician’s office to re-stock the health care provider’s supply of these collateral materials (which is also a helpful way to foster the relationship with that provider).

Do you have any sense of the level of use and need for restocking of materials provided?

Detailers are able to keep track of this with tracking tool provided by the program. When it is an actual educational visit, the detailers fill out an educational encounter form. This will allow the program to look at this information in more detail at a later time. May look to link detailer visit time (length and number) to prescribing results – does this make a difference in outcomes? Get a rough sense of the level use of individual materials from the requests from detailers for more materials. Bright colour patient brochures and prescription pads (ex prevent and treat heartburn) are the most popular. They will be interested in seeing the results of our overall survey.

Section IV: Impact Evaluation

1. Describe the types of Impact Evaluation conducted by your organization to assess the effectiveness of your Academic Detailing intervention. If reports or publications of your impact evaluation are available, please provide copies. Please also include any examples of informal evaluation, surveys or evaluation efforts that are in process.

The program is evaluated on three levels: academic detailer, physician, PACE program.

Academic detailer – Academic detailers are evaluated using comparisons to peers and to their own historical performance. These performance measures are used to determine where academic detailers stand on the completion curve. Expectations are that the majority of physician visits on a particular topic are completed on a roughly quarterly basis (although trends indicate that a 4-month timeframe has been more realistic). Academic detailers’ activity tracking tools and encounter forms are used to gather this type of information. In addition, in-field monitoring provides invaluable qualitative information about detailers;’ performance. Some of the performance measures evaluated include:

What percentage of assigned physicians have agreed to an educational encounter?
What is the average number of completed educational encounters per week?
What is the average total program expenses per week?
What is the average number of office contacts per week?
What percentage of office contacts results in an educational encounter?
What percentage of time in the office is devoted to the educational encounter (vs. wait time, etc)?
What is the total time vs. prep time vs. visit time vs. educational encounter time?
What is the ratio of program expenses to educational encounter?

Could we get an electronic copy of the activity tracking form?

This is an Excel spreadsheet form – they will provide a copy.

The performance measurers for detailers – do you have a summary report of these measures?

They have a report from mid-May – new tracking form in place in June – which should allow better tracking and analysis. Will send an old version and a new version so that we can see what is being changed to allow for better reporting. Were initially tracking program expenses – office supplies, copies, and lunches. Detailers are allowed to provide lunches but this is not encouraged. There is a wide variation in the amount that detailers are spending on providing lunches. Looking at sending a monthly email newsletter to all of the detailers. This will be a way to communicate with detailers, keep them inspired and informed. It could also be used to share information on average expenditures per detailer and their own expenditure (audit and feedback – for lunch expenditures).

Could share the number of educational encounters in a 40-hour week. Activity(in-between visits) versus encounters will also be examined. Mileage and completion rates will also be of interest.

Physician – Physician-level evaluations are conducted both quantitatively and qualitatively. Upon completion of an educational encounter, academic detailers submit an “encounter form” which provides descriptive information about the visit. Some of the variables that are measured include: length of educational encounter, wait time, and continuing medical education (CME) post-test completion (if known), and type of provider (physician, PA, NP, nurse, etc). Program managers also gather data about the number of CME post-tests completed, as well as requests for additional information, and phone, written, and/or email contacts made by a health care provider (to the academic detailer, Boston team, PACE). A physician survey is also distributed, and physicians are asked to respond to the following statements (on a 5-point scale of Strongly Agree to Strongly Disagree):

The program provides me with useful information about commonly used medications.
The content represents unbiased and balanced information about drugs.
The program provides a perspective on prescribing that is different from what I get from other sources.
I find the patient materials useful in my practice.
It makes sense for the Commonwealth of Pennsylvania to devote resources to this activity.
My Drug Information Consultant is a well-informed source of evidence-based information about drugs I prescribe.
Being able to get Continuing Medical Education credits from Harvard is a valuable component of the service.
I would like to see this program continue.

Please provide a copy of the Academic detailer encounter forms.

They will provide an electronic copy of the academic detailer encounter form.

PACE –PACE program level evaluations are designed to measure the effect of the iDiS academic detailing program on drug utilization. This is done by performing a time-trend analysis using concurrent and historical controls. By performing this type of analysis, we attempt to answer questions such as:

Do prescribing behaviors change following educational encounters?
Can these changes be attributed to the educational encounters and not to other factors?

Seasonal variations
Policy changes (e.g., intro of Medicare drug benefit)
Drug changes (e.g., withdrawal of Vioxx)
Physician/practice-related characteristics

Do you generate a report based on these time-trend analyses. If so, please provide an electronic copy.

This is still in process will not have something ready for evaluation until late summer or early fall.

This is partly result of the new people who are just getting started with some topics. It is possible they may do a preliminary analysis on those that have completed the 6-month post intervention period.

2. Who conducts the Impact Evaluation (e.g. conducted by persons within the organization or by external evaluators such as government/insurance agencies)?

Three levels of evaluation:

Detailer: conducted by the data and program managers (Boston team)
Physician: Physician surveys are analyzed by the Boston team (data and program managers and program director)
PACE program: Prescribing data is obtained from PACE, analyses are conducted by the Boston team (including project managers, project director, and physician-researchers), and results of the analyses are evaluated by the Boston team and PACE officials

Are audit and feedback at the level of the individual physician used in you program?

It has been considered. PACE already does some of this in their reviews but there is no direct face to face connection with physicians. Detailers were the first person physicians had seen from PACE.

Provides an opportunity for physicians to provide feedback – and letters from PACE are not appreciated by some physicians. Given this response – a decision was made not to include audit and feedback early in the academic detailing program. Is an important tool and may be used when the program is established. They also have the ability to provide a list of patients that a particular doctor has on a particular drug. The detailers did not feel this was a good idea. They were concerned about privacy issues and did not want to feel like the drug cops. They want to provide a service to help with prescribing decisions in order to establish relationships with physicians.