Newsletters - Evaluation

Introduction

The newsletter is a common instrument used in the provision of drug or prescribing information. Academic detailing organizations often produce and utilize this type of printed educational material to complement the detailing visit. Generally, the newsletter provides a detailed discussion of a specific drug or disease topic. The newsletter differs from a prescribing aid, which is designed to be a tool used to assist the practitioner in managing a given patient. The newsletter is designed to provide a message as well as supporting rationale or background information to support the message.

Methods

Newsletters were collected from a variety of Canadian and International academic detailing organizations. Summary descriptive information was abstracted from each of the representative newsletters evaluated. Word count was estimated by extracting text from the existing file formats and using the “Word Count” tool within Microsoft Word®.

Since there isn’t an existing tool available to evaluate academic detailing newsletters, a pragmatic evaluation tool was developed to perform this task. The design-based elements in the evaluation were derived from the following reference, “Design Criteria and Guidelines for the Visual Presentation of Information”, written by Jorge Frascara and Stan Ruecker (Appendix 23). This guideline document was produced for the Alberta Drug Utilization Program (a member of the CADC) as part of a Best Practices Contribution Program project. The CADC recognized that, as researchers and clinicians, we were adequately skilled to develop content that could assist prescribers, but could do more to enhance the design of our materials to maximize their readability and effectiveness. Hence, Frascara and Ruecker were contracted to research the published design literature and develop practice guidelines on how to visually present our materials. However, this guideline could not be used exclusively to evaluate the newsletters (or prescribing aids and patient materials) since it is entirely designed based with no consideration of content.

A 20-element questionnaire was created for the evaluative component of the newsletter assessment. Each element was assessed on 5-point Likert anchored only at the ends. Two researchers independently assessed each of the newsletters. After initial evaluation, any individual element score that differed by >2 points was discussed with revisions to score made as required. The final score for each newsletter was an average of the total score awarded by each researcher. The final score was a number between 20 and 100 and was considered the overall quality score of the document.

The final version of the evaluation questionnaire for academic detailing newsletters is available in Appendix 24. Each element within the questionnaire was created based on evaluating an element of the newsletter’s design or evaluating the material content of the newsletter. Many aspects of the evaluation were developed based on discussions with the other Canadian academic detailing organizations. The rationale for each element is discussed below:

Element #1

Referencing – Referencing provides the reader with the ability to assess the source of the material. Some established academic detailing groups have spoken about a transition from non-referenced to referenced documents. It may also be important for new detailing groups, without an established reputation, to use referencing. The current focus on evidence-based medicine makes referencing essential.

Element #2

Peer Review – Some type of peer review enhances credibility and is obviously a required standard for medical literature publication. Detailing groups often use peer review to both enhance quality but also as a mechanism to ensure local buy-in of the physician community.

Element#3

Systematic Review – Systematic review is a required standard of evidence based medicine. Explicit systematic review is not currently common in academic detailing materials. A systematic approach, however, is the best defence against criticism of selection bias in the literature that is selected and presented.

Element #4

Statistics – Pragmatic presentation of statistical information on levels of benefit/risk and uncertainty have become an important part of the translation of the medical literature for academic detailers. These tools have been used with increasing frequency as the level of physicians’ acceptance and understanding has increased in established programs.

Element #5

Economics – The reality of escalating medications costs has made both the clinical and cost effectiveness part of the decision making process. In many practice environments, physicians do not have ready access to information on medication cost. Academic detailing materials can be an important source for cost information provided with the appropriate clinical context.

Element #6

Graphics – Graphics can be an important way to translate information. Practitioners with limited available time may only scan documents and graphics can be an important way to convey key messages. Documents devoid of graphics tend to be heavily text based and may discourage time-challenged readers.

Element #7

Key Points – Key or summary points are an important way to convey and reiterate the most important take home messages. Readers may only remember one or two points and key messages may be an effective way to enhance the retention of the newsletter’s targeted messages.

Element #8

Sections – A review of the design of academic detailing materials recommended that dividing the materials into five sections was optimal for enhancing readability and more specifically for enhancing recall (Appendix 23). There is some evidence to support the arbitrary cut off of five sections, and it seems reasonable that the further from the ideal the more problems there may be with readability and recall.

Element #9

Levels of Hierarchy – A review of the design of academic detailing materials recommended that there be a maximum of three levels of hierarchy that are consistently delineated by titles, bullets and/or font size and style (Appendix 23). Readers can easily distinguish two levels of importance, but it is possible to distinguish three levels with careful attention to consistency and style.

Element #10

Tables: Levels of Hierarchy – A review of the design of academic detailing materials suggest the approach to hierarchy in tables should be consistent. For example, the first table within a document, providing information on drugs used to treat a condition, lists the drugs based on effectiveness (with most effective at the top and least at the bottom). All subsequent drug information tables within the document should use the same ordering by effectiveness as opposed to ordering products based on price or alphabetically. This enhances readability and avoids potential confusion as readers move from one table to the next.

Element #11

Editorial Consistency for Tables – A review of the design of academic detailing materials recommended that tables should have a similar format, colour pattern and structure to enhance readability (Appendix 23).

Element #12

Column Width – A review of the design of academic detailing materials recommended that, to enhance readability, columns should have 7 to 12 words per line (Appendix 23).

Element #13

Readability – A review of the design of academic detailing materials suggested than chunks of text are easier to read than long sections of continuous prose (Appendix 23).

Element #14

Justification – A review of the design of academic detailing materials recommended the use of un-justified (left flush) text provides a variety in line length that helps readers locate the next line when reading. Hyphenation should also be limited and should occur on no more than two consecutive lines (Appendix 23).

Element #15

Colour – The use of colour can enhance the attractiveness and readability of a document but inappropriate use of colour can be distracting or decrease readability. The use and choice of colour can be further complicated by the knowledge that 6% of adult males have some form of colour blindness. A review of the design of academic detailing materials suggests that for long text sections black type on a white background is best (Appendix 23). Avoid the use of dark type on dark backgrounds.

Element #16

Main Message Time – Given that available time may be the limiting factors in detailing physicians, it is important that the main messages of the materials can be obtained quickly. This element subjectively assesses the reviewer’s estimate of the time required to extract the main messages. Objective assessment of the speed, accuracy and application of these messages would also be useful but is beyond the scope of the current assessment.

Element #17

Therapeutic Decision Making – Given that the aim of the majority of academic detailing projects is to affect some type of change of medical behaviour, it is critical that the academic detailing document provide the information required to support this behaviour change. This information should include at a minimum the therapeutic alternatives and dosing considerations.

Element #18

Behaviour Target – If goal of the newsletter is to change a behaviour, then the target behaviour should be clear in the academic detailing documents. An amorphous mass of information may enhance knowledge, but if it does not target barriers to the desired behaviour, it may not affect change. This element is a subjective assessment of the clarity of the behavioural target in the document.

Element #19

Storage and Retrieval – After the academic detailing event, it would be helpful if documents were available for future reference when appropriate clinical situations arise. The ability to store documents in paper or electronic form (website) would be considered an asset. The title, date, issue number and website address are important features supporting future retrieval.

Element #20

Consistent Style – Readability is enhanced by ensuring a consistent style throughout the document. All headings, sub-headings, bullets and lists should be managed in a similar and consistent manner. Deviations in format or style of approach may be distracting or confusing to the reader.

A graphical representation of the elements used to evaluate the Newsletters is included in Appendix 25. This chart also displays how the evaluative elements of the Prescribing Aids and Patient Information Materials relate to the Newsletter evaluation.

Results

The newsletters had an average length of 3.6 pages and ranged from 2 to 6 pages in length. They contained an average of 2028 words and ranged from 881 to 4948 words (Figure 1).

Theoretically, it would be possible to meet more of the evaluation elements by producing a longer comprehensive document. This was explored by examining the quality scores and the length of the newsletters. There was no obvious relationship between the length of the newsletter, as measured by word count, and its quality (Figure 2). If we treat the quality score as a continuous variable we can also see this statistically by examining the correlation between newsletter length and quality. Only a weak association (Pearson’s coefficient = 0.268) with a non-significant (p= 0.145) correlation was found between length and quality (Figure 2). With a possible range from 20 to 100, the quality scores fell between 52 and 83 with an average score of 72. The detailed results for each of the individual newsletters evaluated can be found on the cover sheet associated with each newsletter. The score for each of the 20 elements and the overall score are displayed along with a brief discussion of the overall strengths and weaknesses of the newsletter. The catalogue of individual newsletters and their associated evaluation is included in Appendix 26.

Limitations

There are significant limitations to this evaluation. The scale used to evaluate the newsletters is based on a number of indicators of quality but no attempt was made to validate this scale. The even weightings on questions and the number of questions relating to specific areas may not provide appropriate emphasis in relation to newsletter impact and outcomes. The ideal tool would validate newsletter characteristics against outcomes or some surrogate of behavioural change outcomes. This complex task is beyond the scope of the current project. However, the evaluation scale does provide a uniform yardstick to compare newsletters against quality indicators.

Within the spectrum of academic detailing materials, it can be difficult to define which materials should qualify as newsletters. The score of some materials may reflect their deviation from the newsletter format as much as their inherent quality. Overall scores for individual newsletter must be interpreted with this limitation in mind.

It was not possible to fully evaluate some elements with the information available. Documents may have been peer reviewed but if this is not indicated on the newsletter this makes it impossible to assess. Some newsletter may have scored higher if certain production processes were more readily apparent when reviewing the document. Also, materials may be developed with detailing visit in mind, with information held back to be facilitate opportunity for discussion and “value-added” input during the visit.

Discussion

Importance

Newsletters are an important component in the arsenal of materials used by most academic detailing organizations. For many organizations, they are one of the key anchors for the detailing visit and the central core for other materials provided (ex. iDiS, PrISM). While not all organisations use newsletters in this central manner, most could likely find a role for a newsletter within their approach. Newsletters are also important because they provide abbreviated, pragmatic coverage of the complete topic. Two of the Canadian organizations reference larger supporting documents to provide more detailed reference materials. The stand-alone nature of the newsletter has allowed some organizations to broadcast their newsletter beyond the core group of physicians that they see in academic detailing visits. While there is evidence of the limitations for this passive diffusion of information, it may have some impact and is a relatively inexpensive way to reach a larger audience. In this sense, the newsletter may be a component of a toolkit that could be used in areas of the country where no academic detailing programs exist. Its self-contained nature and smaller size would lend itself to this type of passive diffusion. The success of this approach may be limited and may correlate with the complexity of the behaviour change required.

Content

The newsletters reviewed varied from two to six pages. Given the lack of correlation between length and quality, it seems reasonable to suggest that newsletters should be limited to 2 to 4 pages. The word count should also reflect this less-is-more approach. Newsletters that were heavily narrative based were penalised for elements of design, readability and messaging. Obviously, the approach needs to vary to some degree based on the clinical complexity of the topic. Front-end attention to quality indicators used in our evaluation should help to increase the readability and impact potential of newsletters. A variety of formats are included in our evaluated representative newsletter. Most organisations have attempted to develop a relatively uniform style and approach with their newsletter materials. Over time, this consistency likely helps to enhance recognition by physicians and “brand” the product as a useful source of non-commercial drug information.

Format

The best newsletters deliver detailed information that provides practitioners with the information necessary to change behaviour. The need for detail must be balanced with the need to design the information format in a manner that quickly delivers key messages without relying on extended narrative prose. The use of graphics or tables and breaking the text up into readable chunks can all help in this regard. Clearly labelled key messages can also be a practical way to deliver the overt messages of the newsletter. Careful use of colour can increase visual appeal, enhance readability and focus readers’ attentions on main messages.

Process

Most of the Canadian academic detailing organizations produce some form of newsletter on a regular basis as part of their product offering to prescribers. The various organizations employ different distribution strategies. For example, Alberta’s ADUP sends all of its prescribing aids (associated with newsletters) out to all general practitioners in the province. While British Columbia’s CDUP provides their materials primarily to physicians serviced by the hospital region they operate in and secondarily by mail/email to individuals that have expressed interest in their program. Any newsletter produced by COMPUS would likely be incorporated into existing distribution strategies. Well-established organisations with established “branded” materials may have more difficulty with centrally (COMPUS) produced newsletters but would likely use them as resource documents. It might be possible to include a blank section on a COMPUS-produced newsletter to allow the organisation brand to be included under “Distributed By”. This allows continuity with the existing organization brand. Less established programs, new programs, and other organisations (health regions) should be able to make ready use of the newsletters as self-contained information tools.

Based on interviews with the international and domestic academic detailing groups, it is estimated take anywhere from a few weeks to several months to produce a newsletter. The development of newsletters is often integrated into topic research and the development of other materials making it difficult to isolate a specific time for production. Receiving a centrally produced newsletter from COMPUS could be a considerable advantage. Given the planning horizons of academic detailing organizations, it will be important to establish delivery dates and provide information on early drafts of the materials. This will allow organizations to adjust their schedules to take advantage of the materials produced by COMPUS.