Rx for Change Methods for Development: Interventions Directed to Consumers
Searching
Information Specialists have developed search strategies for Medline, EMBASE, The Cochrane Database of Systematic Reviews, DARE (Database of Abstracts of Reviews of Effects) and CINAHL (The Cumulative Index to Nursing and Allied Health Literature). A methodological filter to restrict retrieval to systematic reviews is used as needed (e.g. in databases indexing more than systematic reviews). Search strategies have not remained static over the course of the Rx for Change project. Instead, search terms have changed to reflect emerging terminology and growing understanding of interventions to: modify prescribing habits; influence behaviour of health professionals; change structures and processes within health care organisations and systems. Revised search strategies reflect not only improved conceptual understanding of the subject matter, but also of a more pragmatic concern--precision of the search strategy. Precision is the percentage of relevant citations identified within a group of search results. Updated search strategies for this project have increased precision. Search Strategies Rx for Change - Professional & Organisational Interventions
Screening
Our primary objective was to identify potentially eligible systematic reviews of interventions that evaluate professional, organizational, financial, regulatory, or structural interventions to improve health care delivery or health care systems. Two levels of screening were developed to help identify potential reviews. Each level of screening was conducted by two independent reviewers, and all disagreements were resolved by consensus or consultation with a third reviewer, when necessary.*
Phase 1 — Title and Abstract Screening
This level of screening process was done to ensure that interventions, populations, and outcomes listed in the title or abstract were appropriate for inclusion.
Articles for further review were included if all of the following criteria were met:
Non-systematic reviews (primary studies, editorials, commentaries, etc.) were excluded.
Phase 2 - Full-Text Screening
This level of screening was conducted to ensure articles were systematic reviews and to confirm the appropriateness of the interventions.
Full-text articles were considered to be potentially eligible if:
Reviews were excluded if:
Note: For reviews published in languages other than English, those that were potentially eligible were reviewed, based on the details of the title and/or abstract (published in English). If it could be determined from scanning the full-text article with the help of translation tools that there was a methods section, explicit inclusion criteria, and an appropriate intervention, translating the full-text article for further analysis was considered.
Quality Assessment
For each potentially eligible review, the methodological quality was assessed using AMSTAR (A MeaSurement Tool to Assess Reviews)
The AMSTAR tool consists of an 11-item checklist where reviews score one point for each criterion met, for a maximum score of 11. Reviews that achieve high scores indicate a higher methodological quality than those that achieve low scores.
Two reviewers independently assessed each AMSTAR checklist item and total for every review. The results for each review were compared and disagreements were resolved by consensus*. Where disagreement or uncertainty prevailed, a senior member of the team was consulted and a final decision was made.*
Reviews rated to be of moderate to high methodological quality (AMSTAR scores of 4 or more) qualified for data abstraction and analysis, and have been summarized and included in the Rx for Change database.
Reviews rated to be of low quality (AMSTAR scores of 3 or less) did not qualify for data abstraction. However, the bibliographic details of these reviews were collected and were made available through the Excluded Reviews page of the Rx for Change database.
Data Abstraction
Two reviewers independently abstracted all moderate- to high-quality reviews identified (AMSTAR scores 4 or more). The data abstraction was conducted in two phases. First, data were extracted from the review characteristics for the individual review summaries and, second, data were extracted from the included studies within each review to help with the analysis.
Phase 1 — Individual Review Characteristics
Data on the characteristics of each individual review were abstracted for each of the summary headings used in the database. A description of the type of data captured for each field is presented in the APPENDIX.
Phase 2 — Study Characteristics and Results
Data were abstracted about each included study within the review to provide data for the overall analysis of the review.
Studies that reported only patient outcomes or professional attitudes and knowledge were excluded. Studies that evaluated organizational, structural, regulatory, or financial interventions that did not report a prescribing (appropriate use) outcome were also excluded from the analysis.
* For the 2013 April update, a single reviewer screened the literature, two reviewers independently applied the selection criteria to identified reviews, and one reviewer undertook extraction activities, with some assistance from a second reviewer. Quality assessment was also performed by a single reviewer, with assistance from a second reviewer. Reviews scoring AMSTAR 3+ were assessed by a second reviewer and differences in scoring between reviewers were resolved by discussion or third party adjudication if necessary. For the 2014 April update, a single reviewer screened the literature at the title and abstract level only (Phase 1). Citations of review abstracts that met the inclusion criteria are listed on the database as pending further assessment and have been assigned to the intervention categories they address.
Dual data synthesis and consensus of the results of individual reviews were conducted and intervention summaries were prepared.
Table of Results
A data synthesis of the evidence from individual reviews were compiled and presented in a Table of Results. For each review, there were typically multiple interventions of interest compared against other interventions, including usual care; these comparisons were listed individually in the table.
For each comparison, all outcomes were tabulated under “appropriate care” unless there were only prescribing-related outcomes, in which case they were tabulated according to the “appropriate use” outcome categories. “Change in behaviour” was also used in regulatory or financial intervention comparisons.
Reviews varied greatly in the type and amount of data provided. For consistent presentation of results, vote counting was most often used for data synthesis. Analyses by vote counting were reported as the number of studies favouring the intervention out of the total number of studies for that comparison. When reviews provided additional data, such as a meta-analysis, this was included as well.
The following decision rules were used to create standardized statements of the results regarding the effectiveness of each intervention:
Vote Counting (percentage of studies favouring the intervention), where:
Absolute (or adjusted) risk difference (RD); individual, median, or pooled value, where:
The Results and Conclusions sections for each review were created, based on the data presented in the Table of Results, along with other considerations.
The methodological limitations, results of the comparisons, and any other items of note are described for the overall results. This can include:
If results were presented in the overall results, the methodological limitations were not restated for the prescribing results. The results of the impact of interventions on appropriate use prescribing-related outcomes only (dosage, cost containment, concordance, choice, etc.) — were described. For each intervention, the overall effectiveness and the number of studies for each outcome were reported. For interventions with only one comparison, it was stated that there was insufficient evidence. If there were no prescribing-related outcomes in the review, a statement to that effect was made.
The conclusions restate the methodological quality of the review. The following criteria were used to make statements about the overall effectiveness of the interventions on appropriate care and prescribing-related outcomes:
Intervention summaries are based on our analysis of high-quality and highly relevant (or key) reviews. For each intervention category, overall summary statements of the evidence are presented.
Effectiveness statements describe the number of reviews that were found to be effective, ineffective, or have mixed results for appropriate care outcomes; and, where possible, these reviews are identified. Also reported are reviews with prescribing outcomes and those with insufficient evidence. If no high-quality or key reviews were located for the intervention, a statement to that effect was reported. If there were an insufficient number of studies (n ≤ 4) within the high-quality or key reviews from which to draw conclusions about the effectiveness of the intervention, a statement to that effect was made.
In the Summary of Overall Findings from Reviews and the Summary of Findings Related to Prescribing sections, the effectiveness of the intervention is stated.
Systematic methods were employed to construct the database to maximize transparency and minimize bias. Summaries of the individual reviews have been structured to present the overall characteristics of the reviews and the findings as they relate to change in professional behaviour.
For each individual review summary within the database, the number of included studies, as noted in the review, is indicated in the summary of the review. Also indicated is the number of studies that evaluated professional interventions or organizational, structural, financial, or regulatory interventions with prescribing as a main outcome focus (and, therefore, relevant to the database). A link to the list of individual primary studies included in the analyses of each systematic review is provided in each review summary. At times, the number of studies listed and the number of links to studies in the list are not equal. Possible reasons for this discrepancy include:
Database users are encouraged to consult the full text of reviews, in addition to the summaries listed on the database, for information that may have been beyond the scope of this project.
The following are abbreviations used in the database:
BA before-and-after (study)
CBA controlled before-and-after (study)
CI confidence interval
CT or CCT controlled trial
ITS interrupted time series (analysis)
RCT randomized controlled trial
RD risk difference
RM repeated measures
SS statistical significance
The following are explanations and descriptions of the fields used in the database to present summary data and results for each review.
Review Question
This section describes what question the review addresses, based on the information in the abstract, introduction, and/or methods of the review. The question may include the populations, interventions, and outcomes investigated.
Background
This section describes the intervention(s) evaluated in the review. It includes a general description; a more detailed description is provided when it is made available by the authors of the review. Background information about the condition, state, or problem is not provided. For reviews that use the EPOC taxonomy, this can be the statement for this background section. If a review does not provide this information in the introduction or methods sections, a statement to the effect of “not described by the author” is made.
Search for Evidence
This section describes the most recent date for searching that is provided. If there are multiple databases with different search dates used, the most recent date that a large electronic database such as MEDLINE or EMBASE was searched is noted.
Included Studies
This section describes the number of studies that met the inclusion criteria of the review and provided data about the effects of the interventions. Not included are references or studies cited elsewhere in the review that were not part of the analysis.
Studies Related to Prescribing
This section describes the number of included studies that evaluated interventions for evidence-based prescribing and drug use. This can be a subset of the included studies or it can comprise all of the included studies. If the number of studies included was unclear, this field was left blank.
Study Design
This section describes the types of study designs included in the review. The following categories were used:
Interventions
The interventions evaluated in the included studies are listed, with additional details, as appropriate. The interventions of interest were categorized according to the EPOC taxonomy, and multi-faceted interventions were indicated separately.
Target Population
This is the population to which interventions were directed in the included studies. Populations can be grouped into categories such as health care professionals, patients, or caregivers. Health care professionals have been grouped into broad categories:
Other target populations may be listed (e.g., midwives). For the professionals section, the types of patients indirectly targeted were listed under “Patients Reviewed.”
Setting
This section describes the settings in which the interventions took place in the included studies. Settings were broadly categorized by location, ownership, and institution or organization type (if applicable):
Location Primary care practices |
Ownership Ownership — private |
Institution Academic — teaching |
This section describes the patients who are indirectly affected by interventions that directly target health care professionals. Patient populations among the included studies were grouped into broad categories according to the information provided by the authors and may not include comprehensive reporting for age, residence, or diagnosis for each study.
Age Children (0- to 16-years-old) |
Other Specific diagnosis or clinical condition |
Prescribing-related outcomes reported in the review were grouped into the following categories:
Outcomes that are not related to prescribing or drug use, but are of relevance to professional behaviour in the process of care include the following:
All behaviours were considered as part of the review.
This section describes the methods used by the authors of the review to analyze the data. More than one type of analysis may be used. Categories of analyses include:
Both internal and external sources of funding are listed, if applicable. If nothing is reported by the authors regarding conflict of interest with funding, “not stated” is used. If the authors report no conflict of interest, “no perceived conflict of interest” is stated.
Comparison
For each review, there are typically multiple interventions of interest that are compared against other interventions, including usual care; these comparisons are listed individually in the table. Different multi-faceted interventions may appear together as one comparison, depending on the authors’ original analyses.
Outcome
For each comparison, all outcomes are tabulated under “appropriate care” unless there were only prescribing-related outcomes, in which case they would be tabulated according to the “appropriate use” outcome categories listed previously. “Change in behaviour” is also used in regulatory or financial intervention comparisons.
Number of Studies
This section describes the number of studies providing data for a given comparison and outcome.
Note: The number of studies in this column may not add up to the total number of included studies or studies related to prescribing for various reasons, such as the following:
Analysis
This section describes the analysis method(s) used by the reviewers. Cases exist where these analyses differ from the authors’ analyses; comparisons between this field and the “Authors Analysis of Data” field in the summary can be made to identify such situations. The following are standardized statements used to describe the analysis method used by the reviewers:
A variety of methods may have been used to report the data provided by the authors and include but are not limited to:
Where possible, absolute effect measures were used rather than relative measures, as absolute measures provide a relatively unbiased measure of effect. Where pooled measures were not available or appropriate, a range or median, plus a range of measures, was provided. Analyses by vote counting were always reported as the number of studies favouring the intervention out of the total number of studies.
The following decision rules were used by the reviewers to create standardized statements of the results regarding the effectiveness of each intervention:
Vote counting (percentage of studies favouring the intervention) where:
Absolute (or adjusted) risk difference (RD); individual, median, or pooled value, where: