Use of Real-World Evidence in Single-Drug Assessments Environmental Scan


Project Status:
Project Line:
Health Technology Review
Project Sub Line:
Environmental Scan
Project Number:
Final Biosimilar Summary Dossier Issued:


Traditionally, regulatory and health technology assessment (HTA) organizations rely on evidence derived from randomized controlled trials (RCTs) for the assessment of new drugs, given that RCTs are perceived as the gold standard to demonstrate efficacy and safety. However, patients in RCTs are highly selected and may not reflect the target population in whom the drug may be used. Actual recipients in routine clinical practice present with various comorbidities, co-medications, genetic profiles, behaviours, and perspectives. Long-term effects of drugs are also difficult to assess in RCTs designed to show efficacy in a narrow time window. Drug evaluators, such as regulators and HTA agencies, are therefore faced with making decisions based on incomplete or uncertain information on some aspects of effectiveness.1

In view of these limitations, drug evaluators are considering the use of real-world evidence (RWE) — clinical evidence emanating from real-world data (RWD) — to supplement and enrich the evidence in support of drug regulation and reimbursement. RWD may be defined as “data regarding the effects of health interventions (e.g., safety, effectiveness, resource use, etc.) that are not collected in the context of highly controlled RCTs” and may include “primary research data collected in a manner which reflects how interventions would be used in routine clinical practice or secondary research data derived from routinely collected data” (Appendix 1).2 Although no consensus exists on what RWD is, most organizations include data generated from observational studies (e.g., cohort, case-control, or case series) and from sources such as disease registries, administrative data, health surveys, electronic health records, or medical chart reviews.1-3 For some agencies, RWD may also be generated from pragmatic studies (also known as large simple trials or practical clinical trials), where patients may be randomized to treatments but subsequent care and follow-up more closely resembles standard clinical practice than in a conventional RCT.1,3 RWD may also be obtained from home medical devices or wearable technologies.4

RWE is routinely used to inform some aspects of drug development, such as the natural history and epidemiology of a disease, to provide data on treatment pathways and comparator interventions in clinical practice, safety surveillance, and to determine resources used and costs of health care.1,4-6 In cost-effectiveness analysis, RWE is generally accepted and frequently used.5-7 The growth of accessible data from electronic health records, administrative claims databases and registries, combined with advanced statistical methods, may facilitate greater use of observational data to draw causal inferences on the effectiveness of treatments.4

In keeping with these trends, drug review programs increasingly have to contend with non-RCT data submitted to demonstrate effectiveness and safety of single drugs. This Environmental Scan was developed to better understand how international regulatory and HTA organizations address this challenge. The information presented in this report may be of value to all organizations seeking to implement processes that take into consideration the role of RWE in single-drug appraisal.


This Environmental Scan will identify, describe, and compare how regulatory frameworks and HTA processes in Canadian and international organizations incorporate RWE in single-technology assessment of drugs.

More specifically, this Environmental Scan will aim to meet the following objectives:

  • Describe the eligibility criteria for inclusion of RWE for the purpose of establishing drug effectiveness and safety in single-drug technology assessments performed by international HTA and regulatory organizations.
  • Describe how international HTA and regulatory organizations use RWE of effectiveness and safety that is included as part their single-drug technology assessments.
  • Describe the impact of RWE on single-drug technology assessments performed in various organizations.

This Environmental Scan will focus on the initial assessment of drugs for reimbursement or regulatory approval as part of relative effectiveness assessments and will not address the use of RWE in managed access processes.


The findings of this Environmental Scan are based on responses to the Use of RWE in Single-Drug Assessments survey (Appendix 2) and a limited literature search.

A limited literature search was conducted on key resources including Ovid MEDLINE, PubMed, HTA agencies, domestic and international ministries of health websites, and a focused Internet search. No methodological filters were applied to limit retrieval by publication type, but conference abstracts were excluded from the search results. The search time frame was limited to English and French language documents published between 2012 and 2017 (five-year time frame). Regular alerts were executed until project completion. Reference lists of relevant articles were reviewed. Websites for regulatory and HTA organizations were searched for relevant guidelines or policy papers.

The organizations listed in Table 1 were selected due to commonalities with the Canadian context, including geography and regulatory, HTA or reimbursement processes. Due to feasibility issues such as time constraints, other organizations with some relevance to the Canadian context were excluded.

Table 1: National and International Regulatory and HTA organizations

Country Regulatory Agencies HTA Organizations
Canada Health Canadaa CADTH (CDR, pCODR), INESSSa
US FDA US Department of Veterans Affairs
Europe EMA EUnetHTA
France   HAS
Germany   IQWiG
Netherlands   ZIN
Sweden   TLV
Finland   PPB
Norway   NoMA
Australia TGA PBACa
New Zealand Medsafe PHARMACa

CDR = CADTH Common Drug Review; EMA = European Medicines Agency; EUnetHTA = European Network for Health Technology Assessment; FDA = Food and Drug Administration; HAS = Haute Autorité de Santé; HTA = Health Technology Assessment; INESSS = Institut national d’excellence en santé et en services sociaux; IQWiG = Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; NICE = National Institute for Health and Care Excellence; NoMA = Norwegian Medicines Agency; PBAC = Pharmaceutical Benefits Advisory Committee; pCODR = CADTH pan-Canadian Oncology Drug Review; PHARMAC = Pharmaceutical Management Agency; PPB = Pharmaceuticals Pricing Board; SMC = Scottish Medicines Consortium; TGA = Therapeutic Goods Administration; TLV = Dental and Pharmaceutical Benefits Agency; ZIN = Zorginstituut Nederland. 
a Organizations surveyed.

Following a preliminary review of the available literature, gaps in knowledge were identified for a subset of agencies hosting drug review programs. To fill those information gaps, a survey was distributed electronically to the identified agencies. Survey respondents were asked to consent to the reporting of the information they provide by electronically signing a form attached to the questionnaire. The survey included dichotomous (e.g., Yes/No), nominal (e.g., list of options), and open-ended questions. A summary of the results of the survey were merged with related information from the literature review. Surveys received up to January 30, 2018 were included. See Appendix 2 for the complete survey questionnaire.


From the literature search, 45 articles were identified that provided information relevant to this Environmental Scan. Surveys were distributed to four agencies (Appendix 3) and responses were received from all groups.

The literature search identified sufficient information to forego the need to survey European HTA agencies. Information from regulatory agencies was deemed to be secondary, thus except for Health Canada, the regulatory agencies were not surveyed and only data from the literature review were used to inform the Environmental Scan. No relevant or English or French language information was identified for the regulatory agencies in New Zealand (Medsafe) and Australia (Therapeutic Goods Administration), and the US Department of Veterans Affairs.

Eligibility Criteria of Inclusion of RWE

Regulatory Agencies

In the US, the statutory requirement for marketing approval of new drugs for both common and rare disorders is “substantial evidence” of the drug’s claimed effect.8 Substantial evidence has been defined as data from adequate and well-controlled studies that are able to “distinguish the effect of a drug from other influences, such as spontaneous change in the course of a disease, placebo effects, or biased observation.”8 The guidance also lists specific study design aspects including a valid comparison with a control, which may be concurrent, or in limited circumstances, historical.8 A requirement for at least two adequate and well-controlled trials has been accepted as the evidentiary standard to determine effectiveness, although flexibility has been used in applying these standards, and the FDA has outlined situations where effectiveness of a new indication may be extrapolated from existing efficacy studies, where a single adequate and well-controlled clinical investigation and confirmatory evidence may be accepted, and situations where a single multi-centre study without supporting data may be sufficient.9 What may be accepted as “substantial evidence” takes into consideration the clinical context, including the severity of the disease (and thus patients’ willingness to accept risk) and the availability of alternative treatments.

As part of the 21st Century Cures Act (2016), the FDA had been directed to develop a regulatory framework to evaluate how RWE can potentially be used to support approval of new indications for approved drugs or to support or satisfy post-approval study requirements.10 The Act defined RWE as “data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than randomized clinical trials.”10 The framework shall include information on the sources of RWE, gaps in data collection activities, and the standards and methodologies for collection and analysis of RWE.10 In 2017, the FDA issued guidance on the use of RWE to support regulatory decision-making on medical devices.11

Health Canada accepts all relevant data in support of a drug’s efficacy and safety, including RWD, with no limits by study design or data source (see summary of survey, Appendix 4). All regulatory agencies accept RWD to supplement clinical trial data on the safety of pharmaceuticals (both pre- and post-approval), and real-world studies may be conducted in order to meet post-authorization data requirements requested by regulators.6,12,13 The FDA and the European Medicines Agency (EMA) have developed accelerated or conditional approval mechanisms, whereby drugs may be approved based on phase II studies or surrogate outcomes, with subsequent evidence to be developed that confirms efficacy and safety. In addition, the EMA is exploring adaptive pathway processes, which use an iterative approach to drug development allowing for early and progressive patient access to a medicine combined with RWD generation, in specific patient populations with high unmet medical need.14

HTA Agencies

Two articles reviewed the evidence requirements of European HTA agencies for assessments of new pharmaceuticals.7,15

The European Network for Health Technology Assessment (EUnetHTA) reviewed the evidence requirements for HTA of new pharmaceuticals from individual European national agencies responsible for reimbursement.15 In total, 29 countries provided data which included their manufacturer submission template or submission guidelines in use up to June 2013. The study types accepted to determine clinical effectiveness were specified by 23 countries and included all clinical research (1 country), RCTs and/or clinical trials (21 countries), or comparative studies (3 countries). In addition, eight countries specified observational studies and five specified meta-analyses or systematic reviews. Five countries accepted additional study types for safety data including non-comparative trials, post-marketing surveillance data, case reports, patient registers, observational studies, or pharmacoepidemiological studies.15

Makady et al.7 examined the policies of six HTA agencies on the use of RWE. This included a literature search and interviews with representatives from six HTA agencies: Swedish Dental and Pharmaceutical Benefits Agency (TLV), UK National Institute for Health and Care Excellence (NICE), German Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG), French Haute Autorité de Santé (HAS), Dutch Zorginstituut Nederland (ZIN), and the Italian Medicines Agency (AIFA). The agencies accepted all available evidence for initial drug assessments, including RWE. Although most agencies did not specify which RWD sources or methods should be used, three (NICE, IQWiG, ZIN) provided suggestions for specific RWD sources and guidance on the suitability of these sources to answer different questions.7

Using data from the two reports described above, survey responses and individual agencies guidance or policy documents, a summary of the study types accepted by the HTA agencies specified in the Environmental Scan protocol was compiled (Table 2 and Appendix 4). The agencies accepted both randomized and non-randomized clinical data as part of the initial drug submission. There were four agencies that requested RCTs to demonstrate the efficacy and safety of a new drug, but were willing to accept non-RCT data in certain circumstances (IQWiG, Scottish Medicines Consortium [SMC], Pharmaceutical Benefits Advisory Committee [PBAC], Institut national d'excellence en santé et en services sociaux [INESSS]). Most agencies requested additional non-RCT safety data, such as Periodic Safety Update Reports (PSURs) or other pharmacovigilance data. Information on evidence requirements for resubmissions was found for seven agencies (INESSS, CADTH, HAS, SMC, ZIN, PBAC, and Pharmaceutical Management Agency [PHARMAC]). Of these, CADTH listed specific criteria for new evidence, including data from one or more RCTs (preferred), and non-randomized studies, which may be particularly useful if there was uncertainty regarding the persistence of efficacy or if long-term safety or efficacy data were required, if RCTs were not possible due to a limited number of patients or for ethical reasons, if RCT data lacked relevant comparators, there was uncertainty regarding the dosage in clinical practice, or if RCTs had limited external validity.16

Table 2: Evidence Accepted by Key HTA Organizations

Country (Agency) Efficacy and Safety Safety (additional evidence)a Sources
UK (NICE) All clinical data: RCTs, observational studies Non-comparative trials, post-marketing surveillance data 17
Scotland (SMC) Active-controlled RCTs, meta-analyses; in absence of active-controlled RCTs, other RCTs or uncontrolled studies accepted Data from regulatory authorities 18
France (HAS) Meta-analysis, clinical trials, observational studies PSUR, pharmacovigilance and regulatory data 15,19
Germany (IQWiG) RCTs; observational studies (in exceptional circumstances) Observational studies, pharmacovigilance and regulatory data 20
Netherlands (ZIN) Clinical trials, observational studies, meta-analyses, systematic reviews Voluntary reports 21
Sweden (TLV) RCTs, systematic reviews, comparative studies, other evidence   7,15,22
Finland (PPB) RCTs (EPAR, published articles), other relevant studies, epidemiological studies, meta-analyses, reviews articles PSUR, EPAR 15,23
Norway (NoMA) RCTs, observational studies   15
Europe (EUnetHTA) Systematic reviews, RCTs, indirect treatment comparisons, randomized pragmatic designs, other study designs Epidemiological studies, registries or other RWD, pharmacovigilance data, data from manufacturer or regulatory agencies 24-26
Australia (PBAC) RCTs (NRS accepted if no direct or indirect evidence available from RCTs or other exceptional circumstances) PSUR, NRS, pharmacovigilance studies Survey 27
New Zealand (PHARMAC) All study types accepted Surveillance data Survey 28
Canada (CADTH) All study types accepted   29,30
Canada (INESSS) At least one RCT, unless in exceptional circumstances. Additional study types accepted as supporting data   Survey 31

EMA = European Medicines Agency; EPAR = European public assessment report; EUnetHTA = European Network for Health Technology Assessment; HAS = Haute Autorité de Santé; HTA = health technology assessment; INESSS = Institut national d’excellence en santé et en services sociaux; IQWiG = Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; NICE = National Institute for Health and Care Excellence; NRS = non-randomized study; PBAC = Pharmaceutical Benefits Advisory Committee; pCODR = CADTH pan-Canadian Oncology Drug Review; PHARMAC = Pharmaceutical Management Agency; PPB = Pharmaceuticals Pricing Board; PSUR = Periodic Safety Update Report; RCT = randomized controlled trial; SMC = Scottish Medicines Consortium; TGA = Therapeutic Goods Administration; TLV = Dental and Pharmaceutical Benefits Agency; ZIN = Zorginstituut Nederland. 
a In general, the same study types were eligible for the assessment of efficacy and safety. Those listed specifically for safety were in addition to other data.

Use of Real-World Evidence

Regulatory Agencies

Three articles were identified that examined the frequency with which regulatory bodies used non-RCT data in their deliberations.32-34 Hatswell et al.32 investigated the number of EMA or FDA approvals of drugs based on evidence other than RCTs. Data available from EMA and FDA websites were reviewed for all drugs approved between January 1999 and May 2014 (excluding generic drugs, biosimilars, vaccines, fixed-dose combinations of existing drugs, antimicrobial drugs, and blood products). The number of indications approved based solely on uncontrolled studies, without either the pivotal or supporting studies being RCTs, was reported. In this review, uncontrolled studies were described as single-arm observational studies, historically controlled studies or randomized dosing studies (patients randomized two or more regimens of the experimental drug). In total, EMA issued 795 approvals, of which 44 indications (5.5%) were based on uncontrolled trials.1 Eight approvals were extensions of indications and 36 were for products with no RCT data in an approved indication. Nine applications based on uncontrolled data were either rejected by the EMA or were withdrawn by the manufacturer.32 Between 1999 and 2014, the FDA approved 774 indications including 60 (7.8%) approved based on uncontrolled studies (48 new drugs; 12 extensions of indications).32 One application based on uncontrolled data was not approved by the FDA.32 The majority of indications approved based on uncontrolled studies were in oncology (66%), with 20% of indications approved for rare metabolic disorders.32

The National Organization for Rare Disorders conducted a review of the evidence used by the FDA to approve orphan drugs.33 Publicly available documents for non-cancer orphan drugs approved by the FDA between 1983 and 2010 were reviewed, and those approved based on evidence other than the conventional standard of “two adequate and well-controlled studies” were noted. Of the 135 drugs approved 90 (67%) were based on evidence that did not meet conventional standards, although the report did not quantify to what extent approvals were based on a single well-controlled trial versus other forms of evidence including RWE.33

Davis et al.34 reviewed the evidence available for cancer drugs approved by the EMA between 2009 and 2013. During that time, EMA approved 48 drugs for 68 cancer indications, of which eight indications (12%) were approved based on uncontrolled studies.34

The survey respondent from Health Canada stated that sufficiently large well-constructed RCTs are considered the least biased source of efficacy and safety data to inform risk-benefit assessments, and although RWD is accepted, the weight of this evidence in regulatory decisions varies depending on the situation. For example, RWE may provide significant added value when assessing drugs for populations not well studied in RCTs, where there is significant unmet need, for innovative medicines or priority reviews, and when RCTs are not feasible (ultra-rare conditions) or unethical situations (pregnancy). RWE may provide supportive data that has greater external validity, as well as providing information on subpopulations, off-label use, misuse, adherence, and to validate surrogate outcomes. The Therapeutics Products Directorate is currently in the early stages of exploring the enhanced use of RWE to support pre-market regulatory decisions (Appendix 4).

HTA Agencies

Makady et al.7 examined the policies of six European HTA agencies on the use of RWE for initial drug submissions, using data from a literature review and interviews with representatives from NICE, TLV, IQWiG, HAS, ZIN, and AIFA. With regard to the evaluation of effectiveness, the authors found that all agencies used evidence hierarchies that placed RWD on a lower level of quality and reliability than RCTs.7 Thus the agencies affirmed that RWE may be used to confirm or supplement, not replace, the findings from RCTs on the treatment effects of drugs.7 Under specific circumstances only would RWD be used to demonstrate treatment effects. The examples provided included the following: in the absence of RCT data (NICE, ZIN, IQWiG); in the absence of head-to-head RCTs, RWD may be used to inform indirect treatment comparisons (NICE, ZIN); or to supplement RCT data if data on specific subpopulations or long-term follow-up were lacking (NICE, ZIN). Makady et al.7 reported that in all cases, the agencies required an explicit justification why RWD were used and clear discussion of the biases associated with the RWD and its consequences on treatment effect estimates.7 Any conclusions on treatment effects that were based on RWE would more circumspect than those based on evidence from RCTs.

A subsequent study5 by members from this research team evaluated the use of RWD for reimbursement decisions by five HTA organizations in Europe (NICE, SMC, HAS, IQWiG, and ZIN) for seven drugs indicated for melanoma (ipilimumab, vemurafenib, dabrafenib, cobimetinib, trametinib, nivolumab, pembrolizumab). In total, 52 HTA reports published between 2011 and 2016 were included in the review, and of these, 28 (54%) included RWD. RWD were used to estimate the incidence or prevalence of melanoma in all 28 reports, and to inform drug efficacy and safety in seven (13.5%) and six reports (11.5%), respectively. The study designs providing evidence for efficacy included six observational studies, six non-randomized phase I or II trials, and one registry study. For safety, four non-randomized phase I or II trials and three observational studies were included. In most instances where RWD were used there was no reported appraisal on the validity of the data (33%) or validity was reported as unknown (51%). A negative appraisal of the validity of the RWD or its source was reported in 12% of cases and this was largely due to decision-makers perceptions of the low reliability of RWD to estimate effectiveness because of the potential for bias with observational studies.5

Makady et al.5 noted differences between the HTA organizations in their use of RWD, although given the relatively small number of reports available for some agencies these trends should be interpreted with caution. All 10 NICE reports and both ZIN reports included RWD, whereas RWD were included in 3 of 13 SMC reports (23%). RWD were included in 62% of HAS reports (total N = 8) and 53% of IQWiG reports (total N = 19).5 Melanoma incidence and prevalence was the most common reason for including RWD in relative effectiveness assessment reports, accounting for 6% (SMC) to 100% (ZIN) of the agencies’ use of RWD. IQWiG and ZIN did not use RWD to inform safety or efficacy in any report. SMC used RWD for safety or efficacy in 6% to 12% of cases, HAS for 9%, and NICE for 22% of cases.5 The authors stated that the use of RWD were consistent with the agencies policies toward RWE based on previous work.5,7

The review of policy and guidance documents for Canadian, Australian and New Zealand HTA agencies, and the EUnetHTA showed similar findings as in the first Makady report.7 The agencies stated a preference for RCTs, specifically head-to-head RCTs, with greater weight assigned to well-designed RCTs over other forms of evidence (Appendix 4). RWD could provide complementary data to RCTs, but as the sole source of data is unlikely to represent conclusive evidence of treatment benefits. The agencies listed a number of situations where RWE would provide particular value to decision-making including the following: conditions that without intervention would be fatal within a short period of time (“dramatic effect”); significant unmet need; impractical to conduct RCTs due to the limited number of patients; unethical to conduct RCTs (e.g., during pregnancy); and to identify serious, long-term or rare adverse effects (Appendix 4). Some survey respondents also cited examples where RWE was used to provide efficacy and safety data versus an active comparator for drugs where only placebo-controlled trials were available. In the presence of RCT data, RWE may be used to address applicability issues, or address other outstanding uncertainties from RCTs such as persistence of effects, adherence, dosing, and utilization in clinical practice.

A report by Griffiths et al.35 examined the role of non-comparative evidence in HTA decisions. Between 2010 and 2015, a total of 549 appraisals were extracted from three HTA agencies: NICE (118 appraisals); IQWiG (169) and CADTH (262). Non-comparative evidence was considered in 38%, 12% and 13% of NICE, IQWiG and CADTH appraisals respectively, and was the only evidence presented in 4%, 4% and 6% of appraisals respectively.35 This non-comparative evidence consisted most frequently of single-arm studies (included in 13% to 24% of appraisals per agency), single-arm extension studies (3% to 14%), randomized dosing studies (5% to 13%), or other studies (3% to 8%; case series, individual patient data, audits).35 The disease states where non-comparative data were accepted most frequently included neoplasms (43 appraisals), followed by infections (21 appraisals). Thirteen appraisals that included non-comparative data were for orphan diseases.35

The non-comparative data were used to inform efficacy or safety in 30% to 33% of NICE appraisals, and 28% of CADTH reviews, but only 3% of IQWiG appraisals.35 Although CADTH and NICE were critical of the lower quality of the non-comparative evidence, these agencies were willing to consider the non-comparative evidence in the absence of higher quality data. IQWiG, in contrast, was less willing to consider non-comparative data, and it deemed non-comparative data to be acceptable in only one review for a drug for hepatitis C. The agencies stated that non-comparative evidence may be acceptable in situations where effective treatment alternatives are lacking or there is high unmet clinical need, in small patient populations where an adequately powered comparative trial may not be possible, if the anticipated magnitude of the treatment effect is sufficiently large that it would be unethical to conduct a comparative trial (e.g., hepatitis C), or the disease was life threatening and thus it would be unethical to compare against a less efficacious treatment.35

The review by Griffiths et al.,35 found that few submissions were granted a positive appraisal based solely on non-comparative evidence. NICE issued positive decisions (recommend or recommend with restrictions) in 38 of 45 (85%) appraisals that included non-comparative data, and 3 of 5 (60%) based on non-comparative evidence only.35 For CADTH, positive recommendations were reported for 22/34 (65%) of appraisals that included non-comparative data, and 11/16 (69%) of those based solely on non-comparative evidence.35 Positive recommendations were reported for 7/21 (33%) of IQWiG appraisals that included non-comparative data, and 1/6 (17%) based solely on non-comparative data.35 Among all submissions reviewed, 3%, 0.6%, and 4% were approved based solely on non-comparative evidence by NICE, IQWiG, and CADTH respectively.35 Of note, not all drug evaluators may consider evidence from non-comparative trials as fulfilling their definition of RWE. Indeed, many non-comparative studies are conducted in the same stringent context as RCTs and are thus no more reflective of the “real world.”

With respect to rare diseases, there were discrepancies noted among agencies’ policies regarding the use of RWE. In the paper by Makady et al.7 three agencies (TLV, NICE, and ZIN) stated that non-RCT data could be used for decision-making in situations where RCT data were sparse, but one stated that non-RCT data presents a greater risk to validity of conclusions and should thus be avoided (IQWiG). In their General Methods document,20 IQWiG stated that there is no convincing argument to deviate from the evidence hierarchy when assessing drugs for rare conditions; however, in case of extremely rare diseases, the requirement for parallel comparative trials may be inappropriate. In these situations, use of historical patient data may be required to assess the expected course of disease without the new treatment.20 Three other non-European HTA agencies (CADTH, INESSS, PBAC) expressed a willingness to use non-RCT data in some cases where RCTs were not feasible due to the limited number of patients (Appendix 4).

Nicod et al.36 conducted an analysis of reimbursement decisions of four HTA agencies for 10 orphan drugs. Representatives from NICE, SMC, TLV, and HAS were interviewed on a number of themes including evidentiary requirements for orphan drugs and dealing with uncertainty. None of the agencies had minimum requirements for evidence, but phase III comparative trials were preferred by all.36 The level of evidence accepted differed within the context of the clinical claim for two of the agencies. TLV required higher scientific and methodologic standards be met for interventions claiming superior efficacy with a price premium, and accepted greater uncertainty for noninferior efficacy and low price or for treating otherwise untreatable diseases.36 HAS judged the quality of evidence depending on the prevalence of the disease and availability of recruitable patients, and the relative improvement in clinical benefit rating (Amélioration du service médical rendu).36 The agencies stated that registry data and historical controls may be acceptable in certain situations: if no other data were available (NICE) or if it was the best available data (SMC, TLV); to provide long-term data on safety and efficacy, or on disease progression if no alternative treatments existed (HAS); for economic modelling (NICE); or when the disease was rare or in other exceptional circumstances (SMC).36

Impact and Implications of RWE

Although stakeholders generally agree on many uses of RWD that may contribute valuable information for regulatory and reimbursement decision-making, the use of RWE to answer questions or relative effectiveness of interventions is controversial and some question the possible impact of increased reliance on these data. At the regulatory level, acceptance of a “lower standard” of evidence and accelerated approvals may allow unsafe or ineffective products to reach the market.37,38 The authors of one paper stated that drugs approved based on data with greater uncertainty, such as non-randomized studies, uncontrolled studies or surrogate outcomes, will be a challenge for HTA organizations in making relative effectiveness assessments.39 Modelling cost-effectiveness based on such data will be subject to high uncertainty, and this uncertainty should not be underestimated by decision-makers and payers.39 Others argue that a cultural shift is necessary so that the evidence developed is not so heavily weighted toward generating precise answers to narrow questions.40 Recognition that the evidence needed to support regulatory approval and the evidence needed to inform treatment decisions are part of a single continuum will provide incentive to manufacturers and sponsors to evaluate treatment effects in real-world conditions.40 Integrating these two processes will allow progressive demonstration of a therapy’s safety and efficacy (which may include the use of RWE), and will yield a comprehensive understanding of how to use medical products in practice.40

Among the advantages listed for using RWD, external validity is frequently mentioned.6 However, some HTA agencies have challenged the assumption that RWD has inherently greater generalizability.41 Country-specific observational studies or pragmatic trials may be affected by local clinical practice patterns, thus their external validity should be examined carefully.5,24 Moreover, the purported external validity advantage of RWE is meaningless if the internal validity of the data is in question.41 It has been argued that despite advances in the methods to adjust for bias in non-randomized studies, it is unclear which methods are most appropriate in any given circumstance and the risk of confounding cannot be eliminated.39 Incomplete or invalid data is a major problem for many sources of RWD which may limit the ability to gather meaningful data.6,41,42 For example, in the Netherlands, RWD were used to evaluate bortezomib in patients with advanced multiple myeloma as part of conditional reimbursement scheme.43 RWD were useful to determine who received bortezomib and how it was administered in daily practice but it was limited in generating robust evidence of real-world safety and effectiveness, due to missing data from patient charts (for prognostic factors, efficacy measures, and harms) and due to treatment variations and dynamics in care during the new drug’s uptake in practice.43 In other examples, however, RWD were of sufficient quality to provide supporting evidence on efficacy and safety, which were of value to the regulatory and reimbursement decisions for deferiprone in Canada.44,45 Another issue raised was the potential for publication bias, which is as much a problem for real-world studies as for RCTs.41

Despite growing interest in the use of RWE in decision-making, Makady et al.5 found no substantial change in the use of RWD over time in their review of drugs for melanoma, although these findings should be interpreted with caution given the limited scope of the review and small number of reports included. The authors suggested possible reasons for the limited impact of RWD in decision-making. Robust RWD may not be available at the time when initial HTAs are conducted.5 Others have also noted this issue.6,24 Another factor suggested was the absence of guidance on systematic approaches for the inclusion, analysis, and interpretation of RWD in HTA.5 The authors noted that collaborations such as IMI GetReal and EUnetHTA are working to address some of these issues, and that further dialogue is needed among HTA agencies.5 Another potential factor is the presence of cultural barriers against the use of RWD in which adherence to evidence hierarchies automatically assesses RWE as being of lower quality or lower value.6 Two HTA agencies noted the limitations of strict adherence to evidence hierarchies, and stated that adoption of a hierarchy should not preclude the use of valuable non-RCT data.7 Makady et al.,7 commented that guidance from HTA agencies is generally lacking on the potential relevance of pragmatic trials, and as a result these may be excluded from decision-making.


The intent of the Environmental Scan was to provide a snapshot on the acceptance and use of RWE, rather than a comprehensive review. It was based on a limited literature search, and results were screened for inclusion by a single researcher. Included articles were limited to those available in English and French, thus some relevant references may have been missed or were excluded (e.g., the current version of the IQWiG General Methods paper v. 5.0, available in German only,46 or ZIN guidelines on orphan drugs).47 The scan focused on initial regulatory or reimbursement decisions and did not address the use of RWE in managed access programs or conditional approval processes where drugs are approved based on early evidence with the requirement that additional evidence to be collected to resolve existing uncertainties. In addition, the scan did not consider use of RWE in the proactive reassessment of single drugs, in class-based evaluations of multiple drugs by HTA organizations, in cost-effectiveness assessments, or in the assessment of hospital-only medical products. Given the interest in the use of RWE among HTA and regulatory agencies, their policies in regard to RWE may be evolving, and some material summarized here may be out of date.

There was no consistent definition of RWD or RWE among organizations, with some agencies not using these terms in their submission or guidance documents. A number of the included articles examined the use of uncontrolled clinical trial data in decision-making. While these are non-RCT data, they may not meet the definition of RWD. The lack of a clear and consistent definition of RWD/RWE may complicate comparisons of policies and practices between agencies.


Regulatory and HTA agencies assessing single drugs can manage the influx of RWE either at the level of study eligibility, where evidence is accepted or declined for review; or at the review stage, where evidence is appraised to draw conclusions. Findings in this Environmental Scan indicate that RWE is accepted for inclusion in single-drug technology assessments by the agencies discussed in this report; however, the way in which RWE is used and its value to decision-making depends on the clinical context, the availability or feasibility of conducting RCTs, and the agencies’ policies and practices.

The evidence hierarchies which are used by regulatory and HTA agencies place RWE at a lower level of quality or value than RCTs. Thus, RWE is used to confirm or supplement, rather than replace, the evidence from RCTs on the safety and efficacy of drugs. There was recognition of specific situations where RWE may be of particular value, such as: when RCTs are not feasible (very rare conditions) or are unethical (pregnancy), there is significant unmet need or in life-threatening conditions, and to identify serious, rare or long-term adverse effects. HTA agencies stated that the sole use of RWE to determine the comparative effects of a drug requires a prudent approach and any conclusions based on RWE alone would be more circumspect.

Recent reviews showed that non-RCT data were used infrequently to inform relative benefit assessments by regulatory bodies or HTA agencies; although for certain conditions, such as oncology, RWD use was more common. Regulatory bodies are exploring ways that RWE could play a larger role in initial market access decisions, extension of indications, or in situations where there is considerable unmet need. There is also interest in how RWE can support or satisfy post-approval study requirements.

As more information on the impact of RWE on drug marketing approval and reimbursement becomes available, the place of RWE in single-drug assessments will become more clear, which may be translated into the development of new processes and standards across the globe.


  1. Makady A, de Boer A, Hillege H, Klungel O, Goettsch W, (on behalf of GetReal Work Package 1). What is real-world data? A review of definitions based on literature and stakeholder interviews. Value Health. 2017 Jul;20(7):858-65.
  2. Goettsh W, Makady A. WP1: Deliverable D1.3. Glossary of definitions of common terms (including comments & replies from consultation rounds) [Internet]. Utrecht (NL): GetReal; 2017. Report No.: 115546. [cited 2018 Jan 18]. Available from:
  3. Garrison LPJr, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007 Sep;10(5):326-35.
  4. Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, et al. Real-world evidence - what is it and what can it tell us? N Engl J Med [Internet]. 2016 Dec 8 [cited 2018 Jan 29];375(23):2293-7. Available from:
  5. Makady A, van Veelen A, Jonsson P, Moseley O, d'Andon A, de Boer A, et al. Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA Agencies. Pharmacoeconomics. 2017 Dec 6.
  6. Makady A, Goettsh W. WP1: Deliverable D1.2 Review of current policies/perspectives [Internet]. Utrecht (NL): GetReal; 2014. Report No.: 115546. [cited 2017 Dec 7]. Available from:
  7. Makady A, Ham RT, de Boer A, Hillege H, Klungel O, Goettsch W, et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health. 2017 Apr;20(4):520-32.
  8. Rare diseases: common issues in drug development. Guidance for industry [draft guidance] [Internet]. Rockville (MD): U.S. Department of Health and Human Services, Food and Drug Administration; 2015. [cited 2018 Jan 23]. Available from:
  9. Appendix 1. FDA guidance for industry. Providing clinical evidence of effectiveness for human drug and biological products [Internet]. [Rockville (MD)]: U.S. Department of Health and Human Services, Food and Drug Administration; 1998. [cited 2018 Jan 23]. Available from:
  10. H.R.34 - 21st Century Cures Act. 114th Congress (2015-2016) [Internet]. Washington (DC): U.S. Government Publishing Office; 2016. [cited 2017 Dec 20]. Available from: Public Law No.:114-255 (12/13/2016).
  11. Use of real-world evidence to support regulatory decision-making for medical devices: Guidance for industry and Food and Drug Administration staff [Internet]. Silver Spring (MD): U.S. Food and Drug Administration; 2017 Aug 31. [cited 2017 Dec 7]. Available from:
  12. CDER Small Business and Industry Assistance (SBIA). Real-world data and evidence in drug development [Internet]. Silver Spring (MD): U.S. Food and Drug Administration; 2017 Aug 24. [cited 2017 Dec 7]. (CDER SBIA Chronicles). Available from:
  13. STAMP Commission Expert Group. Real world evidence: Agenda item 4 [Internet]. Brussels (BE): European Commission: Directorate-General for Health and Food Safety; 2016 Mar 10. [cited 2017 Dec 7]. Available from:
  14. Final report on the adaptive pathways pilot [Internet]. London (GB): European Medicines Agency; 2016 Jul 28. [cited 2018 Jan 24]. Available from:
  15. Oyebode O, Garrett Z, George E, Cangini A, Muscolo LA, Warren S, et al. Evidence requirements for reimbursement of pharmaceuticals across Europe. Int J Technol Assess Health Care. 2015 Jan;31(1-2):59-67.
  16. Resubmission criteria for the CADTH Common Drug Review [Internet]. Ottawa: CADTH; 2017 Nov 13. [cited 2017 Dec 7]. Available from:
  17. Single technology appraisal: user guide for company evidence submission template [Internet]. London: National Institute for Health and Care Excellence (NICE); 2015 Jan; updated 2017 Apr. [cited 2017 Dec 7]. (Process and methods [PMG24]). Available from:
  18. Scottish Medicines Consortium: Guidance to manufacturers for completion of New Product Assessment Form (NPAF) [Internet]. Glasgow (GB): Scottish Medicines Consortium; 2017 Nov. [cited 2017 Dec 7]. Available from:
  19. Commission de la transparence. Dossier type: première inscription ou extension des indications d'un médicament sur la liste des médicaments remboursables aux assurés sociaux et/ou sur la liste des médicaments agrées à l'usage des collectivités et divers services publics [Internet]. Saint-Denis (FR): Haute autorité de santé; 2017 Nov 17. [cited 2017 Dec 7]. Available from:
  20. Institute for Quality and Efficiency in Health Care (IQWiG). General methods 4.2. [translation]. Cologne (DE): IQWiG; 2015. Translation based on the German document Allgemeine Methoden (Version 4.2) of 22 April 2015.
  21. van der Meijden C, Grahlmann C. Dutch assessment procedures for the reimbursement of outpatient medicines [Internet]. Hague (NL): Ministry of Health, Welfare and Sport; 2010 Mar 1. [cited 2017 Dec 7]. Available from: Published jointly with the Dutch Healthcare Insurance Board (CVZ).
  22. Guide for companies when applying for subsidies and pricing for pharmaceutical products [Internet]. Stockholm (SE): Dental and Pharmaceuticals Benefits Agency (TLV); 2013 Jan 31. [cited 2017 Dec 7]. Available from:
  23. Application instructions. Reimbursement status and wholesale price for a medicinal product subject to marketing authorization [Internet]. Helsinki (FI): Pharmaceuticals Pricing Board Finland; 2017 Jan 31. [cited 2017 Dec 7]. Available from:
  24. HTA Core Model ® version 3.0. EUnetHTA JA2 WP8 deliverable. HTA core model version 3.0 for the full assessment of diagnostic technologies, medical and surgical interventions, pharmaceuticals and screening technologies [Internet]. Diemen (NL): European Network for Health Technology Assessment (EUnetHTA); 2016. [cited 2017 Dec 20]. Available from: by Work Package 8.
  25. HTA Core Model ® for rapid relative effectiveness [Internet]. Diemen (NL): European Network for Health Technology Assessment (EUnetHTA); 2015 Nov 30. (Joint action on HTA 2012-2015). [cited 2017 Dec 7]. Available from: Developed by Work Package 5.
  26. Methodological Standards and Procedures (MS) for full core HTA content development [Internet]. Diemen (NL): European Network for Health Technology Assessment (eunethta); 2015 Nov 30. (Joint action on HTA 2012-2015). Available from: Developed by Work Package 4.
  27. Guidelines for preparing a submission to the Pharmaceutical Benefits Advisory Committee (Version 5.0) [Internet]. Canberra (AU): Australian Government Department of Health; 2016 Sep. [cited 2017 Dec 7]. Available from:
  28. Guidelines for funding applications to PHARMAC [Internet]. Wellington (NZ): Pharmaceutical Management Agency (PHARMAC); 2017. [cited 2017 Dec 7]. Available from:
  29. pan-Canadian Oncology Drug Review (pCODR) submission guidelines [Internet]. Ottawa: CADTH; 2018. [cited 2018 Feb 20]. Available from:
  30. Submission guidelines for the CADTH Common Drug Review [Internet]. Ottawa: CADTH; 2014 Aug. [cited 2017 Dec 7]. Available from:
  31. Guide de soumission d'une demande à l'INESSS [Internet]. Québec (QC): Institut national d'excellence en santé et en services sociaux (INESSS); 2017 Nov. [cited 2017 Dec 7]. Available from:
  32. Hatswell AJ, Baio G, Berlin JA, Irs A, Freemantle N. Regulatory approval of pharmaceuticals without a randomised controlled study: analysis of EMA and FDA approvals 1999-2014. BMJ Open. 2016 Jun 30;6(6):e011666.
  33. Sasinowski FJ, Panico EB, Valentine JE. Quantum of effectiveness evidence in FDA's approval of orphan drugs update, July 2010 to June 2014. Therapeutic Innovation & Regulatory Science. 2015 Apr 27;49(5).
  34. Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggarwal A. Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13. BMJ. 2017 Oct 4;359:j4530.
  35. Griffiths EA, Macaulay R, Vadlamudi NK, Uddin J, Samuels ER. The Role of Noncomparative Evidence in Health Technology Assessment Decisions. Value Health. 2017 Dec;20(10):1245-51.
  36. Nicod E, Berg BK, Durand-Zaleski I, Kanavos P. Dealing with uncertainty and accounting for social value judgments in assessments of orphan drugs: evidence from four European countries. Value Health. 2017 Jul;20(7):919-26.
  37. Schwartz JL. Real-world evidence, public participation, and the FDA. Hastings Cent Rep. 2017 Nov;47(6):7-8.
  38. Zuckerman DM, Jury NJ, Silcox CE. 21st Century Cures Act and similar policy efforts: at what cost? BMJ. 2015 Nov 23;351:h6122.
  39. Woolacott N, Corbett M, Jones-Diette J, Hodgson R. Methodological challenges for the evaluation of clinical effectiveness in the context of accelerated regulatory approval: an overview. J Clin Epidemiol. 2017 Jul 12.
  40. Sherman RE, Davies KM, Robb MA, Hunter NL, Califf RM. Accelerating development of scientific evidence for medical products within the existing US regulatory framework. Nat Rev [Internet]. 2017 [cited 2018 Feb 1];16:297-8. Available from:
  41. Windeler J. Real world data - an asset for benefit assessment? How can registries and routine data contribute? [Internet]. Cologne (DE): Institute for Quality and Efficiency in Health Care (IQWIG); 2015. [cited 2017 Dec 7]. Available from:
  42. Windeler J, Lauterberg J, Wieseler B, Sauerland S, Lange S. Patient registries for benefit assessments - no replacement for randomized trials [Internet]. Cologne (DE): Institute for Quality and Efficiency Health Care (IQWIG). 2017 Apr 21 [cited 2017 Dec 7]. Available from:
  43. Gaultney JG, Franken MG, Uyl-de Groot CA, Redekop WK, Huijgens PC, van der HB, et al. Experience with outcomes research into the real-world effectiveness of novel therapies in Dutch daily practice from the context of conditional reimbursement. Health Policy. 2015 Feb;119(2):186-94.
  44. CADTH Canadian Drug Expert Committee (CDEC) final recommendation: deferiprone (Ferriprox - ApoPharma Inc.) [Internet]. Ottawa: CADTH; 2016 Mar 18. [cited 2018 Feb 1]. Available from:
  45. Ferriprox: summary basis of decision [Internet]. Ottawa: Health Canada. 2016 Oct 13 [cited 2018 Feb 1]. Available from:
  46. General methods. Version 5.0 (German version) [Internet]. Köln (Germany): Institute for Quality and Efficiency in Health Care (IQWiG); 2017. [cited 2018 Mar 14]. Available from:
  47. Pakketbeheer weesgeneesmiddelen [Internet]. Diemen (The Netherlands): National Health Care Institute (Zorginstituut Nederland); 2015. [cited 2018 Mar 14]. Available from:
  48. pan-Canadian Oncology Drug Review (pCODR) procedures [Internet]. Ottawa: CADTH; 2018. [cited 2018 Feb 20]. Available from:
  49. Commission de la transparence. Dossier type: renouvellement d'inscription ou réevaluation d'un médicament [Internet]. Saint-Denis (FR): Haute autorité de santé; 2017 Nov 17. [cited 2017 Dec 7]. Available from:
  50. Guide to the methods of technology appraisal 2013 [Internet]. London: National Institute for Health and Care Excellence (NICE); 2013 Apr. [cited 2017 Dec 7]. (Process and methods [PMG9]). Available from:
  51. Procedure for reassessment of SMC decisions - overview [Internet]. Glasgow (GB): Scottish Medicines Consortium. 2016 Apr 30 [cited 2017 Dec 7]. Available from:



About This Document

Authors: Gaetanne Murphy, Louis de Léséleuc, David Kaunelis, Lorna Adcock

Cite As: Use of real-world evidence in single-drug assessments. Ottawa: CADTH; 2018. (Environmental scan; no. 74)

Disclaimer: The information in this document is intended to help Canadian health care decision-makers, health care professionals, health systems leaders, and policy- makers make well-informed decisions and thereby improve the quality of health care services. While patients and others may access this document, the document is made available for informational purposes only and no representations or warranties are made with respect to its fitness for any particular purpose. The information in this document should not be used as a substitute for professional medical advice or as a substitute for the application of clinical judgment in respect of the care of a particular patient or other professional judgment in any decision-making process. The Canadian Agency for Drugs and Technologies in Health (CADTH) does not endorse any information, drugs, therapies, treatments, products, processes, or services.

While care has been taken to ensure that the information prepared by CADTH in this document is accurate, complete, and up-to-date as at the applicable date the material was first published by CADTH, CADTH does not make any guarantees to that effect. CADTH does not guarantee and is not responsible for the quality, currency, propriety, accuracy, or reasonableness of any statements, information, or conclusions contained in any third-party materials used in preparing this document. The views and opinions of third parties published in this document do not necessarily state or reflect those of CADTH.

CADTH is not responsible for any errors, omissions, injury, loss, or damage arising from or relating to the use (or misuse) of any information, statements, or conclusions contained in or implied by the contents of this document or any of the source materials.

This document may contain links to third-party websites. CADTH does not have control over the content of such sites. Use of third-party sites is governed by the third-party website owners’ own terms and conditions set out for such sites. CADTH does not make any guarantee with respect to any information contained on such third-party sites and CADTH is not responsible for any injury, loss, or damage suffered as a result of using such third-party sites. CADTH has no responsibility for the collection, use, and disclosure of personal information by third-party sites.

Subject to the aforementioned limitations, the views expressed herein are those of CADTH and do not necessarily represent the views of Canada’s federal, provincial, or territorial governments or any third party supplier of information.

This document is prepared and intended for use in the context of the Canadian health care system. The use of this document outside of Canada is done so at the user’s own risk.

This disclaimer and any questions or matters of any nature arising from or relating to the content or use (or misuse) of this document will be governed by and interpreted in accordance with the laws of the Province of Ontario and the laws of Canada applicable therein, and all proceedings shall be subject to the exclusive jurisdiction of the courts of the Province of Ontario, Canada.

The copyright and other intellectual property rights in this document are owned by CADTH and its licensors. These rights are protected by the Canadian Copyright Act and other national and international laws and agreements. Users are permitted to make copies of this document for non-commercial purposes only, provided it is not modified when reproduced and appropriate credit is given to CADTH and its licensors.

About CADTH: CADTH is an independent, not-for-profit organization responsible for providing Canada’s health care decision-makers with objective evidence to help make informed decisions about the optimal use of drugs, medical devices, diagnostics, and procedures in our health care system.

Funding: CADTH receives funding from Canada’s federal, provincial, and territorial governments, with the exception of Quebec.

Contact [email protected] with inquiries about this notice or legal matters relating to CADTH services.