Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
- device approval
- device safety
- learning curve
- medical device legislation
- process assessment (health care)
Regulation for therapeutic devices is getting more stringent but regulators worldwide avoid providing specific advice on study design and reporting, for market access and surveillance.
IDEAL provides guidance on appropriate methodology at each stage in the life cycle of therapeutic procedures and devices.
IDEAL is well aligned with the principles of the new EU Medical Device Regulation, so suggesting IDEAL as a template could facilitate production of appropriate evidence compliant with the new regulation, for specific devices.
Since IDEAL provides an integrated evaluation pathway, it could also prove useful in developing evidence for health technology assessment, commissioning and other purposes.
The evidence demanded by European medical device regulators is getting tougher, especially for high-risk devices. The new European (EU) Medical Device Regulation1 (MDR) has changed the evidence requirements for CE certification. The regulation states, in general terms, what kind of evidence is required for market approval and for subsequent surveillance, but it does not specify the types of studies which may be most appropriate in providing the evidence. This can pose a major challenge for innovators and developers of devices, many of whom are relatively inexperienced in planning clinical studies and limited in their capacity to fund studies. A framework specifying methodology for producing evidence throughout the lifecycle of new products would therefore be both useful and timely.
IDEAL provides a sequential framework describing the types of studies which are appropriate at each stage of the life cycle of a new procedure or device—from its first use in a human being through to its widespread use. It was originally developed for evaluating new surgical procedures, and subsequently adapted for therapeutic devices (IDEAL-D).2 3 It provides an integrated evaluation pathway, analogous to the phases 1–4 trials sequence used for new medicines, and it could therefore be a useful adjunct to device regulation.
This article analyses the alignment between the evidence demands of the new EU MDR, and the recommendations of IDEAL-D. The findings highlight how adopting IDEAL-D could help innovators in generating appropriate evidence on new devices for MDR licensing purposes. The similarities in regulatory requirements internationally suggest that our conclusions may be relevant to future developments in the USA, the EU and any new UK system.
The Medical Device Regulation
The MDR was developed in the context of device proliferation, increasing speed of innovation, high-profile cases of devices causing harm, and demands for better clinical evidence around risks—especially for implantable devices and high-risk devices. It sets out the evidence requirements for obtaining a CE certificate, and replaces several previous Medical Device Directives. See box 1 for the main changes it introduces.
Key changes introduced by European devices regulation (Medical Device Regulation)
Changes to classification rules
Some devices will be reclassified into a higher risk group, requiring increased scrutiny and clinical evidence premarket and post market.
The scope of the regulation will be increased, to include some devices without a medical purpose (eg, cosmetic contact lenses, dermal fillers, liposuction devices, lasers for hair/tattoo removal).
Unique device identification will be required, providing increased information for safety alerts, potential recalls and surveillance activities.
Introduction of premarket clinical scrutiny of selected novel high-risk devices by ‘Expert Panels’.
Clearer and expanded indications for Competent Authorities (eg, Medicines and Healthcare Regulatory Authority) to inspect manufacturing and clinical investigation sites.
More rigorous vigilance reporting requirements, including continuous assessment of potential safety risks via:
Mandatory postmarket clinical follow-up reports
Mandatory periodic safety update reports.
More prescriptive regulation of Notified Bodies in relation to:
Performance and conduct of business
Market surveillance role
Coordination of vigilance through increased networking and communication
Changes to obligations for manufacturers, importers, distributors and authorised representatives
Changes to clinical evidence requirements:
More stringent requirements for clinical evaluation and for claiming equivalence.
New standards for clinical evidence in conformity assessment
The manufacturer must proactively collect and evaluate clinical data from use in humans.
Clinical evaluation will be viewed as a continuous process throughout the lifecycle of the device
For class III devices and implantable devices, manufacturers will be required to publish key safety and performance data and the outcome of the clinical evaluation in a “Summary of safety and clinical performance” document.
The MDR represents a shift from a binary ‘certified/not certified’ model to a ‘life-cycle’ approach, placing increased emphasis on ongoing generation of clinical evidence after certification. Revision 4 of the Medical Devices (MEDDEV) 2.7/1 guidance document4 forms the basis of the MDR’s tougher requirements, including the need for clinical investigations for higher risk devices. Devices are classified from class 1 (low risk) through 2a and 2b to class 3 (high risk)5 : all class 2b implantable and class 3 devices will require clinical investigations. Devices considered compliant under previous directives are not guaranteed automatic certification under the new MDR. All existing devices will therefore have to be reviewed, and their supporting evidence enhanced where necessary. However, like the directives, the MDR does not specify the types of studies which should be performed to achieve this.
Evidence requirements for CE certification under the MDR
Under the MDR, the need to demonstrate compliance with standards in areas such as toxicology, functional safety, biocompatability, usability and sterilisation remains unchanged. Most of this testing is done before any clinical use.
Thereafter, the MDR requires clinical evidence from at least one well designed clinical study, to assess the safety and performance of a device in patients, unless sponsors can demonstrate sufficient clinical data applicable to their device from other sources. The evidence needed varies depending on the risk classification of the device.
The precise design of these clinical studies is not specified in the MDR and neither the national regulatory authorities (termed Competent Authorities in the EU) nor the Notified Bodies (which perform the assessments prior to issuing CE certification) are permitted to advise applicants on study design or data requirements (see box 2). This deliberate policy of avoiding detailed guidance on evaluation methods allows flexibility, and prevents any suggestion that certification is guaranteed if a specified type of study is done. It is up to manufacturers to justify study designs.
The European (EU) system for medical devices conformity assessment
Certificate of conformity (CE mark)
Required for devices to enter the EU market
Indicates that the device
Performs the functions is it marketed for
Is safe in normal use.
Are performed by commercial companies called Notified Bodies (NB).
Are national government agencies in each EU country which regulate medical devices, by:
Reviewing the work of NB.
Advising whether proposed evaluation plans are acceptable.
Companies with a new class 2 or 3 device need to generate evidence for a clinical evaluation to submit to an NB. (class 1—low risk—devices can obtain a CE mark by a process of self-registration.)
Companies may seek advice from the Competent Authority, but the authority cannot suggest evaluation plans.
The evaluation is carried out by the company and the results are analysed and reviewed by the NB
The NB decides whether the evidence meets the standards of EU law for performance and safety.
If the NB is satisfied, a CE certificate is issued.
This process is protected by commercial confidentiality agreements between the company and the NB. Only the national Competent Authority may see the data and inspect the company’s facilities.
Current UK status
In the UK, the Medicines and Healthcare Regulatory Authority is the Competent Authority. UK policy on device regulation is currently under review following Britain’s departure from the EU.
The Competent Authority of each EU country must independently approve any clinical investigation work in its jurisdiction. It must be informed of any adverse events, and can stop or suspend clinical studies nationally . The authorities are provided with the study results, but do not normally review them unless there are clear reasons for concern.
The ‘clinical evaluation’, incorporating the results from the clinical investigation and/or other relevant clinical data is reviewed by a Notified Body (anywhere in the EU), together with broader technical documentation. The information supplied to the manufacturers must demonstrate that the device is safe and performs correctly when used as intended. For high-risk devices, a CE certificate is issued if there is sufficient evidence that the benefit of the device outweighs the risk, considering its intended purpose. For lower risk devices technical documentation is reviewed on a defined sample of devices. Evidence of superiority over similar devices is not necessary, although consideration of currently available alternative treatments is required. There is no consideration of cost. In certain circumstances, Notified Bodies may place restrictions on a CE certificate—for example, allowing use only in the context of an approved postmarket surveillance study.
Regulatory requirements after CE certification
Unlike previous EU directives, the MDR requires an explicit plan for post-market clinical studies and surveillance, but it does not specify precisely what form these should take. This applies to all device risk classes, but more detailed surveillance is expected for higher risk devices. Manufacturers will be required to submit regular assessments of device performance through Post-Market Clinical Follow-up reports (PMCF) and Periodic Safety Update Reports (PSUR). The EU device database, Eudamed, will compile key safety information.
The IDEAL framework
The IDEAL framework acronym spells out the evolutionary stages of complex therapies: Idea, Development, Exploration, Assessment and Long-term Study. The IDEAL recommendations describe study design and reporting proposals for each stage.6
Box 3 shows the stages in the IDEAL-D framework and some of the key recommendations for studies at each stage. Each stage of IDEAL-D is focused around a key question: stage 0—what is the new device and what does it do? This stage addresses preclinical testing: it was not included in the original IDEAL framework, but is important for devices. A very wide range of studies may be relevant, depending on the nature of the device.
Summary of key IDEAL recommendations
General recommendations for all stages:*
Patient informed consent for treatment should include explanation of the current level of experience with the innovation, known risks and potential unknown risks.
Outcomes, patient characteristics and confounders should be described using well-understood, standardised and validated measures
Patient baseline demographic and clinical characteristics should be reported, including the selection criteria and process for treatment, and numbers of patients excluded, with reasons.
All harms, unexpected events or unintended effects should be reported for each patient.
* In stages 1 and 4, informed consent may not be possible in emergency situations (1) or where ‘real world’ data sources are used (4).
IDEAL stage 1: idea
Rationale/need for the new treatment should be explained
The preclinical development and testing of the technique should be summarised.
Patient selection, and patient and disease characteristics should be clearly described
The new technique/device, should be clearly described, including pre/postprocedure care.
Key clinical and technical outcomes including any adverse events should be reported.
IDEAL stage 2a: development*
The study objectives should include progression towards a stable version of the innovation.
The study design will normally be a sequentially reported prospective case series.
The initial technique/device should be described clearly, including pre/postoperative care
All cases should be reported in sequence, indicating what modifications to technique, device or indications occurred, when and why, and displaying consecutive outcome data graphically.
The discussion should report whether the innovation has reached stability in the hands of the operators, making it ready for evaluation in a multicentre IDEAL stage 2b study.
IDEAL stage 2b: exploration*
Consensus onwhether an randomised controlled trial (RCT) is appropriate/feasible should be a study objective.
The study design will typically be a prospective multicentre cohort study.
Patient inclusion and exclusion criteria should be clearly defined, specifying controversial patient subgroups or technique/device variants for separate outcome reporting.
The technique/device should be clearly described together with a suitable measure of quality of performance/implantation
A method for evaluating operator learning curves should be described, and results reported.
Qualitative study of patient and surgeon preferences and values, relevant to future RCT trial design and feasibility, should be conducted and reported
A planned review of early results should drive discussion of future RCT feasibility and design.
IDEAL stage 3: assessment
At this stage, treatments are normally subjected to a randomised trial to compare outcomes with current state of the art treatment. IDEAL recommends following Consolidated Standards of Reporting Trials guidance (http://www.consort-statement.org/) for RCTs with the following additions:
Reducing recruitment problems by using decision support aids, trained research nurses or investigator training (eg, QuinteT) to avoid transmitting unconscious bias
Using quality measures to determine the fidelity with which treatments were delivered.
IDEAL stage 4: long-term study
Specific study objectives should be stated, for example, assessing late or rare safety outcomes.
The study design should be clearly defined (eg, registry, cohort analysis of real-world data):
All key data fields should have clear, accessible definition so detailed account of inclusion and exclusion criteria for subjects
Description of the intervention/device studied, and comparator if applicable
Description of prespecified primary and secondary outcome measures
Registry funding, and responsibility for design, curation and management should be clarified, addressing possible conflicts of interest.
The extent of missing data for each variable of interest should be reported, if appropriate
The main prespecified outcome measures should be reported, including outcome variations among prespecified subgroups and adjustments for confounders, when applicable
* In IDEAL-D studies to address the issues in stages 2a and 2b may be combined into a single stage 2 study. For a fuller account of IDEAL Recommendations and reporting guidelines, see the IDEAL website (http://www.ideal-collaboration.net/) and http://www.ideal-collaboration.net/wp-content/uploads/2019/03/IDEAL-Update-Table2_AnnalsSurg2019.pdf.
Stage 1 (Idea)—Does the device work as intended in a patient? This stage describes the initial use in patients, including detailed description of patient selection and informed consent, the device, its mode of action, implantation or activation and the outcomes of use.
Stage 2a (Development)—Has the device been developed to a stable state? Modifications to the device, its use or patient selection criteria, and consequent changes in outcomes must be explained and documented. Reporting should therefore include a case-by-case sequential display of outcomes.
Stage 2b (Exploration)—Has clinical consensus been reached on indications, manner of use and quality standards? With wider use, variation develops in both use and patient selection. Variation in experience and training can affect both effectiveness and risks. IDEAL recommends collaborative multicentre cohort studies at this stage, incorporating prespecified subgroup analysis to resolve known controversies, and statistical evaluation of learning curve effects.
For devices, the IDEAL-D variant3 can be used, where stages 2a and 2b can be combined into a single stage 2 study with the potential to alter its focus as it develops.
Stage 3 (Assessment)—Is the new device better than current standard of care?
This question is best addressed in a randomised controlled trial (RCT). However, RCTs may be infeasible for small iterative developments not claiming superior effectiveness and an IDEAL 2b-like study may then be adequate. Real-world evidence (RWE) approaches to evaluating comparative effectiveness are becoming feasible, but require great care in ensuring adequate data quality and avoiding common sources of bias.7
Stage 4 (Long-term)—Does surveillance reveal any unexpected outcomes or trends?
IDEAL-D proposes continuing surveillance, preferably in a comprehensive prospective registry—which may have data linkages to routine healthcare and population databases containing high-quality RWE.
How IDEAL-D could guide evidence generation for CE marking
The MDR illustrates how regulators have adopted a strategy of giving general advice but avoiding specific methodological requirements for clinical evidence for therapeutic devices. At every stage, there are guidance statements which permit a broad range of interpretations. This model clearly has value for regulators, but it poses significant problems for product developers. The temptation to opt for the cheapest way of complying with imprecise statements of principle is likely to depress evidence quality. Conversely, lack of specific guidance increases the risk of being asked for additional studies, resulting in expense and delay for innovators, who are required to develop their own study designs and reporting models instead of conforming to a standard. It also results in major heterogeneity between studies, which may affect equity of treatment for innovators, and the comparability of data in studies of similar devices. The detailed recommendations for study design and reporting in IDEAL-D are fully in tune with the principles of the MDR, and provide clear guidance on how to develop evidence, where the MDR does not.
By defining the types of studies generally needed at different stages of a device’s life cycle, IDEAL-D could provide an independent standard to inform regulatory study design, while still allowing Competent Authorities to comply with their obligation not to prescribe. Since it is an integrated evaluation pathway, IDEAL-D can also allow forward planning of study methods throughout the life cycle.
Studies complying with the recommendations for IDEAL-D stage 0, 1 and the 2a-like part of stage 2 are likely to supply sufficient evidence for CE certification in most cases (figure 1). Devices in the highest risk categories (eg, Transcatheter Aortic Valve Replacement8) might require IDEAL Stage 2b-like or 3 studies, but for most devices these formats would be more relevant for postmarketing clinical follow-up. The preferred design of the proposed new follow-up studies (PMCF and PSUR) under the MDR is not specified, but they will need to study large numbers of patients prospectively, and allow analysis of patient subgroups and variations in device design and in techniques. These are typical features of the IDEAL stage 2b cohort study, which could therefore be a suitable template.
The MDR does not discuss when an RCT (IDEAL stage 3) is necessary for CE certification. Broadly speaking, innovative and highly novel devices which claim superior effectiveness or are high risk (Class 3) will need RCT evidence, while other devices usually will not—but they will require some evidence from clinical studies. For this situation, the IDEAL stage 2b-like design would again be an appropriate choice.
For long-term postmarket surveillance, likely to be needed for class 2 and 3 devices, IDEAL-D recommends prospective registries in stage 4, and specifies what their characteristics should be (figure 1).
IDEAL-D was developed to guide development of high-quality evidence of safety and effectiveness over the life cycle of a device, by focusing on the key questions at each stage. The MDR was developed to rationalise, clarify and tighten device regulation, by systematising the requirements for clinical evidence provision. Their different objectives explain their differences, but they have a great deal in common.
Both IDEAL-D and the MDR emphasise the need to provide different types of evidence at different stages in the device life cycle. Both recommend preclinical studies providing technical information on device performance and safety, and then data on safety and technical success in early clinical studies. Both recognise the importance of high-quality, high-volume data about outcomes in real-world use during early clinical experience. These points of similarity make it possible to align the clinical evaluation process and the IDEAL Framework (see figure 1). In many respects, conformity with the appropriate IDEAL-D stage recommendations is likely to ensure conformity with the regulation. This concordance could help to resolve the problems posed for innovators by the avoidance of specific advice in the MDR. There is a broad consensus among regulators on the types of evidence needed for different devices types at different stages,5 so the template provided by IDEAL-D might be useful in other jurisdictions, as well as providing a simple way to align with MDR requirements.
A system such as IDEAL-D cannot be formally incorporated into MDRs, but if it assists developers in providing relevant evidence, then regulators could signal this by suggesting it is considered by potential applicants for CE certification (and for regulatory approval in other jurisdictions).
By defining the types of studies needed at different stages, IDEAL-D could guide forward planning of study methods throughout a device’s life cycle, helping to optimise speed and efficiency, and to reduce the cost of evidence generation—not only for regulation but also for health technology assessment, commissioning of services, and clinical research.9 This would reduce the waste, cost and inefficiency associated with developing different evidence for a series of different audiences.
The authors would like to thank Graeme Tunbridge (Director of Devices at the MHRA for advice and suggestions which contributed considerably to the article. The IDEAL Collaboration is partly funded by the Oxford Biomedical Research Centre and through grants from the National Institute for Health Research
Contributors CF wrote the first draft, liaised with others to ensure that the article was compatible with MHRA policy and advised on later drafts and figures. BC proposed the idea for the article, supervised the collaboration between the organisations and authors, and edited or commented on all drafts and authored several redrafts. PM, as Chair of the IDEAL Collaboration, ensured that the potential application of IDEAL was accurately represented, wrote the final draft and is the guarantor of the article.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests CF is an employee of the MHRA, and BC is a Non-Executive Member of the MHRA Board. PM has no direct conflicts of interest to declare. The IDEAL Collaboration has received funding from Medtronic, Johnson & Johnson and the Chinese National Engineering Research Centre for Ultrasound Medicine, all as unrestricted educational grants.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.