Creation of objective performance criteria among medical devices

Introduction

The Food and Drug Administration (FDA) is both one of the most vested stakeholders and the primary regulator for medical devices in the USA. The 21st Century Cures Act and the Food and Drug Administration Reauthorization Act require the FDA to decrease the approval time of devices while continuously ensuring that all approved devices are safe and effective.1 2 This expectation is especially daunting when considering class III medical devices given that these devices are implanted in a patient for several decades, and their primary purpose is to sustain life or prevent significant disability. New regulatory approaches, such as the creation of objective performance criteria (OPC), which allow for the swift approval of new devices while simultaneously ensuring that all devices entering and currently on the market demonstrate a reasonable assurance of safety and effectiveness have been explored.3–6 These numerical targets of outcomes of interest are derived from existing data and can be leveraged as comparators for the assessment of safety and efficacy.6

Many stakeholders benefit from OPC during the premarket approval process and postmarket evaluation. Given that medical devices evolve incrementally, approved medical devices currently on the market may serve as suitable comparators to devices seeking approval. OPC derived from devices present on the market can be used as target measures in single-arm clinical trials.6 7 Single-arm trials may be favorable to some stakeholders because they avoid randomization and allow all patients in a clinical trial to receive the novel device. In addition, device manufacturers may benefit from shorter and less costly trials, while clinicians may receive access to novel devices for their patients faster.5 Following a device’s approval, OPC are one way for regulatory bodies, the clinical community, reimbursement agencies and patients to monitor how devices perform outside of a trial setting through benchmarks and provide target objective performance measures for new devices. This may aid regulatory bodies in identifying devices that may need additional postmarket action including safety communication, mandated postmarket studies or compliance action. Patients, clinicians, health systems and payers may use OPC to inform treatment-related and reimbursement-related decisions. Registry representatives, data analysts and informaticians can leverage and strengthen their data sources to accurately inform OPC, potentially expanding the use and strengthening the impact of their data sources.

There are currently no specific documents or publications within the USA) exploring stakeholder perspectives regarding the needed considerations for the creation of OPC within the medical device space.4 8 The purpose of this study is, therefore, to capture reflections from key stakeholders on the creation of OPC that will serve as a precursor for a formalized conceptual framework for the creation of OPC within the USA with general criteria that may be applied to class III medical devices.

Methods

Coding process and analyses

A phenomenological approach was implemented to summarize and describe stakeholder perspectives regarding the priorities of, experiences with, roles within and perceived challenges associated with OPC creation.16 Discussion notes were thematically analyzed using short phrases and entered into a codebook.17 The codebook was modified until no new codes were uncovered.17 A saturation grid was employed to ensure that saturation was achieved between stakeholders within the framework elements.10 Following all discussions with key stakeholders, the initial proposed elements were modified to reflect additional input from the key stakeholders (figure 1).

Request permission

Figure 1

Methodology for the capture of stakeholder reflection regarding the creation of objective performance criteria with general criteria to be applied to class III medical devices.

Results

Thirty individuals within the stakeholder groups were approached. Eight individuals declined; therefore, five further individuals were identified using chain sampling. Of the eight individuals who declined, five represented regulatory bodies and three represented payers. In total, 27 stakeholders participated in this study. Six participants represented more than one stakeholder type. One participant represented three stakeholder types. The number of participants representing each stakeholder is summarized in table 2.

Consensus was achieved among all participants on five elements (figure 2). Within the first core element, Identification of Medical Devices for the Development of OPC, participants emphasized the need for device selection to be a conscientious and careful process as well as the need for the device to be mature enough with sufficient real-world data. During the discussion of the second core element, Identification and Engagement of Stakeholders, seven out of the 10 stakeholder groups stated that OPC generation is better accomplished as a collaborative effort. Regulatory bodies, device manufactures and clinicians were the most frequently mentioned stakeholders needed to be involved in OPC generation. Regarding the selection of an appropriate data source, 70% of participants stated that they prioriticed data sources that capture clinically meaningful and relevant outcomes. In the fourth data element, performance of appropriate statistical analyses, the determination and definition of appropriate endpoints encompassing safety and effectiveness was discussed by 80% of the stakeholder groups. Finally, 70% of stakeholder groups stated that a discussion regarding the operationalization of the OPC was needed when reporting findings. The need for transparent reporting of all analyses and findings was mentioned by 60% of the stakeholder groups. The saturation grid summarizing elicited concepts and demonstrating saturation between stakeholder groups within the framework elements is presented in table 3. Each core element consists of multiple subelements outlined in table 3.

Request permission

Engagement of key stakeholders was cited by 38% of the participants as the most important element. The second most frequently cited element was the selection of the data source (23%). The participants cited six possible anticipated challenges during the OPC generation process (table 4). Access to meaningful and high-quality data sources (47%) and reaching stakeholder consensus (25%) were the most frequently mentioned challenges.

Discussion

Gathered reflections from participants indicate that a framework addressing the generation of OPC within the USA should touch on the following five elements: (1) identification of medical devices for the development of OPC, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings. The elicited subthemes from the participants are consistent with the limited guidance provided by the FDA on OPC and the use of RWE (Real-World Evidence) for medical device evaluation.18 19

Regarding identification of the device for OPC development, the FDA concurs device technology must be sufficiently mature and OPC need to be updated to keep abreast with the evolution of clinical practice and new technologies. FDA guidance acknowledges OPC generation is a collaborative effort and generally cannot be executed by a single stakeholder. It also recommends collaboration with medical or scientific societies or standards organizations to potentially increase the validity of the OPC and engagement with FDA staff prior to study initiation.6 Discussions with participants highlighted additional stakeholder groups to consider for engagement, including: clinicians, patients, data owners, payers, informaticians and hospital health system representatives. Engagement of patients was underscored by several participants reflecting increased attention to patients’ role in product development and assesment.20 21 The importance of the selecting appropriate and high-quality data sources was repeatedly emphasized.18 22 23 Participants indicated prioritization of datasets capturing clinically and meaningful outcomes, as well as linked datasets, such as high-quality registries and claims, should be leveraged to mitigate limitations of individual data sources.

Previous work has identified elements of appropriate statistical analysis for generation of OPC acceptable to all stakeholder groups, including: accounting for statistical uncertainty through sensitivity analyses, identifying prognostic factors of the intended population and capturing clinically meaningful and patient-centerd outcomes.5 In addition to these elements, participants in this project also cited: accounting for missing data, employing methodologies to control for confounders, applying apt regression models and conducting relevant subgroup analyses as components of an appropriate statistical analysis. Determining what constitutes appropriate statistical analysis is challenging given the potential complexity, advantages and disadvantages of various approaches. Transparent reporting of selected methods and dissemination of findings has been a point of focus for many guidelines.24 25 Specific to OPC, participants stated communicated information ought to summarize how OPC were determined, main findings, operationalization of OPC, appropriate use and known limitations of OPC, and implications of findings on clinical, patient and regulatory decision making. Finally, stakeholders noted that if one criterion of the framework cannot be met due to foreseen or unforeseen challenges, then the process of OPC generation needs to be temporarily discontinued until that criterion can be successfully met. For example, if a high-quality, available and appropriate data source is not available to assess selected relevant endpoints then OPC generation may not be possible.

Participant discussions elicited elements that may guide generation of high-quality and widely applicable OPC. The findings encompass a variety of perspectives through the engagement of participants representing a wide variety of stakeholder groups involved in aspects of development and evaluation of medical devices. Furthermore, uses and benefits of OPC vary among stakeholder groups, which may allow the elements to inform the creation of OPC with wider applicability. This study has important limitations that should be noted. Since this study leveraged discussions with key stakeholders, it is important to note that individual stakeholders provided input from their professional perspective. However, this input may not represent the perspective of all stakeholders or all individuals representing the stakeholder. Moreover, discussions were not audiorecorded. Relying on notetaking may introduce recorder bias where the notetaker determines what is important or significant enough to record in the notes. While we attempted to capture all stakeholder types involved in or affected by OPC creation, some stakeholder types indirectly involved, such as device engineers, were not included. Given that limited literature on OPC generation exists, it is important to note that these recommendations may evolve as more OPC studies and clinical area-specific guidance are published.

While previous frameworks and guidance have been tailored to specific clinical areas, discussions with participants in this study elicited foundational elements of a framework that may be generalizable to class III devices requiring OPC as well as adapted to specific clinical areas. This addresses an important gap in the literature and provides suggestions that may be helpful for device development and evaluation in clinical areas where no or little guidance exists. This study identifies five elements that may be considered in a formal framework guiding the creation of OPC within class III medical devices. Further work, including engagement of a large and diverse group of stakeholders with formal consensus building methods, is needed to create a formalized framework on OPC generation. A formalized framework can guide organizations developing clinical area-specific guidance, government agencies providing guidance related to OPC for regulatory purposes and investigators interested in contributing to OPC generation.