Objectives Objective performance criteria (OPC) may serve as a tool to expedite the approval process and continue active surveillance of class III medical devices. Thus far, published guidance on the creation of OPC has been clinical area-specific. This study aimed to capture reflections from key stakeholders on the creation of OPC that may serve as a precursor for a formalized conceptual framework within the USA.
Design Reflections from key stakeholders and guidance from an advisory committee were captured to gain an understanding of the elements that are crucial to the generation of OPC.
Setting A non-probability sampling method using the purposive sampling strategy was employed to identify relevant stakeholders for engagement in semi-structured, open-ended, concept elicitation discussions.
Participants Stakeholders involved in the generation of OPC.
Main outcome measures Elements and themes regarding the priorities of, experiences with, roles within and perceived challenges associated with OPC creation captured through a phenomenological approach.
Results A total of 27 participants were engaged to represent the following contributors: representatives of registries, health systems, health technology assessment bodies, clinicians, device application reviewers, payers, patients, patient representatives, patient caregivers, device manufacturers, data coordinators, data analysts and data informaticians. Consensus was achieved on the five core elements: (1) identification of medical devices, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings. The engagement of key stakeholders (38%) was cited most frequently as the most important core element. Access to meaningful and high-quality data sources (47%) was the most frequently mentioned challenge.
Conclusions The reflections from the participants identified five elements to be considered when generating an OPC within class III medical devices and may provide the needed foundation for the development of official guidance on OPC generation.
- active surveillance
- health services research
Data availability statement
Data may be obtained from a third party and are not publicly available. Not applicable.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What is already known about this subject?
Objective performance criteria (OPC) may be leveraged to expedite the approval process and continue active surveillance. Currently, there are no frameworks detailing the necessary steps for the creation of OPC within class III medical devices.
What are the new findings?
Stakeholder engagement identified five elements that should be considered when generating an OPC: (1) identification of medical devices, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings.
How might these results affect future research or surgical practice?
The identified elements may provide the needed foundation to organizations developing clinical area-specific guidance, to government agencies providing guidance related to OPC for regulatory purposes, and to investigators contributing to OPC generation.
The Food and Drug Administration (FDA) is both one of the most vested stakeholders and the primary regulator for medical devices in the USA. The 21st Century Cures Act and the Food and Drug Administration Reauthorization Act require the FDA to decrease the approval time of devices while continuously ensuring that all approved devices are safe and effective.1 2 This expectation is especially daunting when considering class III medical devices given that these devices are implanted in a patient for several decades, and their primary purpose is to sustain life or prevent significant disability. New regulatory approaches, such as the creation of objective performance criteria (OPC), which allow for the swift approval of new devices while simultaneously ensuring that all devices entering and currently on the market demonstrate a reasonable assurance of safety and effectiveness have been explored.3–6 These numerical targets of outcomes of interest are derived from existing data and can be leveraged as comparators for the assessment of safety and efficacy.6
Many stakeholders benefit from OPC during the premarket approval process and postmarket evaluation. Given that medical devices evolve incrementally, approved medical devices currently on the market may serve as suitable comparators to devices seeking approval. OPC derived from devices present on the market can be used as target measures in single-arm clinical trials.6 7 Single-arm trials may be favorable to some stakeholders because they avoid randomization and allow all patients in a clinical trial to receive the novel device. In addition, device manufacturers may benefit from shorter and less costly trials, while clinicians may receive access to novel devices for their patients faster.5 Following a device’s approval, OPC are one way for regulatory bodies, the clinical community, reimbursement agencies and patients to monitor how devices perform outside of a trial setting through benchmarks and provide target objective performance measures for new devices. This may aid regulatory bodies in identifying devices that may need additional postmarket action including safety communication, mandated postmarket studies or compliance action. Patients, clinicians, health systems and payers may use OPC to inform treatment-related and reimbursement-related decisions. Registry representatives, data analysts and informaticians can leverage and strengthen their data sources to accurately inform OPC, potentially expanding the use and strengthening the impact of their data sources.
There are currently no specific documents or publications within the USA) exploring stakeholder perspectives regarding the needed considerations for the creation of OPC within the medical device space.4 8 The purpose of this study is, therefore, to capture reflections from key stakeholders on the creation of OPC that will serve as a precursor for a formalized conceptual framework for the creation of OPC within the USA with general criteria that may be applied to class III medical devices.
A multistakeholder advisory committee was established and met regularly to guide and oversee all aspects of this work. The advisory committee included representatives of academia, government, regulatory bodies, practising clinicians and a representative of a medical device registry used for the development of OPC and regulatory decision making.
Key stakeholders were engaged to gain an understanding of the elements they believed are integral to a framework. Key stakeholders included registry maintenance representatives, health system representatives, clinicians, device application reviewers, payers, health technology assessment (HTA) body representatives, patients, patient representatives, patient caregivers, device manufacturers, data coordinators, data analysts and data informaticians. Payers and HTA bodies were combined into one stakeholder group since many stakeholders were previously employed by an HTA body or performed value assessment within the payer organisation.
Participants and procedure
A non-probability sampling method using the purposive sampling strategy was employed to identify relevant stakeholders for engagement in semistructured, open-ended, concept elicitation discussions.9–11 Identified relevant stakeholders were invited to participate in discussion through emailed invitations. Concept elicitation discussions were set to a minimum of 25 stakeholders. If necessary, further stakeholders were identified using chain sampling.12 Stakeholders that agreed to participate were asked to identify further stakeholders that would be willing to participate. Stakeholders were engaged until no new concepts were uncovered and saturation of framework elements was reached.10 The stakeholders were asked six questions pertaining to the types of elements that need to be present in the framework, the importance of the elements, feedback on the proposed elements of the framework, the stakeholders crucial to the creation of OPC, the challenges associated with the creation of OPC and any additional information critical to a comprehensive framework. The questions were developed with the guidance of the advisory board and are outlined in table 1. The initial proposed elements of the framework were informed by clinical area-specific published guidance on OPC creation, generation, application and discussion of OPC as well as the input from the advisory board.3–5 13–15 Discussions with each individual stakeholder were held either in person or over the phone in a private setting and lasted approximately 60 min. Notes were taken during discussions with the key stakeholders. A read-back method was used to ensure that all concepts mentioned by the stakeholders were accurately captured. The discussions were not audiorecorded to allow stakeholders to engage in honest informal discussions without fear of professional repercussions.
Coding process and analyses
A phenomenological approach was implemented to summarize and describe stakeholder perspectives regarding the priorities of, experiences with, roles within and perceived challenges associated with OPC creation.16 Discussion notes were thematically analyzed using short phrases and entered into a codebook.17 The codebook was modified until no new codes were uncovered.17 A saturation grid was employed to ensure that saturation was achieved between stakeholders within the framework elements.10 Following all discussions with key stakeholders, the initial proposed elements were modified to reflect additional input from the key stakeholders (figure 1).
Thirty individuals within the stakeholder groups were approached. Eight individuals declined; therefore, five further individuals were identified using chain sampling. Of the eight individuals who declined, five represented regulatory bodies and three represented payers. In total, 27 stakeholders participated in this study. Six participants represented more than one stakeholder type. One participant represented three stakeholder types. The number of participants representing each stakeholder is summarized in table 2.
Consensus was achieved among all participants on five elements (figure 2). Within the first core element, Identification of Medical Devices for the Development of OPC, participants emphasized the need for device selection to be a conscientious and careful process as well as the need for the device to be mature enough with sufficient real-world data. During the discussion of the second core element, Identification and Engagement of Stakeholders, seven out of the 10 stakeholder groups stated that OPC generation is better accomplished as a collaborative effort. Regulatory bodies, device manufactures and clinicians were the most frequently mentioned stakeholders needed to be involved in OPC generation. Regarding the selection of an appropriate data source, 70% of participants stated that they prioriticed data sources that capture clinically meaningful and relevant outcomes. In the fourth data element, performance of appropriate statistical analyses, the determination and definition of appropriate endpoints encompassing safety and effectiveness was discussed by 80% of the stakeholder groups. Finally, 70% of stakeholder groups stated that a discussion regarding the operationalization of the OPC was needed when reporting findings. The need for transparent reporting of all analyses and findings was mentioned by 60% of the stakeholder groups. The saturation grid summarizing elicited concepts and demonstrating saturation between stakeholder groups within the framework elements is presented in table 3. Each core element consists of multiple subelements outlined in table 3.
Engagement of key stakeholders was cited by 38% of the participants as the most important element. The second most frequently cited element was the selection of the data source (23%). The participants cited six possible anticipated challenges during the OPC generation process (table 4). Access to meaningful and high-quality data sources (47%) and reaching stakeholder consensus (25%) were the most frequently mentioned challenges.
Gathered reflections from participants indicate that a framework addressing the generation of OPC within the USA should touch on the following five elements: (1) identification of medical devices for the development of OPC, (2) engagement of key stakeholders, (3) selection of data source, (4) performance of appropriate statistical analyses and (5) reporting of findings. The elicited subthemes from the participants are consistent with the limited guidance provided by the FDA on OPC and the use of RWE (Real-World Evidence) for medical device evaluation.18 19
Regarding identification of the device for OPC development, the FDA concurs device technology must be sufficiently mature and OPC need to be updated to keep abreast with the evolution of clinical practice and new technologies. FDA guidance acknowledges OPC generation is a collaborative effort and generally cannot be executed by a single stakeholder. It also recommends collaboration with medical or scientific societies or standards organizations to potentially increase the validity of the OPC and engagement with FDA staff prior to study initiation.6 Discussions with participants highlighted additional stakeholder groups to consider for engagement, including: clinicians, patients, data owners, payers, informaticians and hospital health system representatives. Engagement of patients was underscored by several participants reflecting increased attention to patients’ role in product development and assesment.20 21 The importance of the selecting appropriate and high-quality data sources was repeatedly emphasized.18 22 23 Participants indicated prioritization of datasets capturing clinically and meaningful outcomes, as well as linked datasets, such as high-quality registries and claims, should be leveraged to mitigate limitations of individual data sources.
Previous work has identified elements of appropriate statistical analysis for generation of OPC acceptable to all stakeholder groups, including: accounting for statistical uncertainty through sensitivity analyses, identifying prognostic factors of the intended population and capturing clinically meaningful and patient-centerd outcomes.5 In addition to these elements, participants in this project also cited: accounting for missing data, employing methodologies to control for confounders, applying apt regression models and conducting relevant subgroup analyses as components of an appropriate statistical analysis. Determining what constitutes appropriate statistical analysis is challenging given the potential complexity, advantages and disadvantages of various approaches. Transparent reporting of selected methods and dissemination of findings has been a point of focus for many guidelines.24 25 Specific to OPC, participants stated communicated information ought to summarize how OPC were determined, main findings, operationalization of OPC, appropriate use and known limitations of OPC, and implications of findings on clinical, patient and regulatory decision making. Finally, stakeholders noted that if one criterion of the framework cannot be met due to foreseen or unforeseen challenges, then the process of OPC generation needs to be temporarily discontinued until that criterion can be successfully met. For example, if a high-quality, available and appropriate data source is not available to assess selected relevant endpoints then OPC generation may not be possible.
Participant discussions elicited elements that may guide generation of high-quality and widely applicable OPC. The findings encompass a variety of perspectives through the engagement of participants representing a wide variety of stakeholder groups involved in aspects of development and evaluation of medical devices. Furthermore, uses and benefits of OPC vary among stakeholder groups, which may allow the elements to inform the creation of OPC with wider applicability. This study has important limitations that should be noted. Since this study leveraged discussions with key stakeholders, it is important to note that individual stakeholders provided input from their professional perspective. However, this input may not represent the perspective of all stakeholders or all individuals representing the stakeholder. Moreover, discussions were not audiorecorded. Relying on notetaking may introduce recorder bias where the notetaker determines what is important or significant enough to record in the notes. While we attempted to capture all stakeholder types involved in or affected by OPC creation, some stakeholder types indirectly involved, such as device engineers, were not included. Given that limited literature on OPC generation exists, it is important to note that these recommendations may evolve as more OPC studies and clinical area-specific guidance are published.
While previous frameworks and guidance have been tailored to specific clinical areas, discussions with participants in this study elicited foundational elements of a framework that may be generalizable to class III devices requiring OPC as well as adapted to specific clinical areas. This addresses an important gap in the literature and provides suggestions that may be helpful for device development and evaluation in clinical areas where no or little guidance exists. This study identifies five elements that may be considered in a formal framework guiding the creation of OPC within class III medical devices. Further work, including engagement of a large and diverse group of stakeholders with formal consensus building methods, is needed to create a formalized framework on OPC generation. A formalized framework can guide organizations developing clinical area-specific guidance, government agencies providing guidance related to OPC for regulatory purposes and investigators interested in contributing to OPC generation.
Data availability statement
Data may be obtained from a third party and are not publicly available. Not applicable.
Patient consent for publication
This study was approved by the University of Maryland Baltimore Institutional Review Board.
Research reported in this publication was supported by the National Center For Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR003107. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Contributors All authors participated in the design of the study, interpretation of the findings and critical review of the manuscript. LEG performed the data analysis and drafted the manuscript. LEG accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Competing interests DMD and PG are Editorial Board members of this journal.
Provenance and peer review Not commissioned; externally peer reviewed.