Objectives A multistakeholder expert group under the Women’s Health Technology Coordinated Registry Network (WHT-CRN) was organized to develop the foundation for national infrastructure capturing the performance of long-acting and permanent contraceptives. The group, consisting of representatives from professional societies, the US Food and Drug Administration, academia, industry and the patient community, was assembled to discuss the role and feasibility of the CRN and to identify the core data elements needed to assess contraceptive medical product technologies.
Design We applied a Delphi survey method approach to achieve consensus on a core minimum data set for the future CRN. A series of surveys were sent to the panel and answered by each expert anonymously and individually. Results from the surveys were collected, collated and analyzed by a study design team from Weill Cornell Medicine. After the first survey, questions for subsequent surveys were based on the analysis process and conference call discussions with group members. This process was repeated two times over a 6-month time period until consensus was achieved.
Results Twenty-three experts participated in the Delphi process. Participation rates in the first and second round of the Delphi survey were 83% and 100%, respectively. The working group reached final consensus on 121 core data elements capturing reproductive/gynecological history, surgical history, general medical history, encounter information, long-acting/permanent contraceptive index procedures and follow-up, procedures performed in conjunction with the index procedure, product removal, medications, complications related to the long-acting and/or permanent contraceptive procedure, pregnancy and evaluation of safety and effectiveness outcomes.
Conclusions The WHT-CRN expert group produced a consensus-based core set of data elements that allow the study of current and future contraceptives. These data elements influence patient and provider decisions about treatments and include important outcomes related to safety and effectiveness of these medical devices, which may benefit other women’s health stakeholders.
- contraceptive devices
- obstetrics and gynecology devices
- real world evidence
- women's health
Data availability statement
Data are available upon reasonable request. The data that support the findings of this study are available from the corresponding author, CEB, upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
What is already known about this subject?
The evaluation of contraceptive medical devices and technologies throughout the total product life cycle is often limited by the lack of interoperable real-world data (RWD) sources. As such, stakeholders identified the need to create a Coordinated Registry Network (CRN) that would be capable of generating RWD to evaluate the safety and effectiveness of long-acting and permanent contraceptives.
What are the new findings?
Regulators, clinicians, patients and manufacturers were engaged in a Delphi process to reach consensus on the core data elements that would be needed for evaluation of contraceptive technologies within the CRN. Stakeholders identified 121 core data elements that would be applicable to both existing long-acting and permanent contraceptives and new contraceptives that would enter the market in the future.
How might these results affect future research or surgical practice?
Collection of these core data elements within registries and clinical studies and RWD sources will create unbiased and high-quality data sources for evaluation of devices and technologies used for contraception. This process will also improve active surveillance capabilities to advance public health.
The evaluation of medical devices throughout the total product life cycle (TPLC) is often limited by the lack of interoperable real-world data (RWD) sources. Between 2012 and 2013, the Center for Devices and Radiological Health (CDRH) of the Food and Drug Administration (FDA) released a plan to strengthen the postmarket infrastructure for the monitoring of medical devices.1 2 This plan has evolved into the implementation of the National Evaluation System for health Technologies (NEST), to more efficiently generate better evidence for medical device evaluation and regulatory decision-making throughout the TPLC by strategically and systematically leveraging real-world evidence and applying advanced analytics unique to medical devices.3 In 2016, the Medical Device Innovation Consortium established the NEST coordinating center (NESTcc) to promote collaborations and oversee activities.4
In 2014, the National Registry Task Force was established as part of the implementation of the CDRH initiative to ‘strengthen the medical device postmarket surveillance system using existing resources and under current authorities and to develop an integrated system that efficiently and effectively achieves its basic functions, from timely identification of postmarket signals to facilitating premarket device clearance and approval’.5 The concept of Coordinated Registry Networks (CRNs) was introduced by the National Registry Task Force in 2015, as no single database exists that can answer all the questions related to the performance of a medical device, indicating an opportunity for databases to be augmented by linkages to other data sources.6 The purpose of a CRN is to develop ‘strategically partnered electronic health information systems that support (1) the implementation of structured device identifiers, core minimum data elements and definitions and (2) the ability to share complementary data across information systems’.6 There are several existing CRNs and other data networks (virtual CRNs) rapidly developing into important data sources for NEST. To support their maturation, the Medical Device Epidemiology Network (MDEpiNet), one of the NESTcc data network collaborators, has worked towards the establishment of the Community of Practice (COP).7 Through this COP, robust CRNs are being developed to serve as a productive node within NEST.
To advance the data infrastructure and capacity to conduct patient-centered outcomes research, the FDA partnered with the Office of the National Coordinator (ONC) for Health Information Technology, the National Library of Medicine (NLM) and a multistakeholder community coordinated by MDEpiNet in a large project funded by the Patient-Centered Outcomes Research Trust Fund administered by the Office of the Assistant Secretary for Planning and Evaluation of the US Department of Health and Human Services.8 In doing so, stakeholders identified the need to create a CRN that would be capable of generating RWD to evaluate the safety and effectiveness of long-acting and permanent contraceptives (both drugs and devices).
Stakeholders acknowledged the importance of creating a core minimum data set that would be applicable to both existing long-acting and permanent contraceptives, and new contraceptives that would enter the market in the future. Core data elements, relevant to reproductive health, that are captured across healthcare systems can help address clinical concerns and questions that arise with respect to contraceptive technologies used in the general population. As a first task, we used the Delphi survey method to reach consensus on a core minimum dataset for the study of current and future contraceptive medical products.
Materials and methods
On 15 September 2017, stakeholders including the FDA, industry, non-profit organizations, patient advocacy group members, payers, professional society leaders, academia and clinical experts met to initiate the CRN.9 At this meeting, the current landscape of data sources for evaluating long-acting and permanent contraceptives were discussed along with stakeholder perspectives. Breakout sessions were held for stakeholders to discuss data elements that should be included in the Delphi survey process to build the core minimum data set.
In the fall of 2017, we established the long-acting and permanent contraceptives working group. This working group included two cochairs from the FDA, two external clinical cochairs and 19 representatives from the American College of Obstetricians and Gynecologists, the Society of Family Planning, American Association of Gynecologist Laparoscopists, FDA (CDRH and Center for Drug Evaluation and Research) and the Centers for Disease Control and Prevention. Working group members were selected by asking each of these stakeholder groups to recruit volunteers to participate in the working group. During this process, the FDA cochairs reviewed incoming participant suggestions to ensure adequate diversity and representation of various stakeholder perspectives. The full list of working group members is reported in online supplemental file 1. Initially, this working group created an initial set of about 300 data elements that were drawn from the meeting recommendations, a review of the literature, regulatory requirements and existing research efforts. The working group streamlined that list to 95 data elements relevant to contraceptive outcomes.
The Delphi survey process was then initiated to identify the core minimum data elements for long-acting and permanent contraceptives. The Delphi survey method is a group decision-making technique that was developed by Norman Dalkey and Olaf Helmer in the 1950s as a way of addressing a specific military problem.10 Traditionally, group decision-making relies on a consensus panel approach that brings experts together and let them discuss an issue until a consensus emerges. Challenges with this traditional approach are that one person with a strong personality can have a large effect on the decision and a lack of anonymity may introduce response bias. The Delphi survey method was developed to retain the strength of a joint consensus, while removing the potential bias from group dynamics and face-to-face responses. With the Delphi survey method, group input is received through a series of anonymous surveys, which are sent to a preselected group of experts. The questionnaires are answered anonymously and individually by each member. Each survey also provides an opportunity for group members to introduce new options and suggestions in between rounds. Results of each survey round are collected, collated and analyzed by a design team. This process is repeated until a group consensus is reached.11–14 Most of these methodologies are standardized for CRN purposes and also used in maturity model framework development,15 and core minimum dataset development for pelvic organ prolapse,16 uterine fibroids,17 peripheral artery interventions,18 and prostrate ablation."19
The focus of the Delphi survey process was to identify a minimum set of data elements relevant to safety concerns for contraceptive technologies that we hypothesized would be either easy to obtain from most electronic health record (EHR) systems or to be entered by clinicians. With these in mind, the Delphi survey process focused on the following categories: (1) patient characteristics, (2) medical and surgical history, (3) procedural data, (4) relevant medications and (5) patient outcomes. Two rounds of surveys were designed and administered by the analysis team at Weill Cornell Medicine and sent to the expert panelists through a secure anonymous online questionnaire (https://www.surveymonkey.com). In each round, we retained data elements that had strong consensus (eg,>50%) and dropped responses that had poor consensus (eg, <40%). For results with moderate consensus, between 40% and 50%, the cochairs discussed considerations and either raised concerns to group members for discussion or revised them to achieve clarification based on respondents’ concerns. Group members were then resurveyed. Consensus was achieved after a total of two rounds were completed.
A patient partner with knowledge of and experience with contraceptives was chosen through a formal standardized selection process and invited to join the working group to ensure that the core minimum dataset reflected the needs and interests of the patient community. The patient partner, selected through this process in spring 2018, reviewed and approved the core minimum data set and currently serves in a volunteer capacity by participating in working group meetings to provide input on patients’ perspectives.20
The Delphi survey method was implemented over a 6-month period from February 2018 to July 2018. The participation rates in the first and second round of the Delphi survey were 83% and 100%, respectively. Completion of the Delphi survey process resulted in 121 data elements identified as relevant to long-acting and permanent contraceptives. Online supplemental appendix 1 lists and categorizes the data elements into medical history, procedure, medications and end points during and after treatment. The level of consensus for both the final data elements and the dropped data elements is reported in online supplemental files 2 and 3. We did not include patient demographic variables in the Delphi surveys. Instead, an existing standard, harmonized set of demographic variables were selected for the core minimum data set that are based on work already conducted by a multistakeholder project sponsored by the Pew Charitable Trusts.21 A critical deliberation that occurred in the group when selecting variables involved considering data that can be extracted from current record systems versus data which needed direct input from the physicians. The general consensus was to minimize unique variable input that would not otherwise be included in routine documentation. The intent is that with data extraction from EHR discrete fields, there will be improved data integrity.
Regarding the reproductive/gynecologic history, consensus among the group was to include not only pregnancy history but other gynecologic historical elements. These variables are believed to be important not only for their influence in provider decision-making regarding contraceptive options but also to determine the potential longitudinal impact that the technology may incur. Some of these items were specifically chosen not only to address issues that have been identified with prior technologies, such as infection or menstrual abnormalities, but also included several broader-based items such as breast and gynecological cancer that may be influenced by these products.
As a contraceptive registry, consensus around the primary end point of pregnancy was easily achieved. Much of the discussion in the group was regarding appropriate capture of variables that will be meaningful for current technologies but robust enough to cover future contraceptive product development. Elements such as thermal injury and perforation needed to be carefully considered for the potentially wide range of contraceptive products. Ultimately, the variables were individualized for the different classifications of long-acting reversible contraception/sterilization such as hysteroscopic sterilization versus contraceptive implants. An important discussion included the use of confirmation imaging for procedures that require or would voluntarily be utilized.
The ultimate goal of the overall Women’s Health Technology CRN (WHT-CRN) project was to create a CRN for numerous women’s health conditions. As such, core minimum data sets were concurrently developed for pelvic organ prolapse, stress urinary incontinence, uterine fibroids as well as long-acting and permanent contraceptives. In order to create a CRN capable of evaluating contraceptive medical product devices used for all of these conditions, it was imperative to harmonize common data elements (CDEs) among all of the clinical areas to ensure interoperability of data sets stemming from future registries. After the core minimum data sets were developed for each individual condition, the WHT-CRN informatics team, consisting of representatives from FDA informatics, NLM and ONC, created the core minimum set of harmonized data elements. First, the informatics team compared terms across each clinical registries’ data elements and questions. They identified and grouped common concepts that occurred in at least two of the registries. Together, the informatics team and clinical working groups reviewed and refined the common concepts in an iterative, collaborative process. During this process, they harmonized individual data element definitions, questions and permissible values, until they arrived at an initial core minimum set of harmonized data elements.
The informatics team drew the permissible values from literature, clinical working group feedback and standard clinical vocabularies, including the Value Set Authority Center (VSAC), which is provided by NLM in collaboration with ONC and the Centers for Medicare & Medicare Service (CMS).22 VSAC is a repository of codes and terms from NLM-hosted standard clinical vocabularies (LOINC, SNOMED CT and others) and an authoring tool for public value sets. The WHT-CRN informatics team created VSAC grouping value sets, that is, lists of several value sets that share a common purpose and similar clinical concepts and are identifiable with VSAC object identifiers.
The initial core minimum set of CDEs was delivered to ONC for inclusion in the WHT-CRN Health Level Seven International Fast Healthcare Interoperability Resources (FHIR) Implementation Guide (IG).23 The WHT-CRN FHIR IG focuses on capturing data related to women’s health and making that data available for exchange to both providers and authorized researchers. In the WHT-CRN FHIR IG, the CDEs are mapped to FHIR resources and profiles. The IG also includes the recommended value sets, capability statements and implementation guidance for developers. The IG leverages the US-Core IG, Structured Data Capture FHIR IG, Data Access Framework for Research IG and the Patient-Reported Outcomes FHIR IG.
ONC partnered with two sets of organizations to develop and pilot test the structured framework and technology described in the WHT-CRN FHIR IG.24 The data collection instruments (CRN instruments) were defined by these CDEs to test six CRN capability statements, which outline conformance requirements (specific profiles, operations, security, search parameters, etc) for each real-world system. The six capability statements are (1) ability to publish a CRN instrument; (2) ability to retrieve the instrument, render the instrument and collect the necessary data; (3) ability to retrieve, render and auto-populate the CRN instrument and collect additional data; (4) ability to retrieve and render the CRN instrument and collect data and transform data into FHIR resources; (5) ability to receive CRN instrument and collected data; and (6) ability to receive CRN instrument, collected data and other FHIR resources. Please see figure 1a + b for illustrations of how the WHT-CRN data will be collected and accessed by organizations. Please see online supplemental file 4 for further clarification on these concepts and a full list of acronyms used in this manuscript.
Data elements from the clinical working groups that were not included in the core minimum harmonized set in the IG are nonetheless valuable. For example, data collection instruments may contain CDEs from the core minimum set as well as non-core data elements. The non-core data elements should be equally well specified, that is, uniquely identifiable, clearly defined, with clear provenance, and linked to standardized terminology. The WHT-CRN informatics team has identified and begun defining these elements in the data dictionaries, which will complement the core minimum set.
Single-purpose registries face many challenges in addressing questions involving multiple therapies and conditions. By leveraging fewer resources to collect predefined data for a greater number of conditions and therapies, this CRN has a potential to improve real-world evidence generation while saving time and reducing cost. We envision this CRN demonstrating that data from different sources (eg, EHRs, registries) may be used to: (1) evaluate the effectiveness, quality of life and safety associated with differing treatment options; (2) provide a framework for clinical studies to be conducted within the infrastructure, including industry-sponsored studies required to fulfill FDA’s request for premarket and/or postmarket regulatory activities and (3) allow healthcare providers to track physician volume, patient outcomes and quality measures for quality improvement activities and fulfill upcoming CMS, Merit-Based Incentive Payment System (MIPS) and maintenance of certification requirements.
By using a Delphi survey method that engaged multiple stakeholders with varying perspectives, we were able to achieve consensus on a minimum data set of variables capable of evaluating the performance of contraceptives. One of the key steps in creating the CRN for contraceptives was identifying a core data set that would require minimal data entry by clinicians but would also be comprehensive enough to conduct safety and effectiveness studies. It was also important that the core data set would be well suited for new contraceptive technologies that enter the market in the future. Our stakeholders foresee many applications for new contraceptive devices and technologies in the coming years, and we believe that the data elements captured in this CRN are generalizable to both existing devices and novel technologies that may enter the market in the future.
We recognize that our core elements are limited to documentation in EHRs and may not fully reflect the experiences of patients. Our measures are more reflective of safety concerns and less reflective of patient experience. For example, our post-procedure data elements largely reflect concerns for adverse outcomes like surgical complications and unintended pregnancies. However, we do not capture the experiences of ongoing users or reasons for discontinuation such as low libido or dyspareunia. Additional work is planned to develop the data elements that reflect the experiences of the patients beyond the documentation in EHRs.
There are limitations to our methodology to identify core data elements for long-acting and permanent contraceptives. While the Delphi survey method overcomes some of the limitations of group consensus, it is still possible that input was not received from all participating stakeholders. Our process relied on voluntary time from working group members, and there were variable rates of participation in each round. Furthermore, even among those participating, if members were not represented in each round (eg, follow-up phone calls for consensus), certain data elements may have been missed although there was 100% participation in the second round to reduce that concern. Additionally, we recognize that concerns regarding long-acting contraceptives differ by method (implant vs intrauterine system) and are often different than concerns for permanent contraceptive methods. Nevertheless, we leaned towards an inclusive approach to acknowledge these varying concerns. Strengths of our methodology include the use of an iterative approach through two survey rounds and the incorporation of perspectives from varying stakeholder groups. Overall, compared with a traditional consensus panel approach, the Delphi survey method strengthens our confidence in the results by eliminating the potential bias from group dynamics and face-to-face responses.
The consensus on the core minimum data set represents a unified effort by multiple stakeholders. The list of identified data elements is relevant not only for the CRN focused on contraceptives but also for other researchers tasked with examining large databases, both prospectively and retrospectively with respect to clinical outcomes. The core minimum data elements will also improve data monitoring by supporting the quicker identification of clinical concerns and potentially avoiding poor patient outcomes. Additionally, widespread use of the core minimum data elements will support MDEpiNet in its endeavor to speed the development and advancement of CRNs as a robust source of real-world evidence for NEST. The next step of this project will be to collect these data elements through a national infrastructure. This will accrue unbiased and high-quality data on devices and technologies used for contraception in the context of a multipurpose CRN. Finally, the data collected by this CRN have the potential to address past and current strategic priorities of FDA, particularly of increasing the use of real-world evidence, engaging patient partners and promoting collaborative communities.25
Data availability statement
Data are available upon reasonable request. The data that support the findings of this study are available from the corresponding author, CEB, upon reasonable request.
Patient consent for publication
Twitter @courtneybaird22, @JialinMao, @Artsytwits
Contributors AS and DM-D initiated the collaborative project. All authors, except for BH, participated in the initial stakeholder meeting to conceptualize the project and design the study protocol. CEB designed the Delphi surveys, collated and analyzed the Delphi survey results, managed the Delphi process and drafted the first version of the manuscript. NL-B, MG and SC served as working group co-chairs, provided clinical/epidemiological insights on the Delphi survey results, and co-lead discussions with working group members. BH served as patient partner for the working group and provided feedback throughout the Delphi process. SG served as the informatics representative and drafted the informatics portion of the manuscript. All authors served as members of the working group, participated in discussions throughout the Delphi process, reviewed manuscript drafts, and participated in editing and revising the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. CEB is the guarantor.
Competing interests None declared.
Provenance and peer review Commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.