Article Text

Download PDFPDF

Determining value of Coordinated Registry Networks (CRNs): a case of transcatheter valve therapies
  1. Gregory Pappas1,
  2. Jesse Berlin2,
  3. Erika Avila-Tang1,
  4. John Carroll3,
  5. Joseph Drozda4,
  6. Douglas Dumont1,
  7. Thomas Gross1,
  8. Kathleen Hewitt5,
  9. Ajay Kirtane6,
  10. David Kong7,
  11. Mitchell Krucoff7,
  12. John Lashinger1,
  13. Nellie Lew1,
  14. Michael Mack8,
  15. Fred Masoudi3,
  16. Danica Marinac-Dabic1,
  17. Roxanna Mehran6,
  18. Sharon-Lise Normand9,
  19. Elizabeth Quin1,
  20. Fred Resnic10,
  21. Art Sedrakyan11,
  22. Ronald Waksman5,
  23. Larry Wood5,
  24. Changfu Wu1 and
  25. Tianay Ziegler12
  1. 1Food and Drug Administration, Department of Health and Human Services, Silver Spring, Maryland, USA
  2. 2Pharmacoepidemiology, Johnson and Johnson Services Inc, New Brunswick, New Jersey, USA
  3. 3Interventional Cardiology, University of Colorado Denver, Denver, Colorado, USA
  4. 4Mercy Health, Chesterfield, Missouri, USA
  5. 5American College of Cardiology, Washington, District of Columbia, USA
  6. 6Cardiovascular Research Foundation, Columbia University / New York-Presbyterian Hospital, New York City, New York, USA
  7. 7Duke Clinical Research Institute, Durham, North Carolina, USA
  8. 8Baylor Scott & White Health, Dallas, Texas, USA
  9. 9Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, USA
  10. 10Division of Cardiovascular Medicine, Lahey Hospital and Medical Center and Tufts University School of Medicine, Boston, Massachusetts, USA
  11. 11Healthcare Policy and Research, Weill Cornell Medical College, New York, New York, USA
  12. 12Princeton University, Princeton, New Jersey, USA
  1. Correspondence to Dr Gregory Pappas; gregory.pappas{at}fda.hhs.gov

Abstract

Background The Transcatheter Valve Therapy (TVT) Coordinated Registry Network (CRN) supported 23 regulatory decisions and ensured evidence-based evaluation of the application of TVT technology. However, there are cost concerns that require value assessment of the TVT CRN compared with traditional study designs.

Objectives We aimed to determine the value created by the TVT CRN based on (1) Return on investment (ROI), (2) Time saved (TS) in conducting necessary regulatory studies.

Methods For both ROI and TS analyses, we compared studies that used the TVT CRN with those that would have been required if the registry did not exist (counterfactual studies). To estimate ROI, we accounted for the costs of investment and gain from investment. Both the counterfactual costs and length of studies were projected using design specifications determined by US Food and Drug Administration (FDA) reviewers.

Results We identified 21 studies using the TVT CRN (supporting 23 FDA decisions) that generated evidence on TVT for three device manufacturers. ROI is estimated to be greater than 550%. TS by using the CRN ranged from months to years.

Conclusions The CRN method to evidence generation creates value for manufacturers and the broader device ecosystem, demonstrated with this example of the TVT CRN. The public health benefits of evidence created by this CRN outweighs the difference in data quality between traditional clinical studies and the CRN method.

  • TVT registry
  • Coordinated Registry Networks

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

View Full Text

Statistics from Altmetric.com

Introduction

Access to reliable and meaningful evidence about the safety, effectiveness, and quality of medical devices is essential to inform care and improve patient outcomes—a goal shared by stakeholders in the medical device ecosystem including patients, clinicians, health systems, payers, device manufacturers, and regulators. Traditional methods of evidence generation for device evaluation are very costly, time-consuming and have well-understood limitations.1 2

The rapidly developing digital health information infrastructure in the country is increasingly being harnessed to support more reliable, affordable, and timely evidence generation. The Coordinated Registry Networks (CRNs), as proposed by the National Registry Taskforce, strategically bring together real-world data from a variety of sources to support improved device evaluation.3

This paper describes the use of one CRN, the Transcatheter Valve Therapy CRN (TVT CRN), which has been successfully used for regulatory decision-making during the past several years. The TVT CRN links a cohort of persons who received TVT enrolled in the national Society for Thoracic Surgery/American College of Cardiology (STS/ACC) TVT Registry with Centers for Medicare and Medicaid Services (CMS) claims data (linked registry studies). The STS/ACC TVT Registry was created through the National Cardiovascular Data Registry (NCDR) via a partnership of the STS and the ACC, in close collaboration with the Food and Drug Administration (FDA), the CMS, and the Duke Clinical Research Institute.4 As of March 2019, the TVT CRN includes more than 600 sites that provide TVT, reporting to the STS/ACC TVT Registry, with over 195 000 individual patient data entries.5

The objective of this study was to demonstrate value created by CRNs in three areas: public health benefits through improved evidence, return on the investment (ROI) for conducting studies, and ‘time saved’ (TS) in conducting studies. The TVT CRN studies were conducted in lieu of traditional studies (a mixture of preapproval studies and postapproval studies (PASs)). We compared use of the TVT CRN (observed studies) to studies that would have been conducted had the TVT CRN not existed (counterfactual studies).

Methods

Data sources and approach

We identified and collected information on studies in the TVT CRN submitted to the US FDA between 1 December 2011 through 30 September 2017 and led to a regulatory decision.6 7 Regulatory decision was defined as approval of premarket approval application (PMA) supplements or postmarket requirements as conditions of approval. Table 1 lists the counterfactual studies that likely would have been conducted if the TVT CRN did not exist. Counterfactual studies included investigational device exemption (IDE) studies for expanded indication, extended postmarket follow-up of IDE studies (traditional methods), and traditional new enrollment PASs, those replaced by studies using the TVT CRN. Not all the studies were included in the model for ROI as some were combined for efficiency in fielding studies. In practice, studies are frequently combined when fielding; for calculation of ROI, therefore, we excluded 6 of the 22 potential counterfactual studies providing 16 studies (A through P listed in table 1) to be included in the ROI calculation.

Table 1

Costs and drivers of counterfactual studies: The TVT CRN*

Costs of counterfactual studies

We calculated an estimated cost of each counterfactual study using a model described by Wimmer et al.8 Cost drivers in the Wimmer et al. model include: study size, number of sites, need for recruitment, need for randomization, need for control groups, total number of sites outside USA (as a percentage), study duration (years), number of patient evaluations per year, patient evaluation type (phone/inperson), follow-up procedure required (yes/no), total procedure count, type of procedure (if yes), and organ system. We reduced the duration of the simplified base case to 1 year (Dr Fredrick Resnic personal communication). The premarket and postmarket review staff at FDA (expert opinion) provided the design features of counterfactual studies (what premarket and postmarket studies would have looked like if the TVT CRN had not been used) for calculation of the costs using the Wimmer et al. model.

ROI and TS

ROI. We defined the ROI as the cost of savings from investing in the CRN divided by the cost of investment in the CRN, multiplied by 100. The cost savings is the sum of costs of the counterfactual studies subtracted from the costs of observed studies conducted in the CRN.

TS. The time interval used for calculating TS is the number of days from enrollment of the first patient in the study to the date that follow-up is complete on the last study participant, the clinical data generation period in figure 1.

Figure 1

Time frame from study design to marketing application submission. IDE, investigational device exemption; IRB,Institutional Review Board

The focus on the clinical data generation period frames a metric that can be meaningfully compared across CRNs, or between CRNs and counterfactual studies, because it provides for standardization. Other intervals, which are variable between studies were not studied.

We calculated TS by using the CRN approach by subtracting the days estimated for each counterfactual study from the number of days required for clinical data generation in the observed studies. TS was calculated on each study that might have been conducted, irrespective of combinations of counterfactual studies used for calculation of ROI.

The number of days required for the observed study is the number of days specified in the study design. For counterfactual studies, the estimate of days required to complete a study is the sum of the days of follow-up specified by the design and the days required for enrollment. The counterfactual study enrollment time (CET) was calculated by applying the rate of enrollment of the pivotal study (a traditional premarket study) to the specifications of the counterfactual study (study size and specific length of study called for by the design) using the following formula:

CET = (number of patients for study by design / rate of enrollment) / study duration specified by design

The rate of enrollment in traditional studies was derived from the Summary of Study and Effectiveness Data description of study design. Specifically, two TVT premarket studies were compared which determine the patients per site per year (both studies enrolled five patients/site/year). This estimate was used for the calculation of counterfactual enrollment times.

Results

The TVT CRN supported 21 studies, which led to 23 decisions by the FDA; data for two submissions were derived from a common study. The TVT CRN was used to support four premarket decisions, including expanded indications, and 19 postmarket surveillance studies in lieu of traditional PASs (table 1). The cost to the three medical device companies for these studies conducted in the TVT CRN is a total of $25.05 million paid to NCDR. This expense of $25.05 million is the cost of the observed the TVT CRN-based regulatory studies.

The costs and some of the study design features for each of the 16 counterfactual studies included in the ROI calculation are provided in table 1. These counterfactual (traditional) study designs included a planned enrollment ranging from 100 to 2000 patients, recruited from 27 to 65 participating US sites, who were to be followed for a median of 5 years. The estimated costs of the counterfactual studies ranged from $2.16 million to $18.93 million with a median estimated cost of $8 million (IQR: $2.59–$12.55 million) and a total projected cost of $164 million.

Return on investment

The cost savings associated with performance of the observed studies using the TVT CRN, rather than the counterfactual (traditional) study design, is estimated to be $138 958 196 ($164 008 196 for the counterfactual studies less the $25.05 million spent by the medical device manufacturers to support the TVT CRN studies performed). Thus, the ROI for the TVT CRN studies is 555% (see figure 2).

Figure 2

Calculating return on investment (ROI) for Transcatheter Valve Therapy (TVT) Coordinated Registry Network (CRN) studies as compared with the counterfactual (traditional) study design.

A sensitivity analysis was performed to demonstrate the robustness of the model used to calculate ROI that increased and decreased (by 5% and 10% in each direction) the numbers of study subjects and the numbers of study sites specified by the counterfactual studies (table 2). Varying the number of subjects creates a range of ROI from 603% to 507%. Note that the original calculations were based on assumptions about meaningful effect sizes, so that reducing sample size beyond a certain point would clearly lead to underpowered studies, which would not typically be accepted by the FDA. Varying the number of sites creates a range of ROIs from 559% to 551%. The table also includes ROI estimations that vary the number of counterfactual studies included in the estimate. A high estimate (682%) includes all possible counterfactual studies. The low estimate (368%) assumes studies would have been conducted (combining individual studies that would have been needed, based on field efficiencies). The lower estimate assumes combining a higher number of studies than is probable.

Table 2

Return on investment (ROI) sensitivity analysis: variation of number of subjects, sites, and counterfactual studies used to calculate ROI

Time saved

Table 3 presents the time (days and years) required to complete observed studies conducted with the TVT CRN as compared with the days required to conduct the counterfactual cases. TS for studies varied between 0.5 years and 12.6 years (median=2.6, IQR: 1.0 years to 4.4 years). The PMA studies for labelling expansion (M, N, and U) saved 1.0 year, 1.0 year, and 0.7 years, respectively. The slow rate of enrollment in traditional (counterfactual) studies compared with the TVT CRN is the major factor driving these calculations.

Table 3

Counterfactual study* times, real scenario study times, and days (years) saved

Discussion

The ROI and TS by using a CRN in place of traditional study designs for evidence generation for TVT was estimated to be 555% and was robust to changes in assumptions. The lowest estimate of ROI (368%) came from reducing the number of studies that were assumed to be required to address the relevant regulatory questions. As noted in more detail below, the ROI calculated for the TVT CRN cannot be used to generalize value created by other CRNs. While the methods used here are presented as robust, comparisons of case studies should be conducted to understand the broader utility of the CRN approach to evidence generation. There will likely be variability in the degree to which CRNs are cost-saving.

Underestimating ROI

The model and assumptions used to calculate ROI may be an underestimate for the following reasons:

  • The Wimmer et al. model was created to estimate the cost of PASs. It is broadly understood that premarket studies are much more expensive than PASs, due to several study specifications that are more stringent for premarket studies, thus suggesting the ROI is an underestimate.9

  • Dropping some of the counterfactual cases from the ROI calculation is justified by the reality that studies may be combined when fielding studies. The site enrollment costs decrease when studies are combined. For ease in calculation, the dropped studies from the ROI calculation include dropping both the cost of site enrollment and the cost of the required subjects. The cost of subjects would be borne in combined traditional studies but is not accounted for in the calculation, thus producing an underestimate of the ROI.

  • The national coverage decision by CMS to pay for TVT was also based on findings from the TVT CRN. The benefit to companies was not included in the calculation in this paper and further contributes to underestimation of the ROI.

Other considerations

The sources of value created by the CRN are derived by leveraging the substantial investment made by hospitals in the registries and by the linkage of outcome data between the registry and CMS (Medicare claims). Outcome data over time are very expensive to obtain using traditional methods and were provided at no cost to the CRN through the linkage of CMS claims records.

This study did not evaluate value created by registry for hospitals; those investments are made for reasons including: quality improvement, comparative effectiveness research, and provision of evidence to negotiate with payers. A study by Ahmad et al. documents ROI for registries at Dartmouth hospital.10 The ROI for hospitals and clinical centers maintaining registries will increase as data collection and curation in clinical settings become more efficient (less costly). Automation of registry work (structured data capture and natural language processing, approaches) is being actively pursued by many researchers.11

STS/ACC TVT Registry data quality is enhanced and promoted through NCDR’s use of aggressive registry site education, data quality reports, data audit strategies along with adjudication of key adverse outcomes. Data in the STS/ACC TVT Registry have been validated.12 The CMS claims data used in the TVT CRN for evaluating longitudinal outcomes have also been validated.13–16 The CRN approach and traditional studies are not considered to be identical or equivalent. The quality, variety, and richness of data collection from traditional studies are not available through the CRN, though, on balance, the judgment of FDA review staff is that the overall improvement in the strength of evidence derived from the TVT CRN outweighs the limitations of the data.

Study sizes that support an understanding of heterogeneity of patients and operating context must be large enough to capture heterogeneity (e.g., across sites) and support analysis to evaluate device benefit and risk. That said, the mechanism for case selection into a registry is crucial. The important aspect is that selection into the registry should ensure that the included patients are representative of the underlying population of interest, and is not, for example, determined by including only those patients whose prognosis is more favorable than that of the underlying population. Entering consecutive patients into the registry, in an unselected manner, from a broad range of sites, is often a preferred way to achieve this representativeness. In the case of the TVT CRN, nearly all cases are included in the registry.

The large number of sites in the TVT CRN saved substantial time in enrollment compared with the traditional studies. The specified length of each counterfactual study is determined by design, but subject enrollment times are variable, which is the source of ‘time saved’ in these calculations.

The TS measured here may be an underestimate for two reasons. First, the metric does not include a measure of time to enrol sites, as those data are not available for this study. Clinical trialists understand that the time needed to enlist sites varies greatly among studies. Documentation from a study of enrollment time in two PASs of a drug-eluting stent (one using registries compared with a second using traditional methods) shows similar dramatic time savings.17 The time required for initial enrollment of sites into the registry must be considered. However, if the registry is already in existence, then subsequent use of the operating registry saves time. If the registry must be built from scratch, then the time saving increases after initial establishment of the registry.

The second reason we believe the TS is an underestimate is that, in the case of PASs, enrollment rates would be expected to be lower than those for PMA studies. It is more difficult to enrol a patient into a study when the device is already commercially available. In the calculations presented, we assume enrollment rates of PMA studies for all the counterfactual studies.

The value (in dollars) derived from TS differs for premarket and postmarket studies. For label expansion studies, the TS contributes to additional ROI, as faster time to market may translate into increased revenue. The revenue generated by faster time to market is different for each device because of the size of the market that becomes available with label expansion, the degree of market penetration, the margin on the specific device, and the reimbursement status. Business decisions also determine how and when this evidence is used. For these reasons, TS in the label expansion studies do not automatically translate into additional ROI for companies. TS in a PAS should be considered as an efficiency but does not directly contribute to ROI.

The label expansions were not planned as part of the original effort to use the TVT CRN for PASs, raising some issues for consideration. Importantly, collecting patient data through a national registry such as the STS/ACC TVT Registryof device off-label use represents an important advance in the evaluation of safety and effectiveness of patient populations previously not consistently or thoroughly assessed. Caution is needed, though, to avoid encouraging inappropriate off-label use, solely for the sake of data generation, as this may raise ethical concerns as well as constitute an investigation subject to regulation.

The meaning of the TS metric may be appreciated by comparing the time saving of the CRN approach to efforts to save time in FDA reviews. Major efforts in monitoring and decreasing review times are well documented.18 Increasing the time efficiency of the review process has resulted in days saved.

The dollar amounts for savings for the TVT CRN case study cannot be directly extrapolated to other device areas. Absolute costs of studies vary widely by device. The ROI as a relative rate comparing registry versus clinical studies provides a more useful comparison.

Public health benefit

The public health benefits of using CRNs stem from robust generalizable study populations and sites, more timely results, and a move towards real-time evidence device generation. The large study sizes made possible by the TVT CRN provide more representative data of a full range of patients and clinical settings, compared with traditional studies. Additionally, data coming from the large number of sites provide an understanding of learning curve effects, diversity in operator effects including volume, and variations in outcome created by differences among implant techniques, concomitant therapy, or hospital settings.

On-going data colleciton (both new patients and outcomes) in the TVT CRN creates near real-time surveillance of the population exposed to the device.19 This active surveillance system can detect problems with devices sooner than traditional approaches currently used (passive surveillance). This improved comprehensive surveillance, the faster detection of device problems, is a public health benefit to patients, clinicians, and industry.

There are also clear benefits to patients who received TVT as a life-saving procedure based on the label expansions made possible with the TVT CRN. To quantify this public health benefit, an analysis can be envisioned that projects the number of person-years of lives extended (e.g., Disability Adjusted Life Years DALYs or Quality Adjusted Life Years QALYs) due to the new device in the new indication.

Acknowledgments

The authors thank the following people who helped produce this manuscript: Daniel Canos CMS, Marta Steliac FDA, Paul Coplan JnJ, and the MDEpiNet Workgroup for Study of Value Created by CRN.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
View Abstract

Footnotes

  • Contributors JB, employed by Johnson and Johnson, John D. JC, member of DSMB/CEC committee for the Tendyne transcatheter mitral valve replacement early feasibility trial (Abbott). AK, Institutional funding to Columbia University and/or Cardiovascular Research Foundation from Medtronic, Boston Scientific, Abbott Vascular, Abiomed, CSI, CathWorks, Siemens, Philips, ReCor Medical, Personal: Conference honoraria and Travel/Meal reimbursements only. DK, institution receives research support from Medtronic and St Jude Medical (now a division of Abbott). MM, Edwards Lifesciences- Co PI Partner 3 Trial, Abbott Vascular- Co PI COAPT Trial, Medtronic- Study Chair Apollo Trial, All uncompensated. FM, contract with the American College of Cardiology (through the University of Colorado) as the Chief Science Officer of the NCDR. RM, consultant to Abbott, and a grant from Medtronic. AS, Cooperative Agreement with MDEpiNet which provided the organizational support for this project. RW, Advisory Board: Abbott Vascular, Amgen, Boston Scientific, Medtronic, Philips Volcano, Pi-Cardia Ltd., Cardioset; Consultant: Abbott Vascular, Amgen, Biosensors, Biotronik, Boston Scientific, Medtronic, Philips Volcano, Pi-Cardia Ltd., Cardioset; Grant Support: Abbott Vascular, AstraZeneca, Biosensors, Biotronik, Boston Scientific, Chiesi; Speakers Bureau: AstraZeneca, Chiesi; Investor: MedAlliance. LW, employed by Edwards Life Science.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.