Determining value of Coordinated Registry Networks (CRNs): a case of transcatheter valve therapies

Data sources and approach

We identified and collected information on studies in the TVT CRN submitted to the US FDA between 1 December 2011 through 30 September 2017 and led to a regulatory decision.6 7 Regulatory decision was defined as approval of premarket approval application (PMA) supplements or postmarket requirements as conditions of approval. Table 1 lists the counterfactual studies that likely would have been conducted if the TVT CRN did not exist. Counterfactual studies included investigational device exemption (IDE) studies for expanded indication, extended postmarket follow-up of IDE studies (traditional methods), and traditional new enrollment PASs, those replaced by studies using the TVT CRN. Not all the studies were included in the model for ROI as some were combined for efficiency in fielding studies. In practice, studies are frequently combined when fielding; for calculation of ROI, therefore, we excluded 6 of the 22 potential counterfactual studies providing 16 studies (A through P listed in table 1) to be included in the ROI calculation.

Costs of counterfactual studies

We calculated an estimated cost of each counterfactual study using a model described by Wimmer et al.8 Cost drivers in the Wimmer et al. model include: study size, number of sites, need for recruitment, need for randomization, need for control groups, total number of sites outside USA (as a percentage), study duration (years), number of patient evaluations per year, patient evaluation type (phone/inperson), follow-up procedure required (yes/no), total procedure count, type of procedure (if yes), and organ system. We reduced the duration of the simplified base case to 1 year (Dr Fredrick Resnic personal communication). The premarket and postmarket review staff at FDA (expert opinion) provided the design features of counterfactual studies (what premarket and postmarket studies would have looked like if the TVT CRN had not been used) for calculation of the costs using the Wimmer et al. model.

ROI and TS

ROI. We defined the ROI as the cost of savings from investing in the CRN divided by the cost of investment in the CRN, multiplied by 100. The cost savings is the sum of costs of the counterfactual studies subtracted from the costs of observed studies conducted in the CRN.

TS. The time interval used for calculating TS is the number of days from enrollment of the first patient in the study to the date that follow-up is complete on the last study participant, the clinical data generation period in figure 1.

Request permission

The focus on the clinical data generation period frames a metric that can be meaningfully compared across CRNs, or between CRNs and counterfactual studies, because it provides for standardization. Other intervals, which are variable between studies were not studied.

We calculated TS by using the CRN approach by subtracting the days estimated for each counterfactual study from the number of days required for clinical data generation in the observed studies. TS was calculated on each study that might have been conducted, irrespective of combinations of counterfactual studies used for calculation of ROI.

The number of days required for the observed study is the number of days specified in the study design. For counterfactual studies, the estimate of days required to complete a study is the sum of the days of follow-up specified by the design and the days required for enrollment. The counterfactual study enrollment time (CET) was calculated by applying the rate of enrollment of the pivotal study (a traditional premarket study) to the specifications of the counterfactual study (study size and specific length of study called for by the design) using the following formula:

CET = (number of patients for study by design / rate of enrollment) / study duration specified by design

The rate of enrollment in traditional studies was derived from the Summary of Study and Effectiveness Data description of study design. Specifically, two TVT premarket studies were compared which determine the patients per site per year (both studies enrolled five patients/site/year). This estimate was used for the calculation of counterfactual enrollment times.

Return on investment

The cost savings associated with performance of the observed studies using the TVT CRN, rather than the counterfactual (traditional) study design, is estimated to be $138 958 196 ($164 008 196 for the counterfactual studies less the $25.05 million spent by the medical device manufacturers to support the TVT CRN studies performed). Thus, the ROI for the TVT CRN studies is 555% (see figure 2).

Request permission

A sensitivity analysis was performed to demonstrate the robustness of the model used to calculate ROI that increased and decreased (by 5% and 10% in each direction) the numbers of study subjects and the numbers of study sites specified by the counterfactual studies (table 2). Varying the number of subjects creates a range of ROI from 603% to 507%. Note that the original calculations were based on assumptions about meaningful effect sizes, so that reducing sample size beyond a certain point would clearly lead to underpowered studies, which would not typically be accepted by the FDA. Varying the number of sites creates a range of ROIs from 559% to 551%. The table also includes ROI estimations that vary the number of counterfactual studies included in the estimate. A high estimate (682%) includes all possible counterfactual studies. The low estimate (368%) assumes studies would have been conducted (combining individual studies that would have been needed, based on field efficiencies). The lower estimate assumes combining a higher number of studies than is probable.

Time saved

Table 3 presents the time (days and years) required to complete observed studies conducted with the TVT CRN as compared with the days required to conduct the counterfactual cases. TS for studies varied between 0.5 years and 12.6 years (median=2.6, IQR: 1.0 years to 4.4 years). The PMA studies for labelling expansion (M, N, and U) saved 1.0 year, 1.0 year, and 0.7 years, respectively. The slow rate of enrollment in traditional (counterfactual) studies compared with the TVT CRN is the major factor driving these calculations.

Underestimating ROI

The model and assumptions used to calculate ROI may be an underestimate for the following reasons:

• The Wimmer et al. model was created to estimate the cost of PASs. It is broadly understood that premarket studies are much more expensive than PASs, due to several study specifications that are more stringent for premarket studies, thus suggesting the ROI is an underestimate.9

• Dropping some of the counterfactual cases from the ROI calculation is justified by the reality that studies may be combined when fielding studies. The site enrollment costs decrease when studies are combined. For ease in calculation, the dropped studies from the ROI calculation include dropping both the cost of site enrollment and the cost of the required subjects. The cost of subjects would be borne in combined traditional studies but is not accounted for in the calculation, thus producing an underestimate of the ROI.

• The national coverage decision by CMS to pay for TVT was also based on findings from the TVT CRN. The benefit to companies was not included in the calculation in this paper and further contributes to underestimation of the ROI.

Other considerations

The sources of value created by the CRN are derived by leveraging the substantial investment made by hospitals in the registries and by the linkage of outcome data between the registry and CMS (Medicare claims). Outcome data over time are very expensive to obtain using traditional methods and were provided at no cost to the CRN through the linkage of CMS claims records.

This study did not evaluate value created by registry for hospitals; those investments are made for reasons including: quality improvement, comparative effectiveness research, and provision of evidence to negotiate with payers. A study by Ahmad et al. documents ROI for registries at Dartmouth hospital.10 The ROI for hospitals and clinical centers maintaining registries will increase as data collection and curation in clinical settings become more efficient (less costly). Automation of registry work (structured data capture and natural language processing, approaches) is being actively pursued by many researchers.11

STS/ACC TVT Registry data quality is enhanced and promoted through NCDR’s use of aggressive registry site education, data quality reports, data audit strategies along with adjudication of key adverse outcomes. Data in the STS/ACC TVT Registry have been validated.12 The CMS claims data used in the TVT CRN for evaluating longitudinal outcomes have also been validated.13–16 The CRN approach and traditional studies are not considered to be identical or equivalent. The quality, variety, and richness of data collection from traditional studies are not available through the CRN, though, on balance, the judgment of FDA review staff is that the overall improvement in the strength of evidence derived from the TVT CRN outweighs the limitations of the data.

Study sizes that support an understanding of heterogeneity of patients and operating context must be large enough to capture heterogeneity (e.g., across sites) and support analysis to evaluate device benefit and risk. That said, the mechanism for case selection into a registry is crucial. The important aspect is that selection into the registry should ensure that the included patients are representative of the underlying population of interest, and is not, for example, determined by including only those patients whose prognosis is more favorable than that of the underlying population. Entering consecutive patients into the registry, in an unselected manner, from a broad range of sites, is often a preferred way to achieve this representativeness. In the case of the TVT CRN, nearly all cases are included in the registry.

The large number of sites in the TVT CRN saved substantial time in enrollment compared with the traditional studies. The specified length of each counterfactual study is determined by design, but subject enrollment times are variable, which is the source of ‘time saved’ in these calculations.

The TS measured here may be an underestimate for two reasons. First, the metric does not include a measure of time to enrol sites, as those data are not available for this study. Clinical trialists understand that the time needed to enlist sites varies greatly among studies. Documentation from a study of enrollment time in two PASs of a drug-eluting stent (one using registries compared with a second using traditional methods) shows similar dramatic time savings.17 The time required for initial enrollment of sites into the registry must be considered. However, if the registry is already in existence, then subsequent use of the operating registry saves time. If the registry must be built from scratch, then the time saving increases after initial establishment of the registry.

The second reason we believe the TS is an underestimate is that, in the case of PASs, enrollment rates would be expected to be lower than those for PMA studies. It is more difficult to enrol a patient into a study when the device is already commercially available. In the calculations presented, we assume enrollment rates of PMA studies for all the counterfactual studies.

The value (in dollars) derived from TS differs for premarket and postmarket studies. For label expansion studies, the TS contributes to additional ROI, as faster time to market may translate into increased revenue. The revenue generated by faster time to market is different for each device because of the size of the market that becomes available with label expansion, the degree of market penetration, the margin on the specific device, and the reimbursement status. Business decisions also determine how and when this evidence is used. For these reasons, TS in the label expansion studies do not automatically translate into additional ROI for companies. TS in a PAS should be considered as an efficiency but does not directly contribute to ROI.

The label expansions were not planned as part of the original effort to use the TVT CRN for PASs, raising some issues for consideration. Importantly, collecting patient data through a national registry such as the STS/ACC TVT Registryof device off-label use represents an important advance in the evaluation of safety and effectiveness of patient populations previously not consistently or thoroughly assessed. Caution is needed, though, to avoid encouraging inappropriate off-label use, solely for the sake of data generation, as this may raise ethical concerns as well as constitute an investigation subject to regulation.

The meaning of the TS metric may be appreciated by comparing the time saving of the CRN approach to efforts to save time in FDA reviews. Major efforts in monitoring and decreasing review times are well documented.18 Increasing the time efficiency of the review process has resulted in days saved.

The dollar amounts for savings for the TVT CRN case study cannot be directly extrapolated to other device areas. Absolute costs of studies vary widely by device. The ROI as a relative rate comparing registry versus clinical studies provides a more useful comparison.

Public health benefit

The public health benefits of using CRNs stem from robust generalizable study populations and sites, more timely results, and a move towards real-time evidence device generation. The large study sizes made possible by the TVT CRN provide more representative data of a full range of patients and clinical settings, compared with traditional studies. Additionally, data coming from the large number of sites provide an understanding of learning curve effects, diversity in operator effects including volume, and variations in outcome created by differences among implant techniques, concomitant therapy, or hospital settings.

On-going data colleciton (both new patients and outcomes) in the TVT CRN creates near real-time surveillance of the population exposed to the device.19 This active surveillance system can detect problems with devices sooner than traditional approaches currently used (passive surveillance). This improved comprehensive surveillance, the faster detection of device problems, is a public health benefit to patients, clinicians, and industry.

There are also clear benefits to patients who received TVT as a life-saving procedure based on the label expansions made possible with the TVT CRN. To quantify this public health benefit, an analysis can be envisioned that projects the number of person-years of lives extended (e.g., Disability Adjusted Life Years DALYs or Quality Adjusted Life Years QALYs) due to the new device in the new indication.