The judgement by Chalmers, the progenitor of the Cochrane Collaboration, that under-reporting of research is scientific misconduct1 may appear harsh, but there are strong ethical arguments why clinical studies should be reported fully—so that unnecessary duplication of research with risks can be avoided and that insecure results can be replicated. Critical readers should be able to understand the data and assess the validity of any conclusions. The Declaration of Helsinki reminds us (at Article 36) that ‘Researchers … are accountable for the completeness and accuracy of their reports’.2
Requirements for reporting secondary research may be less strong but the methods of any study should still be described in sufficient detail to allow others to reproduce it. That has been reinforced by simulation-specific extensions to the Strengthening the Reporting of Observational Studies in Epidemiology recommendations for reporting observational research3 and by the Consolidated Health Economic Evaluation Reporting Standards guidance on reporting economic evaluations in healthcare.4 Aspects of both statements are relevant to the paper by Cronenwett et al5 in this issue of BMJ Surgery, Interventions, & Health Technologies.
The authors present the costs of using the Vascular Quality Initiative (VQI), a registry maintained by the Society for Vascular Surgery in the USA, to undertake investigations of medical devices that were mandated by the US Food and Drug Administration (FDA). They compare them against the costs which they estimated would have been incurred if the manufacturers had instead organised their own studies. Actual and predicted costs were evaluated for six premarket investigations or postmarket surveillance studies that monitored high-risk devices used either for endovascular aortic repair (EVAR) or for peripheral angioplasty. The manuscript does not name the particular devices or disclose the requirements specified by the FDA, in part because of sensitivities of the manufacturers, nor does it break down the particular costs that were incurred by the VQI registry. The hypothetical (‘counterfactual’) costs were estimated using a published model6 but only the resulting total sums are reported. These omissions, which were self-imposed by the investigators, mean that readers are asked to take the results on trust. That limits the generalisability of their findings.
Frustrations about incomplete disclosure of data concerning postmarket surveillance of high-risk medical devices extend to the European Union (EU) where achieving transparency is hampered by the delegation of regulatory assessments to notified bodies, which as independent commercial organisations are not bound by EU legislation on freedom of access to information. The new EU Medical Device Regulation that will be implemented from 2021 continues to protect the privacy of information submitted by manufacturers.7 It is ironic that medical audit databases now place the clinical outcomes of individual surgeons in the public domain while data about the individual devices that they have used or implanted may be inaccessible. Until September 2016, specific device identities were not recorded by the VQI8 but now there should be no reticence about open disclosure and comparisons between competing devices or about the costs of maintaining registries.
An important issue in the use of registry data for regulatory purposes is its completeness. The International Medical Device Regulators Forum has recommended that a medical device registry should have a case ascertainment of at least 95%, if it is to be accepted as providing high-quality data that are sufficiently robust to inform regulatory decisions.9 For registries that can be used to evaluate medicines, 100% inclusion rates are advised.10 In the six reported studies, all eligible patients in the participating hospitals were included and data capture was complete, but centres were selected and the studies enrolled an average of 169 patients5 which is too few to assure safety. Earlier reports from the VQI registry stated that it included 82% of eligible patients who had had EVAR11 and that it captured a different profile of patients with peripheral arterial disease from alternative databases.12 Comprehensive recruitment might be attained by stipulating that entering data in a registry is a condition for institutional reimbursement.
The largest component of the actual costs of the registry studies was the charge per patient to manufacturers for access to the VQI data.5 The investigators have stated that this charge to industry is market driven, based on value and not relatable to specific costs. It includes payments to participating centres for entering additional data, and to the subcontractor responsible for monitoring data. The counterfactual estimates include overhead costs and per patient payments to centres. The net costs are given as two percentages, using alternate denominators—the difference between actual and estimated cost, over the estimated cost, as the ‘cost-saving percentage’; and the estimated over the actual cost, as the ‘return on investment’. Both figures are persuasive but their details are specific to the context of conducting a study in the USA. The focus was about saving costs and time, but what matters more is which method is more clinically effective—giving better quality, more comprehensive and more reliable data.13
There are increasing pressures on regulators worldwide to develop systems that can give accelerated access for patients and physicians to new high-risk medical devices. Such schemes increase the risks of unexpected complications, however, so they can be considered only if they are balanced by more postmarket evidence generation. The risks that are inherent in conditional approvals (or coverage with evidence development) should be shared by patients from all regulatory jurisdictions where a device will be available. Their data should be pooled through international collaborations of networks of registries, to enable the earliest possible detection of any safety signals.14
The benefits that can accrue from comprehensive registries are best exemplified by Sweden, which has 120 National Quality Registries integrated within its healthcare system. Data are collected routinely through the electronic health record and linked with other national databases. Funding is provided by the Swedish Association of Local Authorities and Regions. The Swedeheart registries on high-risk cardiovascular devices have been used as the platform to develop randomised registry trials.15 These may provide more valuable data to regulators more cost efficiently than reported by the VQI investigators, with capture of real-world experience and ready access to control populations.
The conclusion of Cronenwett et al—that it is more cost-effective to collect postmarket data about a high-risk medical device by using an existing registry run by a professional association than by initiating an industry-led study—is a welcome confirmation of the wisdom of current trends, at least if results are fully and promptly reported. Their study reinforces the need for manufacturers of all high-risk devices now to collaborate with academic bodies to support their registries and ensure their long-term sustainability.
