Safe implementation of surgical innovation: a prospective registry of the Versius Robotic Surgical System

Introduction

There are inherent risks in the implementation of innovative surgical procedures. These include unexpected complications, longer operation times, the risks associated with the ‘learning curve’ for new techniques and the occasional requirement for a more traditional technique with proven ability when unexpected factors necessitate conversion. A possible approach to mitigating such risks is the prospective establishment of a data registry so that deviations from expected surgical outcomes can be identified, quantified, measured and corrected at the earliest opportunity, with a view to reducing or eliminating any additional risks to patients. Access to reliable and meaningful evidence about the safety, effectiveness and quality of medical devices is essential to inform care and improve patient outcomes.1 Alternative study designs to randomised controlled trials (RCTs) can provide evidence for beneficial effects of medical devices.2 There is consensus that long-term registries are more effective at detecting and quantifying adverse events (AEs) than RCTs.3 Device registries, through aggregation of real-word evidence, can provide ongoing device safety surveillance and additional evidence for effectiveness,4 and their use is encouraged as part of the Idea, Development, Exploration, Assessment, Long-term study-Devices (IDEAL-D) recommendations for established procedures.5 6 We describe the development of such a registry to accompany the implementation of the Versius Robotic Surgical System (RSS; Cambridge Medical Robotics Surgical, Cambridge, UK) into clinical use.

The RSS is a next-generation surgical system developed for use in robot-assisted minimal access surgery (MAS). The device underwent an iterative development process using feedback from surgeons and surgical teams to improve both end-user experience and surgical outcomes.7 Several innovative features have been designed to address some of the limitations of conventional minimal access instruments and barriers to the uptake of robot-assisted MAS, with a view to improving system manoeuvrability, surgical access, visualisation and ergonomics.8–10 The RSS also captures telemetry data that may provide insights into surgical performance and allow for refinements in surgical techniques.11 These data include console start/stop times, hand-controller movement patterns, robotic arm locking/release times, alarms and the number and type of instruments used. Additionally, surgeons have the option to save the endoscope video recording of the procedure for their review.

In broad alignment with the IDEAL-D framework,5 6 evidence has been reported at all stages of the device’s development. Previous studies have validated the usability of the device by trained intended users.12 After rigorous preclinical testing with successful completion of a range of gynecological, urological, renal and general surgical procedures in both cadavers and live porcine studies,13–16 the device has been successfully and safely used clinically in live-human gynecological, general and colorectal surgical procedures.17–21

The Versius Surgical Registry is a prospective, multicenter, international, observational registry with ongoing cumulative enrollment across surgical specialties. The aim of the registry is to demonstrate that the device can be safely implemented through prospective cohort studies embedded within a prospective clinical registry. The registry will also facilitate continual monitoring of key performance indicators, such as operative time and rate of conversion to a different operating modality, with the aim of improving patient safety through early intervention where required. Boundaries determined through analysis of registry data will help proactively identify any signs of device issues where additional maintenance may be required, and instances where targeted training may be needed to support individual surgeons to prevent potential AEs. For example, additional practice on the Versius Trainer, a purpose-designed simulated training platform,22 may be beneficial. As the body of data contained within the registry increases, the boundaries for the various surgical outcomes recorded will become more established, with well-defined parameters for acceptable performance. Furthermore, systematic collection of surgeon performance data will provide a clear picture of the learning curve for surgeons who are new to this RSS, in line with IDEAL-D recommendations to evaluate learning curves.5 Analysis of skill acquisition and surgical proficiency by number of cases performed using the device will help inform future refinement and expansion of the specially designed and validated training program.22 23

Methods

Demonstrative data reported here (figures 1–3; online supplemental figures 1–5) are from a clinical cohort study embedded within the registry.20

Request permission

Request permission

Request permission

Patient characteristics and surgical peri-operative data and outcomes up to 90 days are systematically collected in the registry via a secure online platform. Data may be entered by the lead surgeon, surgical assistants, nurses or administrative assistants; all new users receive formal online training involving demonstration of complete data entry and navigation of the platform in teleconference calls. Data are ideally entered into the platform immediately following the surgery and on the relevant post-operative days, but can be added retrospectively, if necessary (table 1).

Pre-operative data include date of consent, a patient’s unique identification (ID) number and planned procedure(s), in addition to patient demographics, surgical history and diagnoses data including age, sex, height and weight, body mass index and American Society of Anesthesiologists (ASA) status.5 Patient data are pseudonymized automatically within the platform, with each patient assigned a unique ID number. This number is linked to their medical record number for future reference (eg, if undergoing further surgery), but the medical record number is available only to the hospital in line with General Data Protection Regulation requirements.

Peri-operative outcomes, recorded from the start of surgery to discharge from hospital, include: operative time (skin incision to skin closure), estimated intra-operative blood loss (categorised) and use of blood products, intra-operative complications,24 conversion to an alternative technique with the reason for conversion, use of additional laparoscopic instruments, return to the operating room prior to discharge with reason for return and length of hospital stay (date and time of surgery to date and time of discharge). Post-operative outcomes include complications within 90 days of surgery reported using the Clavien-Dindo classification,24 readmission to hospital within 30 days and 90-day mortality (directly reported or as serious AE classification). Data pertaining to complications include start and end dates of AEs, full details of the complication with diagnosis, and when possible, AE severity, seriousness and relatedness to the device, in addition to the treatment/intervention approach taken and the eventual outcome (such as resolution within a recorded number of days). Relatedness of complications to the device will be determined for any complication that has a reasonably suspected causal relationship resulting from insufficiencies or inadequacies in the instruction for use, deployment of the device or user error. The platform incorporates an option for the surgeons to note if they believe an AE may have been related to the device, and these events, together with serious AEs, are reviewed on a monthly basis by an independent Clinical Events Committee.

Data collected in the registry are ‘cleaned’ and validated on a monthly basis, partly through manual screening processes to identify inconsistencies and apparent human errors and, where possible, to clarify or correct information with the surgeon who originally entered the data. There are plans for real-time ‘on-demand’ data cleaning and validation and automatic generation of certain outputs in the future.

Surgeon reports including summary statistics and performance charts are produced quarterly and shared with individual surgeons directly via email. Reports are provided to institutions on a quarterly or yearly basis and are customized based on discussions with the institution; metrics can be reported at the hospital level, such that individual surgeon performance is not shown, if preferred.

Results

The data collected in the registry may be analyzed as comparative performance metrics, by meta-analyses or by individual surgeon performance using control method analysis, and data are split across binary and continuous outcomes (online supplemental table 1).

To demonstrate interrogation of a continuous data metric, we present operative time for robot-assisted cholecystectomy procedures collected in the registry up to March 2021.

Comparative performance may be illustrated as a funnel plot showing mean operative time for individual surgeons or institutions by total number of operations performed, with the total population mean and CIs also plotted for comparison (figure 1). Mean operative time falling outside the 99% CI (significantly shorter or longer than expected), may be investigated further to understand the cause and if any intervention may be needed. Similar analyses can be performed to compare individuals’ performance metrics against or within more specific subpopulations, such as those in a selected setting, geographical area or other criteria.

Meta-analyses of the same metric may be conducted to provide a different perspective, treating each hospital or surgeon as a separate ‘study’. As the registry data do not contain conventional control groups, a meta-analysis uses the means and SDs of each study (calculated across all surgeons reporting a single mean with inverse variance weighting for pooling), and the distribution of a metric can be compared against the total population (to one or two points, with fixed-effects model and random-effects model for differential weighting (random-effects model between-study heterogeneity, thus allowing for a greater degree of uncertainty in the estimate)). This analysis method is likely to be appropriate once a large amount of data has been collected for particular subgroups, such as the procedure type performed, before comparisons can be made. For example, an analysis may only include data from surgeons who have performed more than a specified number of cases, given that metrics are likely to be affected by their learning curve with effects in unpredicted directions. A larger and wider pool of data may allow subgrouping in the future based on, for example, geographical location and type of institution, which will help to identify risk factors and population attributes that may affect surgical outcomes.

Online supplemental figure 1 presents a meta-analysis of all operative time data for six surgeons who had performed at least 10 cholecystectomies, with weighting according to the number of operations performed. Outliers may be identified based entirely on data from the registry, where CIs indicate significant departure from the group mean and, especially if a surgeon’s data are assigned higher weighting, there may be a need for further investigation to determine how the outcome may be improved. In this case, the mean operative time for surgeon 7 appears to be shorter than the overall mean for the six surgeons included in the meta-analysis. Matching surgeon 7’s cases to the associated surgical outcomes data can then determine whether or not intervention may be required; this observation could indicate that the surgeon was simply more efficient, or there could be scope to take more time for particular surgical steps to reduce patient risk (eg, if there was a higher-than-average rate of conversion, or number of AEs).

The registry data also allow for analysis at the level of the individual surgeon in the form of control method analysis using cumulative sum (CUSUM) charts, as a method of tracking performance against calculated warning and control limits.25 Figure 2 presents a demonstrative standardised CUSUM for an individual surgeon’s operative time by consecutive procedures performed. Here, the CUSUM is centred at 0.00 based on the population mean (target value) and SD; however, it is also possible to standardise the CUSUM based on the individual’s mean (and SD) operative time during an ‘in-control’ period (eg, for the 10 procedures after the surgeon’s first 20). The appropriate standardisation approach will depend on the procedure type; there may be several different surgical approaches for a given indication, and operative times are likely to vary based on the surgeon’s preferred approach.

There are two main lines plotted, a positive (upper) CUSUM and a negative (lower) CUSUM, both accumulating deviations from the target value on which the plot is standardised and both bound to zero. The positive CUSUM accumulates deviations when the operative time is longer than the target value indicating ‘deterioration’, and the negative CUSUM accumulates deviations when the operative time is shorter than the target value indicating ‘improvement’.26 This particular CUSUM illustrates longer operative times for the first 10 cases as surgeons familiarised themselves with the system, before reaching a steady state where operative times are mostly within one-half SD from the mean (figure 2). To that end, data are collected for teaching cases and trainees’ data can be captured and fed back. The negative CUSUM at no point departs from zero as there are no significant deviations towards shorter operative times. Online supplemental figure 2A,B presents examples where either the positive or the negative CUSUM goes beyond control limits, respectively.

Individual control and moving range charts may also be generated alongside the CUSUM to provide a view of the raw data and the temporal evolution of operative time range (figure 3A,B). These types of analyses, which may be provided to individual surgeons every 3 months at their request, are intended to provide a means of tracking performance as well as identifying where operative time may be moving out of steady state and where investigation may be required (online supplemental figure 2).

The below types of analyses use conversion rate for cholecystectomy data from the registry collected up to March 2021 as an example of a binary data metric (recorded as not converted (0) or converted (1)).

Comparative performance, meta-individual and individual performance analyses may be performed through similar approaches to those discussed for continuous data (equivalent data presented in online supplemental figures 3–5). Notably, the conversion rate for the surgeon with the most cases is considerably higher than the upper 95% CI limit illustrated in the funnel plot (online supplemental figure 3). This finding could lead to further investigation using other types of registry data analysis to help determine whether intervention, such as further training, could be beneficial with a view to preventing potentially avoidable conversion in subsequent cases.

Two additional types of analyses that allow tracking of binary surgical outcomes are p-charts and variable life-adjusted display (VLAD) charts. A p-chart is a control chart that provides a means of identifying one-off rises in non-favorable outcomes in the short term.27 For example, the proportion of conversions and corresponding CIs can be calculated each week or each month (the small sample size results in wide CIs but allows for high sensitivity). Figure 4 demonstrates a hypothetical scenario where the conversion rate exceeded the upper control limit at week 29.

Request permission

In the longer term, VLAD charts provide a method of monitoring more persistent changes in binary outcomes.28 In this analysis, a probability of unplanned conversion is assigned and every time an operation is not converted this is added to the probability, while every conversion is deducted as 1‍–‍p. Online supplemental figure 6A presents an example VLAD chart based on an estimated expected rate of unplanned conversion of 0.06. In this example, a few conversions after case number 35 result in a rapid decrease in the VLAD chart indicating deteriorating performance. However, when the same data are mapped to the operating surgeon’s expected rate of unplanned conversions of 0.10, this VLAD chart demonstrates that the surgeon/device had been overperforming until the two last surgeries (online supplemental figure 6B). As such, the VLAD chart is a highly sensitive measure that can provide an early alert of a trend change in performance.

Overall, different analyses using binary CUSUM, p-charts and VLAD charts viewed together can provide a complete view of surgeon and device performance; intervention decisions are not made based on one metric alone. Rather, moving out of control limits in one chart may raise a flag that will lead to further investigation using other analysis types. Furthermore, patient demographics, such as body mass index, ASA status and secondary diagnoses, and procedural plan data recorded in the registry can also be considered alongside performance metrics. For example, a higher conversion rate observed in a monthly p-chart may be combined with a relatively high number of patients with cancer and high ASA status undergoing surgery in this month, in which case a slightly higher rate of conversion may be expected.

Discussion

Transparent reporting of long-term safety and efficacy data for RSSs is crucial as the field evolves, allowing for detailed comparisons of surgical outcomes, such as conversion rates, operative time, intra-operative complications and AE incidence, with published data from other operative modalities.

This RSS registry could help institutions and teams to perform optimally, with detailed monitoring of performance and outcomes through different types of analyses within the registry dataset. Funnel plots can quickly provide a clear summary of how institutions and teams are performing compared with the total population. Further and more detailed investigations may be conducted through meta-analyses, where multiple factors can be included, and their effects explored. At the level of the individual surgeon, control method analysis with CUSUM, control charts and p-charts may help identify and rectify any potential issues and help improve performance and health outcomes. Taken together, these analyses could help proactively identify opportunities for quality improvement in care, including where targeted training for surgeons and surgical teams may improve outcomes for patients.

Additionally, there is increasing evidence that clinical practice may be subject to the Hawthorne effect,29 30 and introducing routine monitoring of outcomes with regular feedback to surgical teams has been shown to have a positive effect on performance. A prospective, nationwide cluster randomized trial in France found that implementation of a control chart-based program had a favorable effect on surgical outcomes for patients undergoing digestive tract surgery, including a statistically significant reduction in major AEs following surgery.31 As such, the implementation of this surgical registry may itself contribute to positive outcomes and further minimize risk to patient safety with use of this RSS.

In the long term, mapping of patient characteristics to surgical outcome data collected in the registry may help identify particular risk factors for specific procedures through the development of predictive models. As a landmark example, a database of 15 000 patients was used to create the European System for Cardiac Operative Risk Evaluation (EuroSCORE) risk model as a method of predicting operative mortality for patients undergoing cardiac surgery.32

While the registry has been meticulously designed to capture several key parameters, accumulation of data relies on their consistent submission into the platform by surgeons and their teams, and so there is inherent potential for missing or incomplete entries. However, the initial training on how to create accurate and complete case entries delivered to new registry users and the routine data cleaning processes in place are intended to help minimize missing data.

Overall, it is anticipated that the registry will allow for high-powered, large-scale analyses of surgical outcome data that could provide insights into risk factors for patients across a range of indications, with data captured for both common and rarer procedures. Harnessing the power of large-scale registry data may transform surgical care, with routine surveillance of device performance in a real-world setting.

Conclusions

Here, we describe the prospective establishment of a new surgical registry developed to facilitate the safe implementation of a next-generation RSS into clinical use. The registry enables static or dynamic comparative surgeon performance analyses, and performance can be assessed against warning and control limits. Binary surgical outcomes including unplanned conversions to alternative surgical modalities can also be evaluated using appropriate short-term and long-term analysis types. We anticipate that long-term monitoring of real-world registry data collected from first implementation of the device in live-human surgery will provide meaningful insights that could inform and improve surgical care.