Surgery is perpetually at the cutting edge of innovation. And like in other innovative industries, the rate of uptake of new technology often outstrips comprehensive understanding of the systems changes and safety implications encountered. Robotic-assisted surgery (RAS) presents potential benefits to patients, including shorter hospital stays, reduced postoperative pain, and quicker recovery time. However, patient safety incidents may be as high as double compared with traditional open surgery,1 revealing the cost of new technology integration, and reminiscent of the rise of laparoscopic surgery in the early 1990s. Along with a supportive culture and effective systems, high-quality training is one of the foundations of successful technology adoption. In the present issue of BMJ Surgery, Interventions & Health Technologies, Butterworth et al present an in-depth training programme for robotic-assisted surgery, focusing on one specific surgical robot. The authors have developed what appears to be a comprehensive hybrid training programme, combining online education followed by face-to-face simulations and cadaver sessions with real surgical teams. This study provides initial validity evidence which is important for technology implementation with the ultimate aim to have a training programme that equips surgeons to expertly embed robotic surgery within their practice. The aim of our editorial is to provide a helpful critique regarding validity, and introduce the role of Human Factors to the successful implementation and evaluation of RAS training.
Like many applied studies of this type there are some conceptual and methodological limitations which limit the validity of findings and also broadly applicable to surgical education research. Butterworth et al aimed to evaluate the effectiveness of the training programme, however without defined standards it is unclear whether the training was aimed at improving surgeons’ technical ability or whether it was to train them to proficiency. Implementing a validity framework such as Kirkpatrick’s2 can be invaluable in this respect, as it allows researchers to evaluate both formal and informal training methods against four levels of criteria: reactions (did the training meet surgeons’ needs?), learning (has knowledge or skill increased?), behaviour (can surgeons now apply robotic surgical skills in real life?), and results (has training improved outcomes and safety?). By applying Kirkpatrick’s lens to the present study, we can say that the highest level of validity is at level 2: learning; as there is some evidence of participant skill improving. However, 2 of the 17 surgeons moved from intermediate to novice level which means that the training was not universally successful and may even have been counterproductive. Heterogeneity in prior experience of surgery and robotics of participants combined with the modest sample size and its subsequent stratification for analysis may have reduced the accuracy of the results. Furthermore, the prerequisite online training seems important for maximising on-site hands-on time; however, the lengthy duration (10 hours) and the lack of detail on content, objectives and assessment raises questions on how this met surgeons’ needs.
A second validity framework, Kane’s, is particularly helpful in the design of surgical education trials as it forces the researcher to justify very clearly the purpose, target sample, and context of intended impact.3 The framework tests validity evidence against four inferences: scoring, generalisation, extrapolation and implications. For the first inference, scoring, Butterworth et al implemented the Global Evaluative Assessment of Robotic Skills (GEARS) tool. This a strength of the study as this tool has validity evidence, however while a statistically significant improvement in GEARS score is encouraging, this does not equate to competence. In the study, expected GEARS scores for each group were calculated from only three previous studies, the first of which included 21 urologists, and looked at the correlation between simulation and intraoperative performance, rather than the correlation between GEARS scores and level of surgical skill. Without confidence in scoring, measured improvement may be unreliable and not reflect actual skill progression. The addition of a control group and blinding of the assessors can improve the validity of results and basis for robust conclusions.
There are many potential confounding variables that complicate the evaluation of training effectiveness, and the study by Butterworth et al presents several of these. Understanding the unique effects of training with unfamiliar surgical team members; impact of prior RAS training, surgical experience and volume, stage of career; and participants’ motivation can all have an important impact on determining whether training is effective or not. One important confounder which the authors described is prior expertise with a different robot. We know from Human Factors research in other contexts that skills gained in one context do not necessarily transfer to another. Similar to airline pilots who are trained and licensed to operate a particular type of aircraft, surgeons with expertise operating one robot may not see their skills transfer easily to another manufacturer’s version. A degree of ‘unlearning’ skills may be required.
Although not the focus of the study by Butterworth et al, the importance of Human Factors in RAS cannot be underestimated. It is now widely appreciated that individual performance is dependent on more than just technical skill: the team, systems and design processes in which surgeons are immersed can promote a good surgeon to excellence. For example, during an open operation, the theatre team stands close together in an open space. This facilitates effective verbal and non-verbal communication, ease of movement around the patient and equipment, improved situational awareness and a natural team ethos to develop. The ergonomics change immediately with the introduction of a laparoscope: the lights are dimmed; focus is shifted from the patient to a screen resulting in loss of eye contact and some non-verbal cues; the instruments used provide less tactile feedback and demand greater skill and dexterity, and maintain situational awareness requires concerted effort. Robotic surgery presents further challenges: the operating surgeon is physically distant from the patient and the team resulting in an inevitable impact on leadership and teamwork; communication through gestures and non-verbal cues is more challenging, and the equipment set-up often results in a reduced range of movement for the surgeon and surgical assistant.4 Dru et al acknowledge the practical and environmental demands that robotics place on the whole operating team, and highlight specific points during robotic-assisted radical prostatectomy during which flow disruption is likely to occur.5 These communication, coordination, equipment and technological hurdles present risk to a safe and effective system, and require mitigation to improve efficiency and reduce errors.6 7
Adopting new technologies at scale is essential to progress, but the challenges cannot be underestimated. In seminal work on implementation of new surgical techniques, Amy Edmondson et al identified several process steps associated with success8 including (1) enrolment strategies to motivate the team, (2) preparation and practice sessions to build psychological safety, and (3) reflection to promote shared understanding and reveal process improvement opportunities. Human Factors science has established its place in healthcare and should be incorporated in surgical training programmes. Training a team together in this way would encourage and normalise inclusive decision-making, and improve interdisciplinary communication and collaboration.
By building on training programmes like the one presented by Butterworth et al, setting specific competence standards, and taking a Human Factors approach to integration of training and systems, RAS can optimise team performance, enhance patient safety, and fulfil the promise of revolutionising surgical care.
