Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation

Objectives

We introduce a novel automated SWBA that is based on an established and current surgical performance framework. Specifically, we aimed to

1. Operationalize the universal framework of intraoperative performance and adapt it to the particular demands of a spine surgery procedure (aim 1).

2. Develop computer-assisted assessment comprising criterion-referenced metrics reflecting these key demands and characteristics of intraoperative competency (aim 2).

3. Implement automated performance assessment in an authentic simulated OR workplace setting involving a multiprofessional team and evaluate validity evidence with particular focus on ACS (aim 3).

Design

Our multistage development process and investigation consisted of the three consecutive steps:

• Step 1: adaptation of performance framework (pertaining to aim 1, see previous discussion).

• Step 2: identification and design of metrics (pertaining to aim 2).

• Step 3: implementation in simulated OR for competency assessment (pertaining to aim 3).

We followed the reporting guidelines for healthcare simulation research.24 This study is part of a larger project on OR workplace simulation in spine surgery.25

Procedure

Step 3: implementation in simulated OR for competency assessment

Setting

A full-scale simulated OR was set up with all necessary equipment. We used a previously evaluated mixed-reality approach to radiation-free, image-guided spine surgery simulation to create an authentic simulated workplace for VP intervention.34 The simulation environment may also be classified as augmented reality (AR) because, as opposed to virtual reality (VR), it is located rather on the ‘real’ side of the mixed-reality spectrum, and central components of the OR context are physically represented. Based on CT data of a patient with an osteoporotic vertebral compression fracture, a patient-specific synthetic spine model (L2 and parts of L1 and L3) was three-dimensional-printed with two distinct materials for cortical and cancellous bones.35 This model was then embedded in synthetic soft tissue (red-coloured gel wax) and covered with synthetic skin (three layers of silicone with different levels of firmness, mimicking skin, subcutaneous fat, and muscle tissue).34 36 The synthetic patient model was then placed between the two parts of a mannequin phantom in prone position on the operating table and fully draped.

Our set-up included preoperative planning and enabled the subsequent simulation of the complete intraoperative VP workflow—from cut to suture. Our complete surgical team consisted of a surgeon (ie, surveyed participant) and a confederate OR team (ie, team members confederated with the study team, instructed to act on their roles in the simulated procedure) including an anesthesiologist, a scrub nurse, and a circulating nurse (see figure 1). The anesthesiologist’s workspace was set up at the patient’s head behind the drapes, equipped with vital sign monitors and a ventilator. The C-arm and the X-ray monitor screens were placed opposite to the surgeon and operated by the circulating nurse. Surgeons used a foot pedal to trigger simulated (radiation-free) X-ray acquisition. A mobile instrument table with all necessary instruments was positioned within the scrub nurse’s workplace, that is, with marker pen, scalpel, jamshidi needle, hammer, bone cement injector, clamps, and suture material. Additional pictures of our simulated workplace setting are shown in online supplemental figures A3–A5. The tasks and interactions of the team members were, for example, to monitor the patient and report the patient’s status to the team (anesthesiologist), to control the C-arm as instructed by the surgeon (circulating nurse), or to hand instruments to the surgeon during the procedure (scrub nurse).

Request reuse permission

Figure 1

Simulated workplace for vertebroplasty: intraoperative scene during needle insertion with surgical team (anaesthetist behind ether screen).

Recruitment procedure and sample

Surgeons were recruited from two university teaching hospitals, following snowball invitations via internal mailings (ie, we included in all e-mails and information our request to forward the invitation to potentially interested colleagues within the department). The surgeons were incentivized by offering them the opportunity to participate in the assessment and receive feedback on the outcome of the simulated surgical procedure. Eligibility criteria included surgical specialists in orthopedic, trauma, or neurosurgery with prior experience of VP or kyphoplasty interventions.

A total of 11 surgeons participated (72.7% male; six trauma surgeons, three orthopedic surgeons, and two trauma and orthopedic specialists). Their tenure on the job ranged between 0 year and 33 years (mean=7.82, SD=9.38) and between 0 and 200 real VP procedures performed previous to this study (mean=35, SD=9.38).

Simulated procedure

The study was conducted in an academic simulation center. First, surgeons were introduced to the patient case on a planning workstation outside of the simulated OR. Then, they were asked to plan and enter needle trajectories for a bipedicular, percutaneous VP at L2. Afterwards, participants entered the OR and were introduced to the surgical team. All participants were allowed to sufficiently familiarize with the set-up. After the scrub nurse helped participants with gowning, simulation started. It ended with the return of the suture material to the scrub nurse. After the procedure, the participants had the option to receive informal feedback from the study team and additionally review the simulated postoperative CT.

Data collection

1. Procedure planning data: preoperatively, planning data for the VP procedure were obtained from each surgeon, that is, trajectory data for needle insertion (see online supplemental table A2).

2. Intraoperative performance data, serving as input to performance metrics.

   1. Motion tracking data of the needle, the C-arm, and patient model. Data were postprocessed by removing extreme outliers (pairwise distances of subsequent positions 1.5 times the IQR above and below lower and upper quartiles) and applying a Butterworth low-pass filter.37

   2. Amount of bone cement injection, measured using a custom developed cement injection device connected via USB.

   3. X-ray/fluoroscopy acquisition, measured by recording foot pedal activation.

   4. Intraoperative teamwork and performance were video-taped (with two opposite cameras).

3. Survey data: After each simulation, surgeons additionally completed a questionnaire evaluation of the simulation environment (rated on a 5-point Likert-scale: 1=strongly disagree, 2=disagree, 3=neither agree nor disagree, 4=agree, and 5=strongly agree).

Measures

1. Standardized observational assessment of surgeons’ technical performance was conducted by a chief orthopedic surgeon using Objective Structured Assessment of Technical Skill (OSATS).25 OSATS is currently ‘considered the gold standard in the assessment of technical skills’ (Goldenberg and Grantcharov, p125)38 and consists of (1) a task-specific checklist, (2) a Global Rating Scale, and (3) a pass-fail rating. Assessment of surgeons’ non-technical performance was conducted by two trained raters using Observational Teamwork Assessment for Surgery (OTAS).25 OTAS is a well-established tool for OR teamwork assessment.39 It measures five non-technical skills essential to surgery. Each skill is measured along behavior exemplars: communication (eg, verbal confirmation of procedure and intra-op requirements), coordination (eg, assessment with nurse on status of instrument preparation before start), cooperation/backup behavior (eg, surgeon responds to questions and requests from nurse), leadership (eg, provides confirmation with nurse for specific surgical requirements ahead of action), and monitoring/situational awareness (eg, reassess set-up and intraoperative requirements in advance). Both OTAS raters underwent observational training before the onset of the simulation (including pairwise observations with discussion of discrepancies).

2. Computer-assisted metrics were applied to data from surgeons’ performances of the full VP procedure with the complete surgical team, except for metrics for DK which were applied to planning data.

Data and statistical analyses

As performance metrics from different categories differ in units of measurement, we normalized values to unit-free scores what facilitates interpretation and score accumulation. Target normalization was applied if target (ideal) and baseline (non-ideal) values were available. Otherwise, we employed ratio normalization, that is, using the minimum and maximum measurement values observed over all participants to normalize single measurements.40 Cronbach’s alpha was computed to analyze intradomain internal consistency, that is, pertaining to what extent scores within one domain measure the same construct. Spearman’s correlation was used to investigate interdomain associations of scores and associations of surgeons’ experience with observed technical (OSATS) and non-technical (OTAS) performance scores. Wilcoxon rank sum tests were used to test for group differences, for example, between OSATS pass/fail group’s associated computer-assisted assessment scores in the performance domains. All statistical analyses were computed with R V.4.0.2.

Step 1: adaptation of performance framework (aim 1)

Relevant indicators were identified and operationalized according to the framework’s intraoperative performance domains: PMS, DK, PR, ACS and an adjunct SP score (cf, table 1).

PMS are ‘required for neuromuscular coordination of various physical motions and motor functions into meaningful sequences, whose outcome is the accomplishment of simple tasks’(Madani et al, p258).14 Expert-defined indicators were path lengths to reach relevant anatomy, repositioning attempts, or target accuracy with respect to the preoperative planned trajectory. DK pertains to recitable facts, that is, knowledge of relevant anatomy, physiology and pathology, surgical technique and instruments, as well as literature-based knowledge regarding various aspects of patient management such as risk factors and complication rates.14 We operationalized this domain using surgeons’ preoperative planning data (cf, table 1). PR describes managing factors with surgeons’ ability to impact mental focus, stress, attentiveness, and objectives.14 Here, we identified total operation time and number of X-rays.41 42 ACS are defined as advanced mental capabilities such as forward planning, error prevention, and reassessment of the tactical approach with the team. Corresponding indicators therefore quantify compliance and deviation from the expected course of the operation without harmful patient outcome.14 Indicators relevant to VP include adhering to safety distances, avoiding violation of danger zones or, pertaining to communication, coordination of C-arm adjustment with assistance from the circulating nurse.27 Pertaining to VP, we defined our SP adjunct as the injection of an adequate amount of cement into the vertebra without causing leakage and avoiding injury of critical anatomical structures.

Step 2: identification and design of metrics (aim 2)

We then identified and designed computer-assisted metrics according to the framework (see previous step 1). Online supplemental table A2 describes resulting metrics’ definitions, normalization, and score aggregation. Metrics definitions were based on preoperative planning and intraoperative performance data, that is, surgeons’ preoperatively planned trajectories, intraoperatively recorded instrument trajectories, adjustments of the C-arm in spatial relation to patient anatomy, X-ray acquisitions, and cement injection data (cf, online supplemental table A2, score definition). Criterion-referenced targets (ideal performance) and baselines (non-ideal performance) could be established for the majority of metrics (cf, online supplemental table A2). Where possible, we used anatomical properties or limits for target and baseline definition. For example, in the needle insertion step, the target is defined as the maximum anatomically possible distance from the pedicle wall, the baseline defined as zero distance from the wall (see online supplemental table A2, ACS₅: safety margin pedicle wall). For other metrics, we relied on published empirical data and recommendations. For example, the target value of cement injection is defined as the mean of three recommendations for optimal cemented fraction30–32 multiplied with the case-specific volume of the vertebra (see online supplemental table A2, SP₃: volume of cement deployed in vertebra). Where anatomical limits or recommendations were not available, we established definitions using annotations collected from n=4 experienced and senior surgeons who were very familiar with this procedure. Online supplemental figure A1 shows the resulting labelling of optimal and acceptable areas for pedicle entry, used for metric definition and normalization in the pedicle entry step (see online supplemental table A2, ACS₂: pedicle entry).

Total scores for PMS, DK, PR, ACS, and SP were aggregated by summation of scores, except when highly critical structures (ie, spinal cord, nerve roots, and large vessels) were injured, in which case the SP score was set to 0 (see online supplemental table A2).

Step 3: implementation in simulated OR for competency assessment (aim 3)

We implemented the assessment in a contextualized full-scale OR simulation of a VP procedure with 11 surgeons within a confederate surgical team. We determined the feasibility and sought to gather first evidence for validity of our computer-assisted SWBA.

Mean procedure completion time for the two-sided VP was 27.81 min (SD=9.87, range=9.68–41.87). Overall procedure success rate was 72.73% (defined as completion without critical patient harm; that is, no highly critical structures were injured and therefore SP score>0).

After each simulation, participants evaluated the simulation via a standardized questionnaire. Participants appraised the authenticity of the simulation and proximity to clinical reality. Surgeons expressed agreement with the overall alignment of the simulation with real VP settings (median=4, IQR=1), strong agreement with workflow replication (median=5, IQR=0.5), and interaction with the OR team members (median=5, IQR=0).

We then determined surgeons’ performance and intracorrelation and intercorrelation within and between the framework’s performance domains (see table 2). Intradomain internal consistency within performance domains (Cronbach’s alpha) was at acceptable levels regarding PMS, PR, and ACS (see table 2). Significant interdomain associations were observed between PMS and PR (p=0.03). Association of domain scores with our adjunct SP was strong for ACS (p<0.01; see table 2).

Then we determined associations of automated performance score with experience as well as expert-based observational technical (OSATS) and non-technical (OTAS) assessment scores (see table 3).

Regarding surgeons’ experience (job tenure; number of procedures performed), we found strong associations with ACS and with SP. Online supplemental figure A2 shows scores as a function of experience (job tenure) and online supplemental table A3 provides a breakdown of participants into experience levels and their OSATS pass/fail rating, SP and overall score. We further observed medium to strong relationships between expert-rated technical performance (OSATS) and computer-assisted performance assessment: for ACS, SP and overall score. Strong associations between expert-rated non-technical performance (OTAS) and computer-assisted assessment were observed for ACS and SP.

In the next step, we investigated group differences between surgeons pertaining to the OSATS pass/fail rating. Figure 2 shows the corresponding box plot. Differences regarding overall score were statistically significant (W=0, p<0.01). Per domain differences were statistically significant for ACS (W=2, p=0.02) and SP (W=0, p<0.01).

Request reuse permission

In a final step, we used the OSATS pass/fail rating for standard setting. Using the contrasting groups method,43 we determined a cut score of 0.54 for our computer-assisted intraoperative competency score (see figure 3). Applying this to our sample, five surgeons were classified as non-competent (below cut score) and six surgeons as competent (above cut score) with no false negatives or false positives in comparison to expert-based OSATS pass/fail judgment.

Request reuse permission

Lastly, we evaluated validity evidence for our approach according to contemporary, unitary conceptualization of validity21–23 across the five sources:

1. Content. Representativeness of the achievement domain was established by the rigorous foundation of our simulation on expert-informed CTA27 and a high degree of contextualization of our simulation approach. This is underpinned by participants’ appraisal of the authenticity of the simulation. Test item representativeness is supported by the holistic characteristic of the universal framework of intraoperative performance and synthesis of our operationalization from multiple sources, that is, CTA, literature and expert knowledge.

2. Response process. Scoring format and combination are informed by the universal framework of intraoperative performance as a profound conceptual foundation. Meaningfulness of score description and interpretation are supported by criterion-referenced scores facilitating interpretation. Our rigorous study design, including standardized and computer-assisted data acquisition, further supports validity of response process.

3. Internal Structure. Our computer-assisted automated approach produced ratings that are reproducible, reliable, and free from bias pertaining to subjective observation. Intraitem/domain internal consistency is at acceptable levels (see table 2). Interitem/domain associations of scores (between performance domains) indicate heterogeneity of constructs across the domains; association with the overall SP outcome was strong for ACS (see table 2).

4. Relationship to other variables. Observed associations of computer-assisted assessment scores with further metrics such as surgeons’ experience and technical (OSATS) and non-technical (OTAS) performance were considerable regarding overall score, and strong for ACS and SP in particular (see table 3).

5. Consequences of testing. The educational relevance of our automated assessment approach is substantiated by statistically significant difference in scores between surgeons in the pass/fail groups. Per domain analysis of differences in particular supports the use of ACS and SP scores as discriminators of competent performance for summative decisions. The consequences of using our approach to classify surgeons as non-competent or competent are favourably supported by no false negatives or false positives when opposing computer-assisted classification with expert rater-based judgment.

Limitations

Our approach has some limitations that should be acknowledged. We laid out our use of performance metrics specifically to minimally invasive spine surgery. Although various of our metrics are generic to surgical performance, such as accuracy, precision, pace, and use of intraoperative imaging, further investigations into further performance metrics are necessary, for example, for open procedures. Although we used a systematic approach to obtain expert consensus on our performance indicators and associated metrics, we cannot infer how specialists in other surgical domains might appraise specific performance outcomes. Given the vast advancements and adoption of technology in surgical performance assessment, we acknowledge that future adoption of computer-assisted assessments will incorporate further performance indicators, for example, intraoperative stress assessed through ambulatory assessments. Our simulation environment was based on a rigorous, in-depth development process to authentically mimic an OR setting as well as multiprofessional surgical practice.27 Yet, surgeons might not have performed to their fullest potential in the simulation, for example, due to lack of familiarity with the setting or hesitation to being observed. Participants other than the surgeon were confederates to the study team. While this helped to standardize assessment, it also added to the costs and efforts in preparations and simulation planning. Further limitations include our convenience sampling approach and limited sample size. Future investigations should include a larger number of participants and a differentiated analysis of experience levels and subgroups (eg, interns, residents, attendings, etc). Finally, the interpersonal communication (IPC) domain suggested in the original framework14 was not considered. Future studies need to further apply all performance domains and gather evidence for validity of assessment across different surgical procedures and specialties beyond spine surgery. Particular focus should be devoted to validity evidence in terms of learners’ operative performance and patient outcomes in the long term, that is, functional or morbidity outcomes.

Implications for research and surgical practice

Our approach may inform further research in several ways: first, future investigations should scrutinize the utility of the performance domain model in other surgical procedures or conditions (ie, varying patient factors like high acuity, high body mass index, pediatric vs adult, or unusual anatomy). Moreover, our empirical findings concerning the key role of ACS should be corroborated with particular attention to implications for patient safety and surgical outcomes.14 Second, the range of skills covered should be broadened further toward non-technical aspects. The domain of IPC should be incorporated using computer-assisted assessment, for example, employing machine learning techniques,48 and association with non-technical observational assessment scores (eg, OTAS) should be investigated. Such assessments need to provide criterion-referenced indicators relating to competencies such as communication, teamwork, or leadership. Moreover, investigations should address how automated assessment results can be best formatted and fed back to support interpretation and provide idiosyncratic guidance, that is, feedback which is ‘individualized, meaningful, and case-specific’.^14(p263) Third, assessments and validity evidence should be further extended to cover the entire surgical team. It should be of particular interest, how intra-team coordination and cooperation can be automatically and objectively assessed and interpreted with regard to procedure and case-specific demands.14 Assessments therefore need to include team members other than the surgeon as surveyed participants and criterion-referenced performance indicators have to be developed to also capture their intraoperative performance. Fourth, operationalization of the universal framework of intraoperative performance should be implemented in traditional observational work-based assessments in real OR settings, eventually reducing the current inflation of assessment tools19 for different specialties and aspects of performance (ie, technical and non-technical). Moreover, consistency of participants’ intraoperative performance should be investigated in both, the real OR and contextualized simulation setting. For summative assessment, the correlation of assessments in both settings is of particular interest as part of the validity argument.5 Regarding formative assessments, investigations into individual as well as joint effects of assessment and feedback in contextualized simulation settings on patient outcomes and surgical performance should be of particular interest (ie, by using a prospective pre intervention–post intervention design).

Concerning implications to surgical practice, our approach draws on the principles of CBME and advocates objective criterion-referenced assessment of a comprehensive range of competencies in authentic and contextualized simulated surgical tasks as a complement to traditional workplace based assessments. Our computer-assisted assessment approach may complement rater-based observational assessment, which is time-consuming, inherently subjective, and therefore fraught with different biases.15 Yet, it shall not be intended as a replacement. Some degree of subjective professional judgment may actually be considered a necessary element of assessment as it can provide valuable feedback and add to a more authentic and holistic appraisal of learners’ competence enhancing the validity of assessments.49

Computer-assisted assessment in simulated workplaces allows standardizing many of the factors that affect assessments in real workplace environments, for example, effects for raters, the specific patient or scenario, and concurrently enables efficient assessment of key competencies in an OR context.50 Standardization supports the scoring and generalisation argument; contextualization strengthens the extrapolation argument of assessment score validity.50 Standardization is of particular interest in formative assessment where alignment with the individual developmental needs of surgical trainees is required. Here, our approach can help to establish required case numbers and cover variation in operative conditions in a structured way.16 In summative assessment, our approach contributes to establishing a balance between standardization and contextualization of assessment criteria.50 51