Article Text

Download PDFPDF

IDEAL approach to the evaluation of machine learning technology in epilepsy surgery: protocol for the MAST trial
  1. Aswin Chari1,2,
  2. Sophie Adler1,2,
  3. Konrad Wagstyl3,
  4. Kiran Seunarine1,2,
  5. Hani Marcus4,5,
  6. Torsten Baldeweg2 and
  7. Martin Tisdall1,2
  1. 1Department of Neurosurgery, Great Ormond Street Hospital, London, UK
  2. 2Developmental Neuroscience, Great Ormond Street Institute of Child Health, University College London, London, UK
  3. 3Wellcome Centre for Human Neuroimaging, University College London, London, UK
  4. 4Department of Neurosurgery, National Hospital for Neurology and Neurosurgery, London, UK
  5. 5Wellcome EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
  1. Correspondence to Dr Aswin Chari; aswin.chari.18{at}ucl.ac.uk

Abstract

Epilepsy and epilepsy surgery lend themselves well to the application of machine learning (ML) and artificial intelligence (AI) technologies. This is evidenced by the plethora of tools developed for applications such as seizure detection and analysis of imaging and electrophysiological data. However, few of these tools have been directly used to guide patient management. In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety.

In this protocol paper, we outline the preclinical (IDEAL stage 0) evaluation and the protocol for a prospective (IDEAL stage 1/2a) study to evaluate the utility of an ML lesion detection algorithm designed to detect focal cortical dysplasia from structural MRI, as an adjunct in the planning of stereoelectroencephalography trajectories in children undergoing intracranial evaluation for drug-resistant epilepsy.

  • neurostimulation devices
  • neurological devices
  • implantable neurostimulators
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Introduction

It is clear that the fields of epilepsy and epilepsy surgery lend themselves ideally to the application of machine learning (ML) and artificial intelligence (AI) technologies to directly benefit patient care in areas such as: seizure detection; analysis of clinical, electrophysiological and imaging data; seizure localisation and prediction of medical and surgical treatment outcomes.1 2 Although many of these applications have already been explored, as yet, there have been few reports of ML algorithms directly being used to guide management decisions in patients.1 3–5 The difficulty in proceeding from the development of algorithms to prospective evaluation in clinical care is attributable to issues surrounding transparency, reproducibility, ethics and effectiveness and changes required in traditional clinical trial designs.6–8 In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety (figure 1).9–12 This has been recently supplemented with a stage 0 that centres around a proportionate preclinical evaluation that balances safety while facilitating innovative first-in-human studies.12

Figure 1

Schematic of the IDEAL stages of evaluation for surgical/medical device innovation. IDEA, Idea, Development, Exploration, Assessment, Long-Term Follow-Up.

In this paper, we present the IDEAL stage 0 evaluation and protocol for a combined IDEAL stage 1/2a study for a lesion detection algorithm designed to detect focal cortical dysplasia as part of the planning of stereoelectroencephalography (SEEG) electrode trajectories in patients with drug-resistant epilepsy undergoing invasive presurgical evaluation.13 14

Algorithm background

The multicentre epilepsy lesion detection (MELD) algorithm was developed to detect focal cortical dysplasia using volumetric T1-weighted and fluid-attenuated inversion recovery (FLAIR) MRI sequences. Its utility is in aiding in the presurgical evaluation of drug-resistant epilepsy - to localise MRI lesions that may be responsible for epilepsy that may be amenable to surgical resection.

Its development and retrospective evaluation have been previously presented.13 14 Cortical surface-based features such as cortical thickness, grey-white matter interface contrast and FLAIR intensity from confirmed lesions were used to train a neural network classifier. The classifier was initially validated using a leave-one-out cross-validation approach but has since been validated on external cohorts with high sensitivity (73.7%) specificity (90.0%) and an area under the ROC curve of 0.75.14 15

IDEAL stage 0 evaluation

The IDEAL-D stage 0 evaluation involves classifying the device, assessing the risks using a failure modes and effect analyses (FMEA) and conducting an evaluation proportionate to the potential risks involved.12 Software is classified according to the associated device. Therefore, in the context of being used to plan SEEG trajectories, the MELD algorithm can be viewed as associated with a non-absorbable implant, putting it in tier 3 of the device categorisation. An FMEA shows that the main risk is the additional electrodes causing bleeding, which would be classified as occasional frequency (<1/100) and potentially of serious severity (could result in injury or impairment).16 Proportionately, the preclinical evaluation includes device, clinician and patient perspective studies, with a systems perspective gleaned from the existing literature. The cost of inserting additional electrodes is minimal compared with the cost of the SEEG procedure as a whole and within the normal clinical variation and, therefore, health-economic analyses are not required at this stage (table 1).12

Table 1

Preclinical evaluation of the MELD algorithm, classified by study type according to the IDEAL stage 0 framework

Protocol for IDEAL stage 1/2a evaluation

The aim of the MELD as an Adjunct for SEEG Trajectories (MAST) Trial, a single-arm single-centre prospective pilot (IDEAL stage 1/2a) study, is to assess whether MELD is a helpful adjunct in the planning of SEEG electrode trajectories. Patients undergoing SEEG at our centre will be eligible for inclusion. Exclusion criteria include tuberous sclerosis or other large structural abnormality and prior resective surgery. The pilot sample size is 20, chosen based on the principles of the IDEAL stage 2a.9 Fully informed consent will be required for participation, either from the parent or the child, in line with UK law. The patient pathway is shown in figure 2 and summarised below.

Figure 2

Study flowchart in the MAST trial, with routine clinical care pathway (orange) and trial-specific (blue) elements shown. MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SEEG, stereoelectroencephalography; MAST, MELD as an Adjunct for SEEG Trajectories.

Following routine planning of the SEEG electrode trajectories by the multidisciplinary team (MDT) using the non-invasive evaluation (clinical semiology, EEG videotelemetry and MRI±other adjuncts, figure 3A), the MRI scans, which are collected as part of the routine clinical care, will be run through the MELD algorithm. The putative lesion clusters identified by the algorithm will be reviewed by an expert neuroradiologist with expertise in presurgical evaluation of epilepsy to ensure that they are not driven by artefact. Once artefacts have been excluded, the top three putative lesion clusters will then be identified (on the ipsilateral side only if the planned implantation is unilateral). If these clusters are not already sampled by existing electrodes, up to three extra electrodes may be added to sample from each of those top three lesion clusters (figure 3B).

Figure 3

(A) Schematic of current clinical planning of electrode locations using the non-invasive evaluation by the MDT. The data from the pre-surgical evaluation are used to generate hypotheses of the location of the SOZ. The primary hypothesis in this example is the temporal lobe (green) with alternative areas involving orbitofrontal and insular cortices (yellow). Electrode trajectories are then planned to sample these areas of interest and establish important resection boundaries, with red dots and lines showing orthogonal and parallel electrodes respectively. (B) Example of planned electrodes on sagittal, axial, coronal and 3D views of a T1-weighted MRI scan. The MELD-identified lesion (grey, yellow arrow) is not sampled as the nearest electrode (yellow on the 3D panel) traverses postero-inferior to it. In this example, an extra electrode would, therefore, be added to sample the identified lesion in the depths of the supramarginal gyrus. EEG, electroencephalography; MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SOZ, seizure onset zone; PET, positron emission tomography; SPECT, single-photon emission computed tomography.

The primary objective is to assess the proportion of patients that had additional electrode (ie, extra electrodes implanted into unsampled identified clusters) contacts in the clinically defined seizure onset zone (SOZ). Given the small sample size, no statistical testing will be conducted, and the results will be descriptive statistics only. For each patient, we will also assess a number of secondary objectives, including blinded neurophysiological and neurosurgical assessments of whether or not the data from the additional electrodes would have affected subsequent management decisions (table 2). To gain an understanding of the situations in which the algorithm might be particularly helpful, we will also collect data on the indication for SEEG and the MDT members’ confidence in the ability of the SEEG implantation to identify a seizure onset zone. A transparent account of harms, including the number of electrodes implanted into ‘false positive’ lesions, will be reported, allowing a balanced assessment of risks and benefits prior to consideration of further stage testing. Steps taken to optimise the workflow will also be reported.

Table 2

Primary and secondary objectives of the MAST trial

The risks and benefits of such an approach were carefully considered during the design phase. In order to evaluate the incremental benefit of the algorithm, the clinical standard of care could not be altered and, therefore, moving existing electrodes was not considered an option. The risk of complications per additional electrode is small.16 In addition, a group of 14 parents of children with epilepsy and one adolescent with epilepsy were surveyed at the Epilepsy in Childhood: Carers Uniting with Researchers Information Day. All 15 (100%) agreed that the risk would be acceptable when balanced against the potential benefits, and all 15 (100%) would enrol in the trial were they/their child to undergo SEEG implantation.

Ethical approval has been received for this study from the UK Health Research Authority (IRAS ID 275480) and the study began recruitment in September 2020. Further details of the protocol are available at https://clinicaltrialsgov/ct2/show/NCT04383028. The study has been funded by EANS-Stryker Research Grant 2020 &amp The Rosetrees Trust (A2665), but the funders had no input into the design or conduct of the study. The study will be reported in according with relevant guidelines including IDEAL stage 1/2a, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) & Standards for reporting diagnostic accuracy (STARD) guidelines.11 17 18

IDEAL stage 2b and beyond

We see the MELD algorithm as an additional part of the presurgical evaluation, which should be available to clinicians before the SEEG electrode trajectories are planned. The current algorithm was developed using single-centre data, with the clinical study being conducted at the same centre. The first step to further generalise the applicability would be to construct a large multicentre data set that accounts for variability in subjects, scan strength and protocols across centres. This is currently being constructed as part of the MELD study.19

If this should progress beyond the current stage 1/2a study, we envision conducting a stage 2b study as a feasibility randomised trial in a small number of centres, which assesses the proportion of patients in whom a SOZ is identified on SEEG with and without the benefit of the algorithm. This would allow us to optimise stability of the algorithm across centres, train other surgeons to integrate it into their clinical workflow, establish estimated effect sizes and assess feasibility of conducting a larger randomised study across multiple centres as an IDEAL stage 3 study.

Analysis of the stage 2b or 3 studies may also result in the additional findings, such as a select group of patients (eg, with interictal positron emission tomography (PET), ictal single-photo emission computed tomography (SPECT) or magnetoencephalography (MEG) concordance with the top identified lesions) in whom it may be debated that SEEG is not necessary prior to resection. If such findings were identified, the subsequent prospective clinical evaluation would have to go through the rigorous process of evaluation from stage 0 again as it would be using the algorithm in a different context to the present evaluation. However, many aspects may overlap with the current evaluation, making the process more streamlined.

Conclusion

ML technologies have much to offer to clinicians and patients in the field of neurosurgery and safe and robust evaluation is crucial to realising these opportunities. In this paper, we have outlined the IDEAL stage 0 evaluation, a protocol for an IDEAL stage 1/2a clinical trial and future directions for the use of the MELD algorithm in assisting the planning of SEEG trajectories. The robust framework balances safety and innovation, ultimately benefitting patients by improving outcomes.

Ethics statements

Patient consent for publication

Acknowledgments

The authors would like to acknowledge the SEEG team at Great Ormond Street Hospital for embracing the study and integrating it into the clinical workflow.

References

Footnotes

  • Twitter @aswinchari

  • Contributors AC, SA, KW, TB and MT conceived the study and developed the protocol. KS and HM provided methodological input to optimise study design. All authors were involved in the drafting and editing of the manuscript and have reviewed and approved the final version.

  • Funding This project is supported by the EANS-Stryker Research Grant 2020 and The Rosetrees Trust (A2665). AC is supported by a Great Ormond Street Hospital (GOSH) Children’s Charity Surgeon Scientist Fellowship. This work has been supported by the GOSH-National Institute of Health Research Biomedical Research Centre.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.