Epilepsy and epilepsy surgery lend themselves well to the application of machine learning (ML) and artificial intelligence (AI) technologies. This is evidenced by the plethora of tools developed for applications such as seizure detection and analysis of imaging and electrophysiological data. However, few of these tools have been directly used to guide patient management. In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety.
In this protocol paper, we outline the preclinical (IDEAL stage 0) evaluation and the protocol for a prospective (IDEAL stage 1/2a) study to evaluate the utility of an ML lesion detection algorithm designed to detect focal cortical dysplasia from structural MRI, as an adjunct in the planning of stereoelectroencephalography trajectories in children undergoing intracranial evaluation for drug-resistant epilepsy.
- neurostimulation devices
- neurological devices
- implantable neurostimulators
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
It is clear that the fields of epilepsy and epilepsy surgery lend themselves ideally to the application of machine learning (ML) and artificial intelligence (AI) technologies to directly benefit patient care in areas such as: seizure detection; analysis of clinical, electrophysiological and imaging data; seizure localisation and prediction of medical and surgical treatment outcomes.1 2 Although many of these applications have already been explored, as yet, there have been few reports of ML algorithms directly being used to guide management decisions in patients.1 3–5 The difficulty in proceeding from the development of algorithms to prospective evaluation in clinical care is attributable to issues surrounding transparency, reproducibility, ethics and effectiveness and changes required in traditional clinical trial designs.6–8 In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety (figure 1).9–12 This has been recently supplemented with a stage 0 that centres around a proportionate preclinical evaluation that balances safety while facilitating innovative first-in-human studies.12
In this paper, we present the IDEAL stage 0 evaluation and protocol for a combined IDEAL stage 1/2a study for a lesion detection algorithm designed to detect focal cortical dysplasia as part of the planning of stereoelectroencephalography (SEEG) electrode trajectories in patients with drug-resistant epilepsy undergoing invasive presurgical evaluation.13 14
The multicentre epilepsy lesion detection (MELD) algorithm was developed to detect focal cortical dysplasia using volumetric T1-weighted and fluid-attenuated inversion recovery (FLAIR) MRI sequences. Its utility is in aiding in the presurgical evaluation of drug-resistant epilepsy - to localise MRI lesions that may be responsible for epilepsy that may be amenable to surgical resection.
Its development and retrospective evaluation have been previously presented.13 14 Cortical surface-based features such as cortical thickness, grey-white matter interface contrast and FLAIR intensity from confirmed lesions were used to train a neural network classifier. The classifier was initially validated using a leave-one-out cross-validation approach but has since been validated on external cohorts with high sensitivity (73.7%) specificity (90.0%) and an area under the ROC curve of 0.75.14 15
IDEAL stage 0 evaluation
The IDEAL-D stage 0 evaluation involves classifying the device, assessing the risks using a failure modes and effect analyses (FMEA) and conducting an evaluation proportionate to the potential risks involved.12 Software is classified according to the associated device. Therefore, in the context of being used to plan SEEG trajectories, the MELD algorithm can be viewed as associated with a non-absorbable implant, putting it in tier 3 of the device categorisation. An FMEA shows that the main risk is the additional electrodes causing bleeding, which would be classified as occasional frequency (<1/100) and potentially of serious severity (could result in injury or impairment).16 Proportionately, the preclinical evaluation includes device, clinician and patient perspective studies, with a systems perspective gleaned from the existing literature. The cost of inserting additional electrodes is minimal compared with the cost of the SEEG procedure as a whole and within the normal clinical variation and, therefore, health-economic analyses are not required at this stage (table 1).12
Protocol for IDEAL stage 1/2a evaluation
The aim of the MELD as an Adjunct for SEEG Trajectories (MAST) Trial, a single-arm single-centre prospective pilot (IDEAL stage 1/2a) study, is to assess whether MELD is a helpful adjunct in the planning of SEEG electrode trajectories. Patients undergoing SEEG at our centre will be eligible for inclusion. Exclusion criteria include tuberous sclerosis or other large structural abnormality and prior resective surgery. The pilot sample size is 20, chosen based on the principles of the IDEAL stage 2a.9 Fully informed consent will be required for participation, either from the parent or the child, in line with UK law. The patient pathway is shown in figure 2 and summarised below.
Following routine planning of the SEEG electrode trajectories by the multidisciplinary team (MDT) using the non-invasive evaluation (clinical semiology, EEG videotelemetry and MRI±other adjuncts, figure 3A), the MRI scans, which are collected as part of the routine clinical care, will be run through the MELD algorithm. The putative lesion clusters identified by the algorithm will be reviewed by an expert neuroradiologist with expertise in presurgical evaluation of epilepsy to ensure that they are not driven by artefact. Once artefacts have been excluded, the top three putative lesion clusters will then be identified (on the ipsilateral side only if the planned implantation is unilateral). If these clusters are not already sampled by existing electrodes, up to three extra electrodes may be added to sample from each of those top three lesion clusters (figure 3B).
The primary objective is to assess the proportion of patients that had additional electrode (ie, extra electrodes implanted into unsampled identified clusters) contacts in the clinically defined seizure onset zone (SOZ). Given the small sample size, no statistical testing will be conducted, and the results will be descriptive statistics only. For each patient, we will also assess a number of secondary objectives, including blinded neurophysiological and neurosurgical assessments of whether or not the data from the additional electrodes would have affected subsequent management decisions (table 2). To gain an understanding of the situations in which the algorithm might be particularly helpful, we will also collect data on the indication for SEEG and the MDT members’ confidence in the ability of the SEEG implantation to identify a seizure onset zone. A transparent account of harms, including the number of electrodes implanted into ‘false positive’ lesions, will be reported, allowing a balanced assessment of risks and benefits prior to consideration of further stage testing. Steps taken to optimise the workflow will also be reported.
The risks and benefits of such an approach were carefully considered during the design phase. In order to evaluate the incremental benefit of the algorithm, the clinical standard of care could not be altered and, therefore, moving existing electrodes was not considered an option. The risk of complications per additional electrode is small.16 In addition, a group of 14 parents of children with epilepsy and one adolescent with epilepsy were surveyed at the Epilepsy in Childhood: Carers Uniting with Researchers Information Day. All 15 (100%) agreed that the risk would be acceptable when balanced against the potential benefits, and all 15 (100%) would enrol in the trial were they/their child to undergo SEEG implantation.
Ethical approval has been received for this study from the UK Health Research Authority (IRAS ID 275480) and the study began recruitment in September 2020. Further details of the protocol are available at https://clinicaltrialsgov/ct2/show/NCT04383028. The study has been funded by EANS-Stryker Research Grant 2020 & The Rosetrees Trust (A2665), but the funders had no input into the design or conduct of the study. The study will be reported in according with relevant guidelines including IDEAL stage 1/2a, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) & Standards for reporting diagnostic accuracy (STARD) guidelines.11 17 18
IDEAL stage 2b and beyond
We see the MELD algorithm as an additional part of the presurgical evaluation, which should be available to clinicians before the SEEG electrode trajectories are planned. The current algorithm was developed using single-centre data, with the clinical study being conducted at the same centre. The first step to further generalise the applicability would be to construct a large multicentre data set that accounts for variability in subjects, scan strength and protocols across centres. This is currently being constructed as part of the MELD study.19
If this should progress beyond the current stage 1/2a study, we envision conducting a stage 2b study as a feasibility randomised trial in a small number of centres, which assesses the proportion of patients in whom a SOZ is identified on SEEG with and without the benefit of the algorithm. This would allow us to optimise stability of the algorithm across centres, train other surgeons to integrate it into their clinical workflow, establish estimated effect sizes and assess feasibility of conducting a larger randomised study across multiple centres as an IDEAL stage 3 study.
Analysis of the stage 2b or 3 studies may also result in the additional findings, such as a select group of patients (eg, with interictal positron emission tomography (PET), ictal single-photo emission computed tomography (SPECT) or magnetoencephalography (MEG) concordance with the top identified lesions) in whom it may be debated that SEEG is not necessary prior to resection. If such findings were identified, the subsequent prospective clinical evaluation would have to go through the rigorous process of evaluation from stage 0 again as it would be using the algorithm in a different context to the present evaluation. However, many aspects may overlap with the current evaluation, making the process more streamlined.
ML technologies have much to offer to clinicians and patients in the field of neurosurgery and safe and robust evaluation is crucial to realising these opportunities. In this paper, we have outlined the IDEAL stage 0 evaluation, a protocol for an IDEAL stage 1/2a clinical trial and future directions for the use of the MELD algorithm in assisting the planning of SEEG trajectories. The robust framework balances safety and innovation, ultimately benefitting patients by improving outcomes.
Patient consent for publication
The authors would like to acknowledge the SEEG team at Great Ormond Street Hospital for embracing the study and integrating it into the clinical workflow.
Contributors AC, SA, KW, TB and MT conceived the study and developed the protocol. KS and HM provided methodological input to optimise study design. All authors were involved in the drafting and editing of the manuscript and have reviewed and approved the final version.
Funding This project is supported by the EANS-Stryker Research Grant 2020 and The Rosetrees Trust (A2665). AC is supported by a Great Ormond Street Hospital (GOSH) Children’s Charity Surgeon Scientist Fellowship. This work has been supported by the GOSH-National Institute of Health Research Biomedical Research Centre.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.