Introduction
It is clear that the fields of epilepsy and epilepsy surgery lend themselves ideally to the application of machine learning (ML) and artificial intelligence (AI) technologies to directly benefit patient care in areas such as: seizure detection; analysis of clinical, electrophysiological and imaging data; seizure localisation and prediction of medical and surgical treatment outcomes.1 2 Although many of these applications have already been explored, as yet, there have been few reports of ML algorithms directly being used to guide management decisions in patients.1 3–5 The difficulty in proceeding from the development of algorithms to prospective evaluation in clinical care is attributable to issues surrounding transparency, reproducibility, ethics and effectiveness and changes required in traditional clinical trial designs.6–8 In recent years, the Idea, Development, Exploration, Assessment, Long-Term Follow-Up (IDEAL) collaboration has formalised stages for the evaluation of surgical innovation and medical devices, and, in many ways, this pragmatic framework is also applicable to ML/AI technology, balancing innovation and safety (figure 1).9–12 This has been recently supplemented with a stage 0 that centres around a proportionate preclinical evaluation that balances safety while facilitating innovative first-in-human studies.12
Schematic of the IDEAL stages of evaluation for surgical/medical device innovation. IDEA, Idea, Development, Exploration, Assessment, Long-Term Follow-Up.
In this paper, we present the IDEAL stage 0 evaluation and protocol for a combined IDEAL stage 1/2a study for a lesion detection algorithm designed to detect focal cortical dysplasia as part of the planning of stereoelectroencephalography (SEEG) electrode trajectories in patients with drug-resistant epilepsy undergoing invasive presurgical evaluation.13 14
Algorithm background
The multicentre epilepsy lesion detection (MELD) algorithm was developed to detect focal cortical dysplasia using volumetric T1-weighted and fluid-attenuated inversion recovery (FLAIR) MRI sequences. Its utility is in aiding in the presurgical evaluation of drug-resistant epilepsy - to localise MRI lesions that may be responsible for epilepsy that may be amenable to surgical resection.
Its development and retrospective evaluation have been previously presented.13 14 Cortical surface-based features such as cortical thickness, grey-white matter interface contrast and FLAIR intensity from confirmed lesions were used to train a neural network classifier. The classifier was initially validated using a leave-one-out cross-validation approach but has since been validated on external cohorts with high sensitivity (73.7%) specificity (90.0%) and an area under the ROC curve of 0.75.14 15
IDEAL stage 0 evaluation
The IDEAL-D stage 0 evaluation involves classifying the device, assessing the risks using a failure modes and effect analyses (FMEA) and conducting an evaluation proportionate to the potential risks involved.12 Software is classified according to the associated device. Therefore, in the context of being used to plan SEEG trajectories, the MELD algorithm can be viewed as associated with a non-absorbable implant, putting it in tier 3 of the device categorisation. An FMEA shows that the main risk is the additional electrodes causing bleeding, which would be classified as occasional frequency (<1/100) and potentially of serious severity (could result in injury or impairment).16 Proportionately, the preclinical evaluation includes device, clinician and patient perspective studies, with a systems perspective gleaned from the existing literature. The cost of inserting additional electrodes is minimal compared with the cost of the SEEG procedure as a whole and within the normal clinical variation and, therefore, health-economic analyses are not required at this stage (table 1).12
Protocol for IDEAL stage 1/2a evaluation
The aim of the MELD as an Adjunct for SEEG Trajectories (MAST) Trial, a single-arm single-centre prospective pilot (IDEAL stage 1/2a) study, is to assess whether MELD is a helpful adjunct in the planning of SEEG electrode trajectories. Patients undergoing SEEG at our centre will be eligible for inclusion. Exclusion criteria include tuberous sclerosis or other large structural abnormality and prior resective surgery. The pilot sample size is 20, chosen based on the principles of the IDEAL stage 2a.9 Fully informed consent will be required for participation, either from the parent or the child, in line with UK law. The patient pathway is shown in figure 2 and summarised below.
Study flowchart in the MAST trial, with routine clinical care pathway (orange) and trial-specific (blue) elements shown. MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SEEG, stereoelectroencephalography; MAST, MELD as an Adjunct for SEEG Trajectories.
Following routine planning of the SEEG electrode trajectories by the multidisciplinary team (MDT) using the non-invasive evaluation (clinical semiology, EEG videotelemetry and MRI±other adjuncts, figure 3A), the MRI scans, which are collected as part of the routine clinical care, will be run through the MELD algorithm. The putative lesion clusters identified by the algorithm will be reviewed by an expert neuroradiologist with expertise in presurgical evaluation of epilepsy to ensure that they are not driven by artefact. Once artefacts have been excluded, the top three putative lesion clusters will then be identified (on the ipsilateral side only if the planned implantation is unilateral). If these clusters are not already sampled by existing electrodes, up to three extra electrodes may be added to sample from each of those top three lesion clusters (figure 3B).
(A) Schematic of current clinical planning of electrode locations using the non-invasive evaluation by the MDT. The data from the pre-surgical evaluation are used to generate hypotheses of the location of the SOZ. The primary hypothesis in this example is the temporal lobe (green) with alternative areas involving orbitofrontal and insular cortices (yellow). Electrode trajectories are then planned to sample these areas of interest and establish important resection boundaries, with red dots and lines showing orthogonal and parallel electrodes respectively. (B) Example of planned electrodes on sagittal, axial, coronal and 3D views of a T1-weighted MRI scan. The MELD-identified lesion (grey, yellow arrow) is not sampled as the nearest electrode (yellow on the 3D panel) traverses postero-inferior to it. In this example, an extra electrode would, therefore, be added to sample the identified lesion in the depths of the supramarginal gyrus. EEG, electroencephalography; MDT, multidisciplinary team; MELD, multicentre epilepsy lesion detection; SOZ, seizure onset zone; PET, positron emission tomography; SPECT, single-photon emission computed tomography.
The primary objective is to assess the proportion of patients that had additional electrode (ie, extra electrodes implanted into unsampled identified clusters) contacts in the clinically defined seizure onset zone (SOZ). Given the small sample size, no statistical testing will be conducted, and the results will be descriptive statistics only. For each patient, we will also assess a number of secondary objectives, including blinded neurophysiological and neurosurgical assessments of whether or not the data from the additional electrodes would have affected subsequent management decisions (table 2). To gain an understanding of the situations in which the algorithm might be particularly helpful, we will also collect data on the indication for SEEG and the MDT members’ confidence in the ability of the SEEG implantation to identify a seizure onset zone. A transparent account of harms, including the number of electrodes implanted into ‘false positive’ lesions, will be reported, allowing a balanced assessment of risks and benefits prior to consideration of further stage testing. Steps taken to optimise the workflow will also be reported.
The risks and benefits of such an approach were carefully considered during the design phase. In order to evaluate the incremental benefit of the algorithm, the clinical standard of care could not be altered and, therefore, moving existing electrodes was not considered an option. The risk of complications per additional electrode is small.16 In addition, a group of 14 parents of children with epilepsy and one adolescent with epilepsy were surveyed at the Epilepsy in Childhood: Carers Uniting with Researchers Information Day. All 15 (100%) agreed that the risk would be acceptable when balanced against the potential benefits, and all 15 (100%) would enrol in the trial were they/their child to undergo SEEG implantation.
Ethical approval has been received for this study from the UK Health Research Authority (IRAS ID 275480) and the study began recruitment in September 2020. Further details of the protocol are available at https://clinicaltrialsgov/ct2/show/NCT04383028. The study has been funded by EANS-Stryker Research Grant 2020 & The Rosetrees Trust (A2665), but the funders had no input into the design or conduct of the study. The study will be reported in according with relevant guidelines including IDEAL stage 1/2a, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) & Standards for reporting diagnostic accuracy (STARD) guidelines.11 17 18
IDEAL stage 2b and beyond
We see the MELD algorithm as an additional part of the presurgical evaluation, which should be available to clinicians before the SEEG electrode trajectories are planned. The current algorithm was developed using single-centre data, with the clinical study being conducted at the same centre. The first step to further generalise the applicability would be to construct a large multicentre data set that accounts for variability in subjects, scan strength and protocols across centres. This is currently being constructed as part of the MELD study.19
If this should progress beyond the current stage 1/2a study, we envision conducting a stage 2b study as a feasibility randomised trial in a small number of centres, which assesses the proportion of patients in whom a SOZ is identified on SEEG with and without the benefit of the algorithm. This would allow us to optimise stability of the algorithm across centres, train other surgeons to integrate it into their clinical workflow, establish estimated effect sizes and assess feasibility of conducting a larger randomised study across multiple centres as an IDEAL stage 3 study.
Analysis of the stage 2b or 3 studies may also result in the additional findings, such as a select group of patients (eg, with interictal positron emission tomography (PET), ictal single-photo emission computed tomography (SPECT) or magnetoencephalography (MEG) concordance with the top identified lesions) in whom it may be debated that SEEG is not necessary prior to resection. If such findings were identified, the subsequent prospective clinical evaluation would have to go through the rigorous process of evaluation from stage 0 again as it would be using the algorithm in a different context to the present evaluation. However, many aspects may overlap with the current evaluation, making the process more streamlined.
