💻 computer science

A Neurosymbolic Framework for Interpretable Skeleton-Based Seizure Detection via Concept-Driven Logical Reasoning

This paper introduces a novel neurosymbolic framework that achieves interpretable, high-accuracy video-based seizure detection by extracting patient-centric skeleton sequences, predicting clinically grounded spatio-temporal concepts, and composing them via differentiable logic into auditable Boolean rules while mitigating false positives through fine-grained sub-classification of non-seizure activities.

Original authors: Talha Ilyas, Deval Mehta, Zongyuan Ge

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Talha Ilyas, Deval Mehta, Zongyuan Ge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to spot a seizure in a video of a patient. Usually, deep learning computers are like genius but silent magicians: they can tell you "Seizure!" or "No Seizure!" with high accuracy, but they can't explain why. If you ask, "What made you say that?" they just shrug. In a hospital, doctors can't trust a black box; they need to know which specific movements triggered the alarm.

This paper introduces a new system that acts less like a silent magician and more like a detective with a checklist. Here is how it works, broken down into simple steps:

1. The "Skeleton" Translator

First, the system looks at a messy hospital video. The patient might be covered by blankets, or a nurse might be walking in front of the camera.

The Analogy: Imagine trying to follow a dance routine through a foggy window.
The Fix: The system uses a smart "tracker" (like a digital highlighter) to ignore the blankets and the nurses. It strips away everything except the patient's skeleton (a stick-figure outline of their joints). This turns a confusing video into a clean set of moving dots.

2. The "Concept" Vocabulary (The Checklist)

Instead of just guessing, the system is taught a specific vocabulary based on real medical rules (called ILAE guidelines).

The Analogy: Think of this like a grammar book for movement.
The Process: The system learns specific "concepts" or "words" of motion, such as:
- Spatial words: "Arm stiffening," "Head turning," "Leg jerking."
- Temporal words: "Rhythmic shaking," "Sudden stop."
The Innovation: Most systems just say "Seizure" or "Not Seizure." This system breaks "Not Seizure" down into specific, fine-grained categories like "Patient adjusting their pillow" or "Patient scratching an itch." This prevents the system from confusing a patient stretching with a seizure.

3. The "Logic" Engine (The Detective's Reasoning)

This is the core "Neurosymbolic" part. Once the system spots the "words" (concepts), it doesn't just guess; it uses logic rules to combine them.

The Analogy: Imagine a recipe or a flowchart.
How it works: The system builds rules like:
- IF (Arm is stiff) AND (Leg is jerking rhythmically) AND (Head is turning) THEN = Seizure.
The "Black Box" Breaker: Because these rules are written in clear logic (like math equations), the system can show its work. It can say, "I called this a seizure because 80% of my decision came from the 'Arm Stiffening' rule and 20% from the 'Rhythmic Jerking' rule."

4. The Results: Smarter and Clearer

The authors tested this on two public video datasets from hospitals.

Accuracy: It caught seizures better than almost every other method tested (about 90% sensitivity).
Fewer False Alarms: By teaching the system the difference between "adjusting a pillow" and "seizing," it made far fewer mistakes. It reduced false alarms by nearly 3.5 times compared to older methods that just saw "movement."
Transparency: For the first time, the system provides a full "audit trail." Doctors can see exactly which body parts moved, how those movements were logically combined, and how much each part contributed to the final decision.

Summary

In short, this paper presents a transparent, logic-based AI for spotting seizures. Instead of a mysterious black box that just guesses, it acts like a medical student who:

Draws a stick figure of the patient.
Checks a specific medical checklist for movement "words."
Follows a strict logical recipe to decide if it's a seizure.
Shows the doctor the exact steps it took to reach that conclusion.

The authors have made all their code, data, and the "stick figure" datasets available for others to use.

Technical Summary: A Neurosymbolic Framework for Interpretable Skeleton-Based Seizure Detection via Concept-Driven Logical Reasoning

Problem Statement

Video-based seizure detection is critical for managing epilepsy, offering a non-invasive alternative to electroencephalography (EEG). However, existing deep learning approaches for this task face two primary limitations:

Lack of Interpretability: Current state-of-the-art (SOTA) models function as "black boxes." In clinical epileptology, decisions rely on understanding specific motor semiology (observable signs like tonic posturing or clonic jerking) defined by the International League Against Epilepsy (ILAE). Models that detect seizures without explicating these underlying motor signs offer limited clinical value and hinder adoption.
High False Positive Rates: Most methods adopt a binary formulation (seizure vs. non-seizure). This collapses heterogeneous normal activities (e.g., a patient adjusting position) into a single negative class, causing the model to confuse movement-intensive non-seizure activities with seizures, leading to high false detection rates.

Methodology

The authors propose a neurosymbolic framework that integrates neural perception with symbolic reasoning to address these gaps. The framework operates in three stages:

1. Data Pre-processing and Fine-Grained Labeling

Patient-Centric Pose Extraction: To handle the unique challenges of Epilepsy Monitoring Unit (EMU) videos (e.g., supine patients, occlusions by bedding), the authors employ a two-stage foundation model pipeline. SAM 3 segments and tracks the patient, while Sapiens-2B extracts 133 COCO-WholeBody keypoints. These are mapped to a standard 25-joint NTU RGB+D format to leverage pretrained Graph Convolutional Network (GCN) backbones.
Fine-Grained Activity Sub-classification: Instead of a binary "non-seizure" label, the authors use a Vision-Language Model (Qwen3-VL) with clinical prompts to sub-classify non-seizure segments into 8 distinct normal activity categories. These are verified by human annotators. This provides fine-grained discriminative supervision to reduce false positives.

2. Concept Bank Construction

A unified concept bank ( $C$ ) is constructed, grounded in ILAE motor semiology and normal activity primitives:

Normal Activity Concepts: Derived from the NTU RGB+D 120 dataset, 74 concepts (53 spatial, 21 temporal) are generated by decomposing actions into body-part groups and using LLMs to create verb-direction-modifier patterns.
Seizure-Specific Concepts: 19 spatial concepts are derived directly from the ILAE glossary (e.g., arm_tonic_posture, head_versive_deviation), covering various seizure types (tonic, clonic, myoclonic, etc.).
Shared Temporal Concepts: The 21 temporal concepts from normal activities are shared across both regimes to capture motion dynamics.
Association Matrix: An LLM generates a binary association matrix linking actions to these concepts, refined to ensure discriminability.

3. Neurosymbolic Inference Framework

The core architecture processes skeleton sequences through four stages:

Spatio-Temporal Encoder: A GCN backbone (Hyper-GCN) extracts motion features from the skeleton sequence.
Concept Bottleneck: Features are projected to binary concept activations ( $\hat{c}$ ). The model is supervised to predict the presence of specific motion primitives (e.g., "arm stiffening") before making a final decision.
Differentiable Logic Layers: Stacked conjunction (AND) and disjunction (OR) layers compose the binary concept activations into Boolean rules. To enable training, discrete Boolean operations are replaced with continuous relaxations (using a Straight-Through Estimator) that allow gradient flow while maintaining discrete weights for inference.
Rule Aggregation: A linear layer aggregates the triggered rules into class logits. This produces a three-level audit trail:
- Concept Explanations: Which motor primitives were detected.
- Rule Explanations: How concepts were logically composed.
- Rule Contributions: The weight of each rule in the final decision.

Key Contributions

First Neurosymbolic Framework for Seizure Detection: The paper presents the first approach to video-based seizure detection that learns differentiable Boolean rules over ILAE-grounded motion concepts, directly bridging the gap between deep learning and clinical reasoning.
Fine-Grained Skeleton-Based Dataset: The authors augment two public benchmarks (SAHZU and IEEE) with 8 normal activity subclasses via a patient-focused pose extraction pipeline and VLM-assisted annotation, publicly releasing all annotations and code.
Empirical Validation of Fine-Grained Supervision: The study demonstrates that combining fine-grained normal activity supervision with neurosymbolic reasoning significantly reduces false positives on movement-intensive segments while preserving interpretability.

Experimental Results

The framework was evaluated on the SAHZU and IEEE Seizure Video datasets, comparing against 18 SOTA baselines (including I3D, R3D, and various GCN-based skeleton methods).

Performance:
- SAHZU: Achieved 89.78% sensitivity with 0.06 false detections per hour (FDR). This represents a +7.37% sensitivity gain and a 68% reduction in FDR compared to the strongest skeleton baseline (HyperGCN).
- IEEE: Achieved 85.27% sensitivity with 0.09 FDR, outperforming the best baseline by +5.74% sensitivity and 63% FDR reduction.
Ablation Studies:
- Pre-training: Pre-training on NTU-RGB+D 120 was essential; removing it caused an 11.16% drop in sensitivity.
- Multi-label Formulation: Switching from binary to multi-label (fine-grained) classification improved sensitivity by 6.47% and reduced FDR by 3.5×.
- Temporal Concepts: Including temporal concepts boosted sensitivity by 9.17 points compared to spatial-only concepts.
Interpretability: Unlike all other methods in the comparison, this framework provides full interpretability, decomposing every prediction into detected motor primitives, logical rules, and rule contributions.

Significance and Claims

The authors claim this work represents a significant step toward the clinical translation of AI in epilepsy management. By aligning the model's internal reasoning with established clinical guidelines (ILAE semiology), the framework offers auditable decision-making rather than opaque predictions. The ability to decompose a seizure detection into specific logical rules (e.g., "bilateral limb stiffening" AND "rhythmic jerking") allows clinicians to verify the model's reasoning against their own expertise. Furthermore, the reduction in false positives through fine-grained activity modeling addresses a major practical barrier to deploying automated seizure detection in real-world hospital settings. The authors position this as the first exploration of neurosymbolic AI in this specific domain, setting a foundation for future work in seizure sub-classification and broader clinical validation.