Technical Summary: NEURON – A Neuro-symbolic System for Grounded Clinical Explainability

1. Problem Statement

The adoption of Artificial Intelligence (AI) in clinical settings is hindered by the "black-box" nature of high-performing machine learning (ML) models. While these models offer accurate risk estimates crucial for care prioritization and intervention, their reasoning remains opaque to clinicians with diverse expertise. Existing Explainable AI (XAI) techniques, such as SHAP (SHapley Additive exPlanations), provide static feature-attribution visualizations that lack ontological grounding and narrative transparency. These methods often fail to incorporate established clinical knowledge or support interactive, causal reasoning, limiting their utility for professional-level sensemaking and trust. Furthermore, current approaches often treat ontology-grounded Retrieval-Augmented Generation (RAG) and neuro-symbolic modeling in isolation, without a unified framework that simultaneously enhances predictive performance and generates clinically grounded natural language explanations.

2. Methodology: The NEURON Framework

The authors propose NEURON (Neuro-symbolic Understanding with Retrieval Oriented Narration), a unified neuro-symbolic framework designed to bridge the gap between raw data, medical nomenclature, and human-aligned explanation. The system operates through three primary components:

A. Neuro-Symbolic Predictive Modeling

NEURON integrates SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) ontology-informed structural representations with standard sub-symbolic ML models.

• Data Source: The framework was validated on the MIMIC-IV dataset, specifically focusing on Acute Heart Failure (AHF) in-hospital mortality prediction.
• Feature Engineering:
  • Tabular Baseline: 18 clinical features (e.g., vital signs, lab results) selected via LASSO regression.
  • Ontology Integration: Instead of treating SNOMED codes as one-hot features, the system constructs an "is-a" knowledge graph. It generates Node2Vec embeddings for SNOMED concepts and aggregates them using TF-IDF-weighted pooling to create fixed-length vectors per admission.
  • Ontology Counts: Explicit counts of ontology categories (Level 1 and Level 2) are derived for each admission.
• Model Architecture: The framework was tested across four distinct ML architectures: XGBoost, Multi-Layer Perceptron (MLP), Autoencoder, and RBF-SVM. These models were trained under three configurations: (1) Tabular only, (2) Tabular + SNOMED embeddings, and (3) Full Neuro-symbolic (Tabular + Embeddings + Ontology counts).

B. RAG-Grounded Explanation Generation

To translate model outputs into clinician-friendly narratives, NEURON employs a Retrieval-Augmented Generation (RAG) layer grounded in a Large Language Model (LLM).

• Input Sources: The RAG layer synthesizes four distinct information streams:
  1. SHAP Drivers: Feature attribution scores from the predictive model.
  2. Ontology Representations: Structural mappings from the SNOMED knowledge graph.
  3. Clinical Notes: Unstructured, patient-specific ICU notes retrieved via metadata filtering.
  4. Curated Clinical Knowledge: A manually curated knowledge base containing thresholds, reference ranges, and interpretive guidelines for the 18 predictors.
• Generation Process: The system uses a constrained prompt template to enforce a strict structure (e.g., separating model drivers, tabular features, and ontology evidence). It utilizes a local LLM (qwen2.5:14b-instruct) to generate natural language explanations, ensuring the output is grounded in retrieved evidence to minimize hallucinations.

C. Evaluation Metrics

The study employs a dual-evaluation strategy:

• Predictive Performance: Measured via AUC, Accuracy, Sensitivity, Precision, F1-score, and Specificity.
• Explanation Quality:
  • Computational Metrics for SHAP: Completeness, Additivity Gap, Infidelity, Max-Sensitivity, and Robustness.
  • Human-Aligned Metrics for Narratives: Evaluated using an LLM-as-a-judge approach (using gpt-oss:20b) to assess Faithfulness, Plausibility, Usefulness, Sensemaking, and Coherence. Narrative completeness is also measured via "Explingo-based" coverage of feature names, values, and directions.

3. Key Contributions

1. Unified Neuro-Symbolic Framework: NEURON is the first study to present a controlled comparison of sub-symbolic and neuro-symbolic pipelines on the same clinical cohort, demonstrating that ontology integration improves predictive performance across multiple model families.
2. Integrated Explanation Pipeline: The system uniquely couples SHAP attribution directly with a RAG-grounded LLM, combining structured ontology representations, unstructured clinical notes, and curated knowledge to generate holistic narratives.
3. Comprehensive Evaluation: The paper provides a rigorous comparison between static SHAP visualizations and dynamic narrative explanations using both computational attribution metrics and human-aligned criteria (faithfulness, sensemaking).
4. Ontology-Aware Embedding Strategy: The authors introduce a non-trivial method for integrating SNOMED CT, utilizing Node2Vec embeddings and TF-IDF pooling rather than simple one-hot encoding, to better capture clinical relationships.

4. Results

Predictive Performance

The integration of ontology-derived features consistently improved predictive performance across all four ML models.

• AUC Improvement: The best-performing models (XGBoost and RBF-SVM) under the full neuro-symbolic configuration achieved an AUC of 0.88 (95% CI: 0.86–0.89), a significant increase from the tabular-only baseline (0.74–0.77).
• Sensitivity and F1: The neuro-symbolic configuration yielded the strongest improvements in sensitivity (e.g., XGBoost increased from 0.21 to 0.37) and F1-score (XGBoost increased from 0.31 to 0.49), indicating better detection of high-risk cases.

Explanation Quality

• SHAP Performance: SHAP explanations demonstrated high completeness (0.932) and low additivity gaps, making them suitable for regulatory traceability. However, they scored poorly on human-aligned metrics, with an overall score of 0.504 and a sensemaking score of 0.307, indicating difficulty for clinicians to interpret static visualizations.
• NEURON (RAG) Performance: The narrative explanations generated by NEURON achieved substantially higher human-aligned scores (Overall: 0.849; Sensemaking: 0.827).
• Coverage: The LLM narratives successfully covered 76.3% of the top-K SHAP attribution mass, demonstrating that the natural language output effectively synthesized the model's drivers.
• Stability: The narrative generation showed high run-to-run stability (Cosine similarity of 0.999), whereas SHAP attributions showed moderate stability (0.777).

5. Significance and Claims

The authors claim that NEURON offers a robust, scalable engineering solution for deploying trustworthy, human-centered connected health applications. The significance of the work lies in its ability to:

• Enhance Predictive Reliability: By grounding ML models in clinical ontologies, the system improves risk stratification accuracy without sacrificing the flexibility of sub-symbolic learning.
• Facilitate Clinical Sensemaking: The system moves beyond static feature attribution to provide coherent, natural-language narratives that align with clinical workflows and knowledge.
• Bridge the Gap: It successfully bridges the gap between raw data and medical nomenclature, offering a mechanism for interactive and narrative reasoning that is currently missing in standard XAI.

The authors acknowledge limitations, including the potential for subject-level overlap in the dataset, the dependency of narrative quality on the specific LLM used (with smaller models exhibiting hallucinations), and the need for careful prompt engineering to prevent the generation of unsupported clinical claims. They conclude that while NEURON's narratives outperform SHAP in human-aligned metrics, they should be viewed as complementary to existing XAI techniques rather than a complete replacement, as they still rely on the underlying model's fidelity.