LLM StructCore: Schema-Guided Reasoning Condensation and Deterministic Compilation

1. Problem Statement

The paper addresses the challenge of automatically filling Case Report Forms (CRFs) from unstructured clinical notes, specifically for the CL4Health 2026 Dyspnea CRF task.

• The Task: Map unstructured clinical notes (in English and Italian) to a strict, 134-item schema.
• Key Challenges:
  • Extreme Sparsity: The median document contains only ~12 known items out of 134; the majority are "unknown."
  • Strict Scoring: The evaluation metric (Macro-F1) heavily penalizes both False Positives (FP) (predicting a value where none exists) and False Negatives (FN) (missing a present value).
  • Output Instability: End-to-end LLMs often fail due to format drift (invalid JSON) or "hallucinated" predictions that increase FPs.
  • Cost: High inference costs for large models and the risk of data privacy issues when sending patient data to external APIs.

2. Methodology: The Two-Stage Pipeline

The authors propose LLM StructCore, a contract-driven, two-stage architecture that decouples clinical information extraction from strict schema enforcement.

Stage 1: Schema-Guided Reasoning (SGR) Condensation

• Goal: Condense the clinical note into a stable, sparse, machine-readable summary.
• Mechanism: Instead of asking the LLM to predict all 134 fields directly, it is prompted to generate a JSON object with exactly 9 domain keys:
  1. Demographics
  2. Vitals
  3. Labs
  4. Problems
  5. Symptoms
  6. Medications
  7. Procedures
  8. Utilization
  9. Disposition
• SGR Patterns:
  • Cascade: The model follows a fixed sequence of these 9 domains to ensure uniform coverage.
  • Cycle: Prompts include implicit checklists (e.g., specific vital signs, 28 specific problems) to improve recall on sparse items.
  • Routing: The LLM outputs free-text strings within the JSON keys, deferring strict vocabulary normalization to Stage 2.
• Stability: The prompt is designed to achieve 100% JSON parse success even on quantized 4B models (e.g., MedGemma) without requiring complex grammar-constrained decoding (like XGrammar).
• Multi-Slice Strategy: For long notes, the input is split into overlapping windows, summarized independently, and merged (last-writer-wins for single facts, concatenation for lists) to stay within context limits.

Stage 2: Deterministic Compiler (0-LLM)

• Goal: Transform the Stage 1 JSON summary into the final 134-item submission format.
• Mechanism: A fully deterministic, rule-based Python compiler (no LLM calls).
• 4-Step Algorithm:
  1. KV Extraction: Parses "Key: Value" lines from the Stage 1 text.
  2. Canonicalization: Normalizes item names using exact matching, case-folding, abbreviation expansion (70+ rules), and an optional UMLS alias map (134/134 coverage).
  3. FP-Gates & Derivation:
     • Derivation: Infers composite items (e.g., "poly-pharmacological therapy" if ≥8 meds are listed).
     • Gating: Strict regex-based filters drop unsupported positive predictions for high-risk items (e.g., blocking "active neoplasia" unless specific cancer activity cues are present; blocking "arrhythmia" if "Normal Sinus Rhythm" is mentioned).
  4. Vocabulary Normalization: Maps extracted values to the official 13-category controlled vocabulary (e.g., converting "105 bpm" to "tachycardic").
  5. Expansion: Fills all 134 items, marking absent ones as "unknown."

3. Key Contributions

1. SGR Decomposition: Shifting the burden of structural correctness from the LLM to code. By reducing the target schema from 134 fields to 9 stable keys, the system achieves 100% JSON stability even on small, local models.
2. Deterministic Compilation: Replacing the second LLM pass with a rule-based compiler ensures bit-for-bit reproducibility, zero inference cost, and perfect adherence to the controlled vocabulary.
3. Edge AI Readiness: The pipeline enables privacy-preserving deployment on local hardware (e.g., Apple M3 Pro) using quantized 4B models, avoiding the need to send patient data to external APIs.
4. Language Agnosticism: The system works for both English and Italian without language-specific engineering in Stage 1. The deterministic Stage 2 handles multilingual variations via the UMLS map.

4. Experimental Results

• Datasets: Used the official CL4Health 2026 splits (Train: 10, Dev: 80, Test: 200).
• Teacher Models: Tested Mistral Large 3 (675B), Qwen 3.5 (397B), and Claude Sonnet 4.6.
• Performance (Dev80):
  • English: Mistral Large 3 + Deterministic Stage 2 achieved Macro-F1 0.6543.
  • Italian: Same configuration achieved Macro-F1 0.6905 (outperforming English, likely due to more structured Italian source notes).
  • Student Model: A local MedGemma 4B model achieved F1 0.6521 on Train10 with very low FPs, proving the viability of edge deployment.
• Test200 (Codabench):
  • The submitted English variant achieved F1 0.63, placing within 0.05 of the top-ranking system (Aurum, F1 0.68).
  • The Italian variant (F1 0.6905 on Dev) was prepared but not submitted due to attempt limits.
• Error Analysis:
  • FP Reduction: The FP-gate mechanism reduced False Positives for high-risk items by ~70% (e.g., reducing "active neoplasia" FPs from 18 to 4).
  • Bottleneck: >85% of errors originated in Stage 1 (missing facts), confirming that the deterministic Stage 2 is not the performance bottleneck.

5. Significance and Conclusion

The paper demonstrates that Schema-Guided Reasoning combined with Deterministic Compilation is a highly effective strategy for clinical NLP tasks characterized by extreme sparsity and strict output contracts.

• Reliability: It solves the "hallucination" and format drift problems common in end-to-end LLM extraction.
• Efficiency: It allows the use of smaller, cheaper, and privacy-friendly local models for the extraction phase while maintaining high accuracy.
• Reproducibility: The deterministic nature of Stage 2 ensures that results are fully reproducible and explainable, a critical requirement for clinical applications.

The authors conclude that this approach is competitive with top-tier proprietary systems while offering superior control, cost-efficiency, and deployment flexibility for edge environments. Future work will focus on improving Stage 1 recall for rare conditions and exploring hybrid approaches for context-dependent items.