Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes

1. Problem Statement

The healthcare industry relies on the HL7 FHIR (Fast Healthcare Interoperability Resources) standard to ensure interoperability between complex health data systems. However, a significant bottleneck exists in converting unstructured free-form clinical notes (e.g., discharge letters, doctor's notes) into structured, schema-compliant FHIR resources.

• Limitations of Current Approaches: Previous attempts to automate this translation often rely on:
  • Modular pipelines (separate NER, relation extraction, and linking steps) that lack flexibility in complex clinical contexts.
  • Large Language Models (LLMs) with instruction tuning or constrained decoding, which frequently suffer from limited generalizability and structural non-conformity (i.e., generating JSON that violates FHIR schemas).
  • Retrieval-based approaches that incur high latency and computational costs for every query.
• The Challenge: Ensuring semantic correctness, strict adherence to the FHIR schema, and accurate mapping to external coding systems (like SNOMED CT) simultaneously is difficult for standard LLMs.

2. Methodology: The Infherno Framework

The authors propose Infherno, an end-to-end agentic framework that transforms unstructured text into FHIR resources using a Thought-Code-Observation loop (inspired by the ReAct framework).

Core Components

1. Agentic LLM Architecture:

   • The system uses an LLM agent that iteratively plans, acts, and observes.
   • It utilizes the Smolagents library to manage multi-step reasoning and Python code execution.
   • The agent does not output raw JSON directly; instead, it writes and executes Python code to construct FHIR objects.

2. Structured Data as Code (fhir.resources):

   • Instead of generating JSON strings, the agent uses the Python fhir.resources library to instantiate FHIR objects (e.g., Patient, Condition, MedicationStatement).
   • Benefit: This allows the system to catch morphological and syntactic errors immediately during the code execution phase, ensuring the final output is structurally valid before serialization.

3. Terminology Search & Tool Integration:

   • The agent has access to external tools to query SNOMED CT (Systematized Nomenclature of Medicine -- Clinical Terms) via a FHIR terminology server.
   • When the agent identifies a clinical concept (e.g., "vomiting"), it calls a tool to retrieve the correct SNOMED CT code (e.g., 422400008) and system URI, ensuring semantic accuracy and standard compliance.

4. Workflow:

   • Input: Unstructured clinical text (e.g., a discharge letter).
   • Reasoning: The agent analyzes the text, identifies entities (patients, conditions, medications), and determines necessary codes.
   • Tool Calls: It searches SNOMED CT for codes and executes Python code to build the objects.
   • Validation: The code execution environment validates the objects against the FHIR schema.
   • Output: The agent aggregates all resources into a FHIR Bundle (JSON) and returns it via a final_answer function.

5. User Interface:

   • A Gradio-based front end allows users to input text, view real-time agent reasoning (thoughts, tool calls, observations), and replay logs. It supports both local and API-based models (via LiteLLM).

3. Key Contributions

1. End-to-End Agentic Implementation: A novel framework for text-to-FHIR translation that integrates LLM agents, SNOMED CT terminology lookup, and schema validation via code execution, moving beyond simple prompt engineering.
2. Comprehensive Evaluation:
   • Evaluation on both real-world anonymized data (n2c2 2018 challenge) and synthetic data (German discharge letters).
   • Quantitative analysis (Precision, Recall, F1) and qualitative manual analysis comparing the agent against a Human Baseline (HB).
3. Open Demonstrator: A lightweight, accessible front-end application supporting custom and synthetic data, compatible with proprietary (Gemini, GPT) and open-source (DeepSeek, Qwen) models.

4. Results

The study evaluated multiple models (Gemini 2.5 Pro, GPT-5, Claude Sonnet 4.5, DeepSeek, Qwen) on two datasets.

• Performance on n2c2 Data (Real-world):
  • Gemini 2.5 Pro achieved the best results with an F1 score of 0.754 (Precision 1.0, Recall 0.605).
  • All models demonstrated high precision (near 1.0), indicating very few hallucinations or fabricated concepts.
  • Recall varied significantly, with smaller models (e.g., Qwen3-8B) struggling to extract all relevant information.
• Performance on Synthetic Data:
  • All models performed exceptionally well (F1 > 0.93), suggesting the framework is robust when ambiguity is low.
• Human Baseline Comparison (Manual Analysis):
  • In cases of quality divergence, the agent often outperformed the human annotator (better in nearly twice as many cases as worse).
  • The agent showed superior recall for clearly stated information that humans missed (e.g., inferring onset dates or capturing address fields).
  • Disagreements primarily occurred on secondary items (vague descriptions) or due to the subjective nature of FHIR encoding for minor health issues.
• Code vs. JSON: The code-based approach successfully prevented structural errors that plague pure JSON generation.

5. Significance and Future Work

• Significance:
  • Infherno addresses the critical gap in clinical data interoperability by automating the conversion of unstructured notes into standardized, queryable FHIR resources.
  • It demonstrates that agentic workflows with tool use and code execution are superior to static generation for complex, schema-constrained tasks in healthcare.
  • The system proves that LLMs can achieve higher consistency and recall than human annotators in repetitive, detail-oriented extraction tasks.
• Limitations:
  • Subjectivity: The human baseline relies on a single annotator, introducing subjectivity.
  • Scope: Currently limited to Patient, Condition, and MedicationStatement resources.
  • Licensing: Requires a SNOMED CT license for self-hosted terminology servers.
• Future Directions:
  • Fine-tuning smaller models specifically for text-to-FHIR tasks.
  • Expanding support to a broader range of FHIR resource types.
  • Evaluating on more diverse, multilingual datasets.

In conclusion, Infherno represents a robust, scalable solution for healthcare data integration, leveraging the reasoning capabilities of agentic LLMs to ensure both semantic accuracy and strict structural compliance with the FHIR standard.