FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

1. Problem Statement

The paper addresses the critical gap between patient access to Electronic Health Records (EHRs) and the ability to derive precise, trustworthy answers from them. While HL7 FHIR has standardized EHR data structures, current interfaces rely on keyword searches or dense document retrieval, which fail to answer specific, patient-centric questions (e.g., "Why is my heart racing?" vs. "tachycardia status").

Existing solutions face three major limitations:

• Retrieval-Augmented Generation (RAG) & LLMs: Directly feeding full EHR records into Large Language Models (LLMs) is computationally expensive due to context window limits, prone to hallucinations, and raises privacy concerns regarding data transmission.
• Text-to-SQL: Traditional approaches rely on SQL, which requires direct backend database access (inaccessible to patient-facing apps) and lacks interoperability across different hospital systems.
• Ambiguity: Patient language is often vague and non-technical, making it difficult for models to map intent to the strict, nested graph structure of FHIR resources.

2. Methodology

The authors propose a Text-to-FHIRPath paradigm. Instead of generating free-text answers, the system synthesizes executable FHIRPath queries (the native path-based query language for FHIR) which are then executed against the patient's data bundle to retrieve deterministic answers.

Dataset Construction (FHIRPath-QA)

• Source Data: Derived from MIMIC-IV on FHIR Demo, a real-world clinical dataset converted to FHIR R4 standards.
• Scale: The dataset contains 14,295 unique samples based on 100 distinct patients and 61 clinical question templates.
• Multi-Perspective Paraphrasing: To bridge the gap between medical and layperson language, the team used GPT-4o to generate two types of questions for each template:
  1. Clinician Perspective: Technical, precise terminology.
  2. Patient/Caregiver Perspective: Conversational, intuitive phrasing (e.g., removing specific IDs, using natural language).
• Validation: Queries were validated for syntactic correctness using the OctoFHIR (Rust-based) parser and semantically verified by executing them against curated patient bundles to ensure the answers matched the ground truth.
• Splits: The dataset is divided into training, validation, and test sets. Crucially, it includes holdout sets for:
  • Unseen Queries: New question templates not seen during training.
  • Unseen Resources: Queries requiring FHIR resource types (e.g., Location, MedicationAdministration) absent from the training data.

Experimental Setup

The authors compared two architectural paradigms:

1. Retrieval-First (Ingestion-based): The LLM ingests raw FHIR resources and generates a free-text answer.
2. Query-First (Text-to-FHIRPath): The LLM generates a FHIRPath query string, which is executed by an external engine to produce the answer.

They evaluated standard models (OpenAI o4-mini, 4.1-mini, 4o-mini, 4.1-nano) and tested the impact of Supervised Fine-Tuning (SFT) on the "Query-First" approach using the large-scale training corpus.

3. Key Contributions

1. FHIRPath-QA Dataset & Benchmark: The first open dataset pairing natural language questions (both clinician and patient phrasing) with validated, executable FHIRPath queries and answers derived from real clinical data.
2. Text-to-FHIRPath Paradigm: A shift from generative QA to symbolic query synthesis, offering determinism, safety (pre-execution validation), and data minimization (retrieving only specific data points).
3. Patient-Oriented Paraphrasing Pipeline: A strategy to translate clinical intents into layperson language, explicitly testing the model's ability to handle semantic ambiguity.
4. Empirical Analysis of SFT: Demonstrating that while base LLMs struggle with FHIRPath synthesis, supervised fine-tuning significantly improves performance, though it introduces specific generalization limits.

4. Results

Efficiency and Scalability

• Token Reduction: The Query-First approach reduced token usage by 391x compared to Retrieval-First (e.g., ~1,600 tokens vs. ~630,000 tokens per question using o4-mini).
• Stability: Query-First token usage was stable regardless of EHR size, whereas Retrieval-First usage scaled linearly with record length, leading to unpredictable costs and context window overflows.

Accuracy and Performance

• Base Models: Standard LLMs struggled, with no model achieving >42% accuracy on the benchmark.
  • Retrieval-First: Performed better on clinician phrasing (relying on lexical overlap with JSON keys) but failed on patient phrasing.
  • Query-First: Performed better on patient phrasing in base models but struggled with complex logic.
• Impact of Fine-Tuning (SFT):
  • SFT on the FHIRPath-QA Large corpus dramatically improved Query-First accuracy. For example, 4o-mini accuracy nearly tripled (from ~27% to ~80%) on novel paraphrases.
  • Models learned to generalize compositional mappings between clinical intent and FHIR schema.
• Generalization Limits:
  • Models excelled on unseen queries (new templates) but degraded on unseen resource types (e.g., Location), suggesting overfitting to specific training schemas.
  • The "Patient Ceiling": Post-SFT, models performed significantly better on clinician phrasing than patient phrasing. The inherent ambiguity and lower semantic similarity of layperson language impose a performance ceiling that single-turn systems struggle to overcome.

Error Analysis

Common failure modes included:

• Resource Profile Misunderstanding: Referencing non-existent elements (e.g., trying to find time data in a Location resource).
• Semantic Ambiguity: Confusing MedicationRequest (prescriptions) with MedicationAdministration (hospital administration).
• Date Filter Mismatch: Correctly identifying resources but miscalculating relative dates (e.g., "last year").

5. Significance and Conclusion

The paper establishes that Text-to-FHIRPath is a viable, safer, and more efficient alternative to RAG for patient-facing EHR applications. By shifting the reasoning burden from the LLM to a deterministic query engine, the system mitigates hallucinations and privacy risks.

However, the study highlights a fundamental tension: while SFT enables models to master FHIRPath syntax, translating vague patient intent into precise clinical logic remains a bottleneck. The authors conclude that future systems may require multi-turn clarification strategies or schema-aware grounding to fully bridge the gap between informal patient language and structured EHR data.

The release of the FHIRPath-QA dataset and generation code provides a critical foundation for developing interoperable, safe, and scalable consumer health applications.