Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation

1. Problem Statement

Medication reconciliation at clinical handoffs is a high-stakes process where errors (dropped medications, duplications) can lead to adverse events. While Large Language Models (LLMs) are increasingly proposed to automate this task using FHIR (Fast Healthcare Interoperability Resources) structured records, a critical variable remains unstudied: how FHIR data is serialized (formatted) before being passed to the model.

Standard FHIR records are deeply nested JSON structures containing numeric ontology codes (RxNorm, SNOMED) rather than readable text. A single patient's record can exceed 800,000 lines of JSON, far exceeding the context windows of most models. Prior work focused on selecting relevant resources, but this paper investigates whether the presentation format of those selected resources significantly impacts model performance in extracting active medications.

2. Methodology

The study employed a controlled benchmark involving synthetic data, multiple open-weight models, and four distinct serialization strategies.

• Dataset:
  • Generated 200 synthetic patients using Synthea, an open-source simulator producing medically realistic FHIR R4 records.
  • Ground Truth: Defined as the set of distinct medication names from all MedicationRequest resources with status = "active".
  • Complexity: Patients had 1–16 active medications (median 5) and up to 25 years of history.
• Serialization Strategies (The Independent Variable):
  1. Raw JSON (Strategy A): Indented JSON with clinical fields intact. Constraint: Capped at 100 resources for smaller models to fit context windows (active medications prioritized).
  2. Markdown Table (Strategy B): Flattened into a 6-column table (Medication, RxNorm, Status, Date, Dose, Frequency), sorted chronologically.
  3. Clinical Narrative (Strategy C): Plain English sentences. Active medications listed first under a specific heading, followed by historical ones. Explicitly signals the task-relevant partition.
  4. Chronological Timeline (Strategy D): Pipe-delimited lines sorted strictly by date (oldest to newest), mixing active and historical statuses without explicit grouping.
• Models Evaluated:
  • Phi-3.5-mini (3.8B)
  • Mistral-7B (7B)
  • BioMistral-7B (7B): A domain-pretrained model without instruction tuning.
  • Llama-3.1-8B (8B)
  • Llama-3.3-70B (70B)
  • Setup: All models served locally via Ollama on an AWS g6e.xlarge instance (NVIDIA L40S, 48 GB VRAM) using 4-bit quantization.
• Evaluation Metrics:
  • Precision, Recall, F1: Calculated using exact string matching against ground truth medication names.
  • Statistical Analysis: Wilcoxon signed-rank tests with Bonferroni correction.
  • Total Runs: 4,000 inference runs (200 patients × 4 strategies × 5 models).

3. Key Contributions

1. Systematic Benchmark: The first controlled comparison of FHIR serialization strategies across varying model sizes, demonstrating that format choice is a performance-critical variable, not just a cosmetic one.
2. Model-Size-Dependent Optimization: Identified that the optimal serialization strategy flips based on model scale (Narrative for small models, JSON for large models).
3. Error Mode Characterization: Established that omission (missing a medication) is the dominant failure mode over hallucination (fabricating a medication) across all configurations.
4. Capacity Ceiling Discovery: Demonstrated a hard performance ceiling for smaller models (7B–8B) when handling polypharmacy patients (>7–10 active meds).
5. Instruction Tuning Necessity: Provided empirical evidence that domain pretraining alone (BioMistral) is insufficient for structured extraction tasks without instruction tuning.

4. Key Results

A. Serialization Strategy Effect

• Small Models (≤8B): Clinical Narrative (Strategy C) significantly outperformed Raw JSON.
  • Mistral-7B: F1 increased from 0.7247 (JSON) to 0.9149 (Narrative). This is a 19-point gain with a large effect size ($r=0.617, p < 10^{-10}$).
  • Llama-3.1-8B: F1 increased from 0.9180 to 0.9471.
• Large Model (70B): Raw JSON (Strategy A) became the best performer.
  • Llama-3.3-70B: Achieved a mean F1 of 0.9956 on Raw JSON, slightly outperforming Narrative (0.9850).
  • Note: The 70B model struggled significantly with the Chronological Timeline (F1 = 0.8742), suggesting that while large models handle JSON natively, they still struggle with complex temporal reasoning in unstructured timelines.

B. Omission vs. Hallucination

• In all 20 model/strategy combinations, Mean Precision > Mean Recall.
• Models are more likely to omit an active medication than to hallucinate a non-existent one.
• Clinical Implication: Safety auditing should focus on "completeness checks" (what did the model miss?) rather than just "accuracy checks" (is the output correct?).

C. Capacity Ceiling & Polypharmacy

• Small Models: Recall degrades sharply as the number of active medications increases.
  • Mistral-7B recall dropped from 0.96 (1 active med) to 0.24 (11 active meds).
  • This creates a systematic underservice for polypharmacy patients, who are the highest-risk group for reconciliation errors.
• Large Model: Llama-3.3-70B maintained near-perfect recall even with 16 active medications.
• History Length: The length of the total medication history (years) did not predict failure; only the count of concurrent active medications did.

D. BioMistral Failure

• BioMistral-7B (domain-pretrained but not instruction-tuned) achieved F1 = 0.0000 across all strategies.
• It failed to follow instructions, producing garbled text or repeating the system prompt. This proves that domain knowledge alone is insufficient; instruction tuning is a prerequisite for structured extraction tasks.

5. Significance and Practical Recommendations

• Deployment Strategy:
  • For 7B–8B Models: Use Clinical Narrative serialization. It provides a massive performance boost (up to +19 F1 points) with zero additional training cost, simply by changing the preprocessing format.
  • For 70B+ Models: Use Raw JSON. These models can natively parse the complex structure, and JSON offers a marginal edge over narrative formats.
• Safety Auditing: Clinical workflows should be designed to treat model output as a "high-confidence subset" rather than a complete list. Human review should prioritize identifying omissions rather than verifying every listed item.
• Model Selection: For complex patients (polypharmacy), model scale is the limiting factor. Small models (7B–8B) cannot reliably handle patients with >10 active medications regardless of the serialization format.
• Instruction Tuning: Do not use base models or domain-pretrained models without instruction tuning for structured clinical extraction tasks, even if they possess medical vocabulary.

6. Limitations

• Synthetic Data: Results are based on Synthea-generated data, which may be cleaner than real-world EHRs (which contain transcription errors and inconsistent formatting).
• Exact Matching: The evaluation used strict exact string matching. In production, fuzzy matching (e.g., RxNorm normalization) would likely improve scores, though the relative ranking of strategies would likely remain similar.
• Single Prompt: The study used a single prompt template; interactions between prompt phrasing and serialization format were not fully ablated.

Conclusion

The paper concludes that serialization strategy is a critical deployment variable. For the majority of current clinical deployments using 7B–8B models, converting FHIR data into a Clinical Narrative format is the most effective, evidence-based approach to maximize medication reconciliation accuracy. For massive models, raw JSON is sufficient, but the risk of omission remains the primary safety concern across all scales.