Training a Large Language Model for Medical Coding Using Privacy-Preserving Synthetic Clinical Data

1. Problem Statement

Medical coding (assigning ICD-10-CM and CPT codes to clinical documentation) is a critical but labor-intensive, error-prone process that directly impacts healthcare revenue, compliance, and secondary data quality. While Large Language Models (LLMs) have shown promise in medical text understanding, applying them to structured coding tasks faces significant hurdles:

• Zero-Shot Failure: Base foundation models perform poorly on coding tasks without specific adaptation, often generating invalid codes or hallucinations (exact-match F1 scores as low as 0.18).
• Data Privacy: Training on real Electronic Health Records (EHRs) poses severe privacy risks due to the potential memorization of Protected Health Information (PHI) by LLMs. De-identification is often insufficient to mitigate re-identification risks.
• Complexity: Coding requires navigating heterogeneous documentation styles, nuanced guidelines, long-tail distributions of rare codes, and multi-step clinical reasoning (e.g., for Advanced Illness or Frailty).

2. Methodology

The authors propose a framework to adapt a modern open-weight foundation model for expert-level coding using privacy-preserving synthetic data rather than real patient records.

A. Model Architecture and Training

• Base Model: Llama-3-70B-Instruct.
• Training Strategy: Supervised Fine-Tuning (SFT) on paired clinical notes and gold-standard codes.
• Output Constraint: The model is constrained to generate structured JSON outputs containing the predicted code, clinical rationale, and evidence localization to ensure syntactic validity.
• Infrastructure: Full-parameter fine-tuning using the ArcticTraining framework with ZeRO-3 optimization, FP16 precision, and Adam optimizer (learning rate 1e-5) across 8 H200 GPUs for 4 epochs.

B. Privacy-Preserving Synthetic Data Generation

To eliminate PHI exposure, the team developed a controlled pipeline generating clinically realistic but entirely synthetic data:

• ICD-10-CM Generation:
  1. Phase 1 (Synthesis): Real clinical documents were analyzed to extract abstract meta-descriptions (structure/style) while stripping patient details. A seed ICD-10 code was selected, and an LLM generated a synthetic clinical note conditioned on the meta-description and code set.
  2. Phase 2 (Labeling): Synthetic charts were segmented. A retrieval system identified candidate codes, and an LLM selected the most appropriate codes with explicit evidence attribution.
• CPT Generation: Similar to ICD-10 but focused on procedural notes. Seed CPT codes were sampled, and procedure notes were generated. To avoid hallucination, the system generated short procedural descriptions first, which were then matched against an indexed CPT catalog via cosine similarity to retrieve valid codes.
• Data Augmentation: 30% of the training set underwent difficulty-based augmentation where multiple clinical notes were concatenated to force the model to reason across multi-problem encounters.
• Sequence Packing: To improve efficiency, multiple note-code pairs were packed into single 8,192-token sequences, reducing padding overhead.

C. Evaluation

• Datasets: Held-out synthetic charts with gold-standard annotations.
• Metrics:
  • ICD-10-CM: Evaluated at 5 hierarchical levels (from coarse category to exact code with evidence localization). Metrics included Exact Match, Code-only Match, and Evidence Localization (Jaccard similarity).
  • CPT: Evaluated using exact set matching (Precision, Recall, F1).
  • Human Review: A subset of 100 charts was reviewed by clinical experts to establish a clinically grounded ground truth.
  • General Knowledge: Benchmarks (MedQA, MMLU, PubMedQA) were used to ensure fine-tuning did not degrade general medical reasoning.

3. Key Contributions

1. Privacy-First Training Pipeline: Demonstrated a viable method for training high-performance medical coding models using entirely synthetic data derived from EHR-grounded templates, effectively decoupling model training from PHI exposure.
2. Significant Performance Gains: Showed that fine-tuning on synthetic, policy-aware data can transform a general-purpose LLM into a specialized coding agent, bridging the gap between zero-shot failure and expert-level performance.
3. Hierarchical Stability: Proved that the model learns diagnostic intent across varying levels of granularity, maintaining high performance even as the task shifts from broad categorization to exact code assignment.
4. Preservation of General Knowledge: Validated that specialization for coding does not result in "catastrophic forgetting" of general medical knowledge.

4. Key Results

• Overall Performance:
  • ICD-10-CM: The fine-tuned model achieved an F1 score of 0.704 for exact code matching, a massive improvement over the zero-shot baseline of 0.180.
  • CPT: The fine-tuned model achieved an F1 score of 0.736, compared to a baseline of 0.193.
• Domain-Specific Performance:
  • Frailty & Advanced Illness: Highest performance (F1 > 0.86), likely due to explicit documentation patterns.
  • Social Determinants of Health (SDoH): Lower but still strong performance (F1 = 0.767), indicating the model can capture implicit social signals despite documentation inconsistencies.
• Hierarchical Analysis: Performance declined gradually with increasing specificity (from Category Level 0 to Exact Code Level 3) but remained stable, with no abrupt drops. The most stringent task (Exact Code + Evidence) achieved an F1 of 0.629.
• Human Expert Validation: When evaluated against expert-validated ground truth, the model showed high Recall (0.86–0.93) but lower Precision (0.32–0.40). This suggests the model is excellent at identifying relevant concepts but tends to over-generate candidate codes.
• General Knowledge: Fine-tuning resulted in only modest declines in general medical benchmarks (average drop of ~0.03), confirming the model retained its foundational reasoning capabilities.

5. Significance and Implications

• Practical Pathway for AI in Healthcare: This work provides a safe, scalable route to deploy LLMs in medical coding without the legal and ethical risks of training on raw patient data.
• Shift in AI Paradigm: The results suggest that LLMs should not be viewed as fully autonomous replacements for coders but as AI-augmented assistants. Their high recall makes them ideal for surfacing candidate codes and reducing cognitive load, while human experts handle final adjudication and exception handling.
• Policy-Aware Training: The study highlights that "policy-aware" synthetic data (incorporating coding guidelines and evidence linking) is crucial for teaching models the specific constraints of billing and compliance.
• Future Directions: The authors identify that while exact code selection remains a bottleneck (especially for implicit SDoH codes), the technology is mature enough for integration into human-in-the-loop workflows to improve efficiency and reduce denial rates.

In conclusion, the paper demonstrates that synthetic, policy-aware data is a powerful enabler for training specialized medical LLMs, achieving expert-level performance while rigorously protecting patient privacy.