Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

1. Problem Statement

The Brazilian Unified Health System (SUS) serves over 200 million citizens and relies on 178 official Clinical Protocols and Therapeutic Guidelines (PCDTs) to dictate diagnostic criteria, treatments, dosages, and monitoring procedures. However, current Large Language Models (LLMs) face significant challenges in this domain:

• Knowledge Gap: State-of-the-art medical LLMs are predominantly trained on English-language literature (e.g., PubMed, USMLE) and lack specific knowledge of Brazilian Portuguese clinical protocols.
• Evaluation Deficit: No existing benchmark evaluates clinical recall grounded in official Brazilian protocols.
• Accessibility: Healthcare professionals struggle to recall specific recommendations from the massive volume of complex documents at the point of care.

2. Methodology

The authors propose a comprehensive pipeline to adapt a general-purpose LLM to the Brazilian clinical domain using synthetic data generation, continual pre-training, and reinforcement learning.

A. Data Collection and Synthesis

• Source: 178 official PCDT documents (approx. 5.4M tokens).
• Synthetic Data Generation: Instead of training directly on the raw guidelines, the authors used four diverse generator LLMs (GPT-4.1-mini, GPT-5-nano, GPT-OSS-20B, and Qwen3-235B) to transform the guidelines into ~70M tokens of synthetic data in three formats:
  1. Rephrases: Paraphrased versions preserving all clinical content.
  2. Wiki-style Articles: Expository articles restructuring the protocols.
  3. Question-Answer (QA) Pairs: Detailed clinical questions with step-by-step rationales.
  • Note: To prevent data leakage, QA pairs were generated only from the training split of guidelines (89 guidelines), while rephrases and wiki articles were generated from all 178.

B. Training Pipeline

The base model used was Qwen2.5-14B-Instruct. The training involved two stages:

1. Continual Pre-Training (CPT): The model was trained on the ~70M token synthetic corpus using a standard causal language modeling objective.
2. Reinforcement Learning (RL): The authors applied Group Relative Policy Optimization (GRPO) using the training split of a new benchmark (HealthBench-BR).
   • Reward Signal: The model received a reward of 1.0 for correct True/False verdicts accompanied by at least 50 words of reasoning, and 0.0 otherwise. This encouraged explicit clinical reasoning and reduced "reward hacking" (guessing without justification).

C. Evaluation Benchmarks

Two new, protocol-grounded benchmarks were introduced:

1. HealthBench-BR: 1,780 balanced True/False assertions (10 per guideline). These test precise factual recall (e.g., specific dosages, administration routes).
2. PCDT-QA: 890 open-ended clinical questions scored by an LLM judge (GPT-4.1) against reference answers.

3. Key Contributions

1. Reproducible Training Protocol: A novel pipeline combining multi-generator synthetic data, continual pre-training, and GRPO for domain adaptation in a low-resource (relative to English) language setting.
2. New Benchmarks: Introduction of HealthBench-BR and PCDT-QA, the first benchmarks specifically designed to evaluate LLMs on official Brazilian clinical guidelines.
3. Open Release: Public release of all synthetic datasets, benchmarks, and model weights to support reproducible research in Brazilian Portuguese clinical NLP.
4. Ablation Insights: Detailed analysis showing that generator diversity and reinforcement learning are critical for performance, while full fine-tuning outperforms LoRA in this specific context.

4. Key Results

The best model configuration (CPT with 4 generators + GRPO) achieved state-of-the-art performance on both benchmarks, significantly outperforming much larger proprietary models and retrieval-augmented systems.

• Performance Metrics:
  • HealthBench-BR: 83.9% accuracy (Test set).
  • PCDT-QA: 85.4% accuracy (Test set).
• Comparisons:
  • Outperformed GPT-5.2 (high), Claude Sonnet 4.6, and Gemini 3.1 Pro by 4.6 to 5.4 percentage points.
  • Surpassed Google AI Overview (a web-grounded RAG system) by 13.4 points on HealthBench-BR and 8.1 points on PCDT-QA, despite the RAG system having access to the open web.
• Ablation Findings:
  • Generator Diversity: Scaling from 1 to 4 generators improved PCDT-QA performance by 20 points (from 66.3% to 86.3%), highlighting the importance of stylistic and lexical diversity.
  • Reinforcement Learning: GRPO provided a 12.8 point gain on HealthBench-BR over CPT alone and corrected "sycophancy" (the tendency to agree with users), reducing the agreement bias from 58.9% to 53.0%.
  • Forgetting: Full fine-tuning (Full FT) was found to be superior to LoRA for this task, as LoRA resulted in higher catastrophic forgetting on out-of-distribution tasks (e.g., instruction following).

5. Significance

• Domain Specificity: This work demonstrates that a relatively small, open-source model (14B parameters) can outperform massive proprietary models when specifically adapted to a country's official medical protocols.
• Equity in Healthcare AI: It addresses the "English-centric" bias in medical AI, providing a pathway for high-quality clinical decision support for Portuguese-speaking populations.
• Safety and Reliability: The study highlights the dangers of "sycophancy" in medical AI (agreeing with false premises) and shows how RL can mitigate this, a crucial step for deploying AI in clinical settings.
• Resource Efficiency: The results suggest that targeted knowledge injection via synthetic data is more effective than relying on general web retrieval (RAG) for specific, high-stakes clinical guidelines.

The paper concludes that while the model is a research artifact and not a certified medical device, it represents a significant leap forward in making clinical AI accessible and accurate for the Brazilian public health system.