📄 medical education

Infoxmed2.0-27B: Instruction Tuning, Preference Alignment, and GRPO-Based Reward Model Training for Medical LLMs

The paper introduces Infoxmed2.0-27B, a medical foundation model derived from Qwen3.5-27B that achieves superior performance on medical benchmarks through a rigorous multi-stage post-training pipeline involving proprietary data synthesis, LoRA-based instruction tuning, DPO preference alignment, and GRPO-based reward model training.

Original authors: Xie, J., Guo, Z., Zhao, H., Ni, H.

Published 2026-06-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xie, J., Guo, Z., Zhao, H., Ni, H.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Picture: Building a "Medical Super-Intern"

Imagine you have a brilliant, well-read university student (the Qwen3.5-27B base model). This student knows a lot about the world, can write essays, and understands general logic. However, if you ask them a complex medical question, they might guess, make up facts, or give advice that sounds good but is actually dangerous.

The authors of this paper wanted to turn this "general student" into a "medical super-intern" who is accurate, safe, and follows strict medical rules. They did this not by teaching them a whole new degree from scratch, but by running them through a rigorous four-stage training camp.

Stage 1: The Textbook Factory (Data Synthesis)

Before the student can study, they need the right textbooks.

The Problem: You can't just scrape random medical advice off the internet; it's full of errors and duplicates.
The Solution: The team built a custom factory.
1. Mining: They dug through a private, organized database (like a massive library) to pull out structured questions and answers.
2. Sorting: They organized these questions into a "Medical Category Tree" (like a filing cabinet with folders for Oncology, Cardiology, etc.) so the student doesn't get overwhelmed.
3. The Editors: A team of Medical PhDs acted as strict editors. They checked every answer for accuracy. If a fact was wrong, it was tossed.
4. The Filter: They used a smart AI tool (Chinese RoBERTa) to find and remove duplicate questions, ensuring the student learns from unique examples.
5. Polishing: Finally, they used an API to smooth out the language, making the answers sound natural and clear without changing the medical facts.

Analogy: It's like taking a messy pile of handwritten notes, having a professor verify every fact, removing the photocopies, and then hiring a professional editor to make the text readable.

Stage 2: The Classroom (Instruction Fine-Tuning / SFT)

Now that the student has the textbooks, they go to class.

The Method: They used a technique called LoRA (Low-Rank Adaptation).
The Analogy: Imagine the student already has a full brain. Instead of rewriting their entire brain (which is expensive and slow), the team added a small, specialized "notebook" (LoRA) to their pocket. This notebook contains specific medical rules and patterns.
The Result: The student learned how to follow medical instructions and structure their answers correctly. They went through three versions of this class (v0.0, v0.2, v0.4), getting better each time.

Stage 3: The Debate Coach (Preference Alignment / DPO)

Knowing the facts isn't enough; the student needs to know how to choose the best answer when multiple options exist.

The Setup: The team created 6,283 "decks" of cards. Each deck had two answers to the same medical question:
- The Winner: A perfect, detailed, evidence-based answer.
- The Loser: A vague, incomplete, or hallucinated (made-up) answer.
The Training: They used a method called DPO (Direct Preference Optimization). Instead of hiring a judge to grade every answer (which is slow), they simply showed the student: "Here is a good answer, here is a bad one. Learn to prefer the good one."
The Process: They ran this training 8 times (v0 to v7), tweaking the "strictness" of the coach each time. They picked the version (v7) that was best at spotting the difference between a good and bad answer.

Analogy: It's like a coach showing a chess player a game where they won and a game where they lost, then saying, "Don't play like you did in the losing game."

Stage 4: The Scorekeeper (Reward Model Training / GRPO)

While the student was learning to prefer good answers, the team also trained a separate "Scorekeeper" AI.

The Goal: This Scorekeeper's job is to grade the student's future answers automatically.
The Hybrid Approach: The Scorekeeper uses two types of grading:
1. Internal Rules: A checklist (e.g., "Did you use the right medical terms? Did you cite evidence?").
2. External Wisdom: It asks a powerful outside AI (DeepSeek) for a general "gut feeling" on the quality.
The Technical Trick: The base model they used had a quirk where it expected two grades instead of one. The team cleverly adapted the system to work with this quirk rather than fighting it, subtracting the "bad" score from the "good" score to get a final grade.

Analogy: It's like a teacher who uses a strict rubric (Internal Rules) but also asks a visiting expert (External AI) for their opinion, then combines both to give a final grade.

The Final Exam (Results)

The team tested their new "Medical Super-Intern" (Infoxmed2.0-27B) against other top medical AI models using two exams:

MedMCQA (Multiple Choice):
- The model got 77% accuracy.
- It scored higher than its competitors (like Baichuan-M2 and MedGemma) in almost every category, especially in "Coverage" (covering all parts of the question) and "Structure" (writing clearly).
- The Progression: The base model started at a score of +6.69. After the classroom (SFT), it went to +7.06. After the debate coach (DPO), it reached +7.18.
HLE (Hard Logic Evaluation):
- This tested deep reasoning. The model scored +2.59.
- The Catch: While it was great at structure and safety, it actually got slightly worse at "Grounding" (sticking to facts) on very hard, out-of-domain questions compared to the base model. The paper admits this is a limitation: the model is very good at looking professional, but sometimes it still struggles with extremely obscure facts.

Summary

The paper claims that by combining expert-verified data, efficient classroom training, preference-based debate coaching, and a hybrid scorekeeper, they created a medical AI that is currently the best among its peers in terms of overall quality and structure, though it still has room to improve on avoiding "hallucinations" in very difficult scenarios.

Important Note: The paper explicitly states this is a research preprint and should not be used to guide clinical practice (real-life patient care) yet. It is a step forward in research, not a finished medical tool.

Technical Summary: Infoxmed2.0-27B

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities in general domains, their application in specialized medical contexts is hindered by a lack of domain-specific knowledge, reasoning patterns, and safety awareness. Existing medical LLM efforts often suffer from three primary limitations: (1) inconsistent training data quality, frequently relying on synthetic data without rigorous professional validation; (2) reliance on single-stage fine-tuning, missing the synergistic benefits of combining instruction tuning, preference alignment, and reward modeling; and (3) underexplored integration of reward models specifically trained for medical quality assessment. These gaps necessitate a systematic, multi-stage domain adaptation framework to ensure accuracy and safety in high-stakes clinical scenarios.

Methodology

The authors present Infoxmed2.0-27B, a medical foundation model derived from Qwen3.5-27B. The model is developed through a comprehensive, four-stage post-training pipeline:

1. High-Quality Medical Data Synthesis

To address data quality bottlenecks, the authors constructed a proprietary pipeline:

Source: Structured Q&A pairs were extracted from a MySQL database containing curated medical knowledge across subdomains (anatomy, pharmacology, pathology, etc.).
Organization: Data was structured using a hierarchical MedicalCategoryTree (discipline → subdiscipline → specific topic) to ensure balanced coverage.
Quality Assurance: A medical PhD team conducted audits for accuracy, logical completeness, and clinical boundary labeling. A Chinese RoBERTa model performed semantic deduplication (threshold 0.85).
Refinement: An API-assisted process optimized language organization and naturalness while strictly preserving medical content accuracy.

2. Instruction Supervised Fine-Tuning (SFT)

The Qwen3.5-27B base model was fine-tuned using LoRA (Low-Rank Adaptation) with rank $r=8$ and scaling factor $\alpha=32$ across all linear layers.

Framework: MS-Swift was used for training on 4× NVIDIA A800 80GB GPUs.
Process: The process involved iterative rounds (Infoxmed2.0.0 → 2.0.2 → 2.0.4) with systematic evaluation at each stage to refine hyperparameters and data quality.

3. Direct Preference Optimization (DPO) with RPO Regularization

To align model outputs with human preferences without explicit reward model training, the authors applied DPO on 6,283 curated medical preference pairs (derived from MedMCQA and HLE sources).

Data Construction: Pairs consisted of "chosen" responses (full reasoning, structured, evidence-cited) and "rejected" responses (vague, hallucinated, or incomplete).
Loss Function: The training utilized a DPO-RPO loss, combining preference optimization with SFT regularization ( $\alpha_{RPO} = 0.1$ ) to prevent excessive deviation from the SFT baseline.
Progressive Selection: Eight training iterations (v0–v7) were conducted. The final model was selected based on the checkpoint with the optimal chosen–rejected reward margin without rejected reward inflation.

4. GRPO-Based Reward Model Training

Parallel to DPO, a dedicated medical reward model was trained using Group Relative Policy Optimization (GRPO) on Qwen3.5.

Hybrid Reward Signal: The model combined internal rule-based rewards (checking field correctness, boolean operators, syntax) with external signals from DeepSeek (holistic quality assessment).
Architecture Adaptation: A specific technical challenge arose where the base model's configuration forced a num_labels=2 output despite a num_labels=1 setting. The authors adapted by extracting positive and negative scores from the dual outputs to compute the pairwise loss.
Outcome: The trained reward model achieved 46% pairwise accuracy, demonstrating functional discriminative capability.

Key Contributions

Synergistic Multi-Stage Pipeline: Unlike prior works that treat techniques in isolation, this paper proposes a cohesive framework integrating data synthesis, LoRA-SFT, DPO-RPO, and GRPO-based reward modeling.
Professionally Validated Data: The introduction of a PhD-guided data synthesis pipeline with semantic deduplication and API-assisted refinement addresses the critical issue of medical hallucination and factual errors in training data.
Advanced Alignment Techniques: The application of DPO with RPO regularization and the parallel training of a GRPO-based reward model specifically for medical quality assessment.
Technical Adaptation: Successful adaptation of the Qwen3.5 architecture to handle reward modeling constraints (specifically the num_labels discrepancy) through dual-output score extraction.

Experimental Results

The model was evaluated using an LLM-as-Judge framework (GPT-5.4) on two benchmarks: MedMCQA (200 questions) and HLE (Hard Logic Evaluation, 200 questions).

MedMCQA Performance:
- Accuracy: 77.0% (outperforming peers like Baichuan-M2-32B at 72.5% and MedGemma-27B at 69.0%).
- Quality Score: Achieved a mean score of +7.18, showing a clear progression from the base model (+6.69) and SFT stage (+7.06).
- Dimensions: Strong performance in Coverage (+8.70) and Expression & Structure (+8.25).
HLE Performance:
- Accuracy: 95.5%.
- Quality Score: Mean score of +2.59, leading peer models.
- Observation: While the model improved in expression and safety, the "Grounding" score was -3.08 (worse than the base model's -1.41), indicating persistent challenges with hallucination on out-of-domain questions.
Reward Model: The GRPO-trained reward model achieved 46% pairwise accuracy, validating the combined internal-external reward strategy.

Significance and Claims

The paper claims that Infoxmed2.0-27B comprehensively leads peer models (including Baichuan-M2-32B, MedGemma-27B, Lingshu-32B, and HuatuoGPT-3-32B) under identical evaluation conditions. The significance of the work lies in demonstrating that a systematic, multi-stage post-training pipeline—anchored by professionally validated data and advanced preference optimization techniques—can significantly enhance the medical reasoning and safety capabilities of general-purpose LLMs.

However, the authors maintain a modest tone regarding limitations:

The model still exhibits negative grounding scores on hard logic evaluation, suggesting that hallucination on out-of-domain questions remains a challenge.
The reward model's pairwise accuracy (46%) is functional but not perfect.
The evaluation is currently limited to two benchmarks (MedMCQA and HLE).
Safety boundaries, while improved, are not yet perfect.

The work serves as a proof-of-concept for integrating rigorous data curation with modern alignment techniques (DPO, GRPO) to build specialized medical AI, while acknowledging that further research is needed to fully resolve issues of medical hallucination and safety.