Technical Summary: Note2Chat – Improving LLMs for Multi-Turn Clinical History Taking Using Medical Notes

Problem Statement

Effective clinical history taking is a foundational component of clinical reasoning, requiring clinicians to iteratively ask targeted questions, interpret responses, and refine diagnostic hypotheses. While Large Language Models (LLMs) have demonstrated proficiency in static, single-turn medical benchmarks (e.g., answering questions given a complete vignette), they struggle in dynamic, multi-turn diagnostic settings. Current LLMs often fail to generate focused follow-up questions, prioritize clinically relevant details, or adapt their reasoning based on patient responses. This limitation is exacerbated by the scarcity of high-quality, sensitive doctor-patient dialogue data required for training, which restricts the development of models capable of autonomous, exploratory history taking.

Methodology

The authors propose Note2Chat, a framework that leverages widely available, structured medical notes (specifically discharge notes containing the History of Present Illness and primary diagnosis) as a supervision signal to train LLMs for multi-turn history taking. The methodology consists of three core components:

1. Data Curation Pipeline

Instead of relying on scarce dialogue data, the authors convert real-world medical notes into synthetic doctor-patient dialogues through a three-step process:

• Finding Extraction: Relevant clinical findings are extracted from the History of Present Illness (HPI) section of discharge notes to construct patient vignettes, excluding downstream information (e.g., lab results) unknown during initial history taking.
• Decision Tree-Guided Generation: A decision tree maps findings to candidate diagnoses, providing a structured outline for an LLM to generate task-oriented dialogues that reflect realistic differential diagnosis workflows.
• Critic and Revision: An LLM-based critic evaluates generated dialogues to identify missing key findings or "context leakage" (where the doctor infers symptoms not yet revealed). The critic revises these dialogues to ensure comprehensive coverage and realism.
• Dataset: This pipeline yields a dataset of 8,944 synthetic dialogues across 4,972 patients covering 10 clinically relevant conditions (e.g., Heart Failure, Pneumonia, Cellulitis).

2. Three-Stage Fine-Tuning Strategy

The framework employs a progressive training strategy to enhance the base model (Qwen2.5-7B):

• Stage 1: Supervised Fine-Tuning (SFT): The model is trained on the note-guided dialogues to learn foundational clinical reasoning and dialogue structure.
• Stage 2: Self-Augmentation with Trajectory Sampling: To address the over-idealization of SFT data, the trained model engages in self-play with a simulated patient agent. Multiple dialogue trajectories are rolled out, and those achieving correct differential diagnoses with high recall (capturing the most documented findings) are selected to augment the training corpus. This introduces diverse and imperfect interaction dynamics.
• Stage 3: Direct Preference Optimization (DPO): The model is optimized to prefer concise, effective, and clinically sound conversations. Preference pairs are constructed by contrasting high-reward dialogues (high information recall, accurate diagnosis, efficiency) against low-reward ones. The reward function explicitly balances information gathering, diagnostic accuracy, and dialogue length.

3. Single-Turn Reasoning Paradigm

To overcome the limitations of long-horizon multi-turn training (e.g., error compounding and coarse supervision), the authors reframe history taking as a sequence of single-turn reasoning problems.

• Structure: Between each dialogue turn, the model generates a hidden reasoning block containing a Summary (of accumulated history) and a Plan (rationale for the next action).
• Advantages: This allows for local supervision at each turn, enabling the model to make context-aware decisions independent of long-term trajectory noise. It facilitates explicit reward shaping based on information gain and diagnostic relevance at every step, improving interpretability and sample efficiency.

Key Contributions

1. Note-Driven Dialogue Generation: A novel pipeline that converts medical notes into high-quality, clinically grounded training data, overcoming the bottleneck of scarce dialogue datasets.
2. Three-Stage Training Framework: A strategy combining SFT, self-augmented trajectory sampling, and DPO to produce models capable of proactive questioning and efficient information gathering.
3. Single-Turn Reasoning Paradigm: A reformulation of multi-turn history taking that enables fine-grained, interpretable supervision and dynamic adaptation, leading to more efficient dialogues.
4. Comprehensive Dataset: A curated dataset of 8,944 dialogues and 11,403 preference pairs across 4,972 patients.

Experimental Results

The authors evaluated Note2Chat against a diverse set of baselines, including proprietary models (GPT-4o, Gemini-2.5), open-source models (DeepSeek-R1, Qwen series), and domain-specific medical models (HuatuoGPT-o1, MedGemma).

• Performance Gains: Note2Chat-ST (Single-Turn) achieved a 46.1% F1 score and 70.0% Top-1 diagnostic accuracy, representing a +16.9 F1 and +21.0 Top-1 improvement over GPT-4o.
• Efficiency: While Note2Chat-MT (Multi-Turn) achieved the highest recall (55.4%), it required an average of 27.5 turns. Note2Chat-ST achieved superior overall performance (F1, Precision, Top-K accuracy) with significantly fewer turns (17.3 on average).
• Ablation Studies: Results confirmed that each stage of the training pipeline (SFT, Self-Augmentation, DPO) contributed incrementally to performance. SFT alone provided a baseline boost, but self-augmentation and DPO were critical for improving recall and diagnostic precision.
• Symptom Coverage: The model demonstrated significant improvements in recalling specific symptom categories (e.g., Onset, Radiation, Severity) compared to baselines, aligning more closely with structured clinical inquiry (SOCRATES mnemonic).
• Clinician Comparison: On a small held-out test set of 20 cases, the model's performance in information gathering and diagnosis was comparable to that of a practicing clinician.

Significance and Claims

The paper claims that Note2Chat addresses a critical gap in AI-assisted clinical reasoning by shifting the focus from optimizing solely for final diagnostic accuracy to prioritizing high-quality information gathering. By leveraging the abundance of medical notes rather than scarce dialogue data, the framework offers a scalable and generalizable solution for training LLMs in dynamic clinical environments.

The authors assert that their single-turn reasoning paradigm not only enhances interpretability and sample efficiency but also enables models to autonomously decide when to stop questioning and issue a diagnosis. The results suggest that with appropriate training strategies, LLMs can effectively support real-world history taking, potentially reducing cognitive load on clinicians and improving the completeness of diagnostic conversations, particularly in resource-constrained settings. The work emphasizes that effective clinical AI requires not just medical knowledge, but the conversational competence to iteratively explore and refine hypotheses.