Technical Summary: ClinPivot – Do Clinical Models Change Treatment Decisions?

Problem Statement

Current clinical foundation models are predominantly evaluated using factual or exam-style medical question answering (QA) benchmarks, such as MedQA and PubMedQA. These benchmarks reward models for recalling fixed disease-drug associations where the relevant facts and answer sets remain static. However, clinical treatment decisions are inherently dynamic; a drug appropriate for a disease may become unsafe or inappropriate when patient-specific constraints change (e.g., the emergence of a contraindicating comorbidity, an interacting medication, or a new allergy).

The paper identifies a critical gap: strong performance on static medical QA does not guarantee that a model can reliably update its treatment choices when the "action space" shifts due to new clinical constraints. The core research question is whether clinical models can pivot their decisions correctly when patient context changes, rather than merely recalling a pre-learned disease-drug association.

Methodology: The ClinPivot Benchmark

To address this gap, the authors introduce ClinPivot, an auditable benchmark designed to test "pivot-sensitive" decision-making.

• Construction: Examples are built from biomedical knowledge graph relations (specifically PrimeKG-style relations including disease, drug, phenotype, indication, contraindication, interaction, and off-label links).
• Task Structure: Each example presents a short patient vignette containing a "decision pivot"—a clinically meaningful change in patient state (e.g., a new contraindication). The model is given a candidate set of treatments and must select the gold treatment derived from graph constraints under the new state.
• Auditability: Unlike benchmarks relying on LLM-generated labels, ClinPivot labels are derived directly from graph operations. Each item stores the disease node, candidate drug nodes, pivot type, banned drugs, and supporting evidence triples. This allows for precise measurement of both decision accuracy and pivot sensitivity (the ability to avoid a treatment explicitly banned by the pivot).
• Pivot Families: The benchmark includes five pivot types:
  1. Contraindication (drug unsafe for a new condition).
  2. Comorbidity conflict.
  3. Drug-drug interaction.
  4. Allergy (synthetic but labeled).
  5. Off-label tradeoff (testing resistance to switching to a less supported option).

Experimental Setup

The authors evaluated models across three axes: standard medical QA, pivot-sensitive clinical decision making, and general capability retention.

• Models Evaluated:
  • Frontier Models: GPT-o1, Claude Sonnet 4.5, Qwen3.5-397B-A17B-FP8, DeepSeek R1, Qwen3-235B-A22B, and Llama-4-Maverick.
  • Controlled Training: Qwen3-4B and Qwen3-8B were fine-tuned to compare training signals.
• Training Conditions:
  • QA SFT: Fine-tuning on disease-drug facts rendered as standard QA.
  • Decision SFT: Fine-tuning on ClinPivot-style decision examples (action selection under constraints).
  • Mixed/Replay: Variants combining QA and Decision data, or adding "replay" of general instruction/reasoning examples (from the Dolma mix) to mitigate catastrophic forgetting.

Key Results

1. Medical QA Performance Does Not Predict Pivot Sensitivity

There is a significant disconnect between static knowledge and dynamic decision-making.

• Performance Gap: While frontier models achieved high accuracy on MedQA (93–96%) and PubMedQA (75–82%), their performance on ClinPivot was substantially lower (61–69%).
• Ranking Shifts: Model rankings changed between regimes. For instance, DeepSeek R1 had the second-highest MedQA score but the second-lowest ClinPivot score among evaluated models. Conversely, Claude Sonnet 4.5 and Qwen3.5 performed relatively better on ClinPivot despite lower MedQA scores.
• Conclusion: High QA accuracy is not a reliable proxy for the ability to update treatment decisions when constraints change.

2. Decision-Structured Supervision Outperforms QA-Only Supervision

Training format significantly impacts decision-making capabilities.

• Decision SFT vs. QA SFT: Fine-tuning with decision-structured examples (ClinPivot style) consistently improved ClinPivot scores more than QA-only fine-tuning.
  • For Qwen3-8B, Decision SFT improved ClinPivot accuracy by 8.42 points over the base model, whereas QA SFT actually reduced decision accuracy.
  • Decision SFT also improved pivot sensitivity (avoiding banned drugs) by over 20 points for Qwen3-8B.
• Mechanism: Decision examples train models to connect facts to actions under constraints, rather than just recalling associations.

3. The Retention Tradeoff and Replay Mitigation

Clinical decision training can degrade general assistant capabilities.

• Tradeoff: Decision SFT improved clinical metrics but reduced general retention scores (e.g., IFEval and average retention across ARC, TruthfulQA, etc.) by approximately 6–8 points.
• Replay Solution: Adding a lightweight replay of general instruction and reasoning examples (equal in size to clinical data) during training mitigated this loss.
  • Decision SFT + Replay retained most clinical gains (70.12% decision accuracy vs. 70.90% for pure Decision SFT) while recovering general retention (64.87% vs. 59.83% for pure Decision SFT).

Significance and Claims

The paper claims that ClinPivot isolates a specific, critical capability: the ability of a model to update treatment decisions when structured patient constraints change. The authors emphasize that:

1. Current Benchmarks are Insufficient: Standard medical QA benchmarks fail to capture the conditional nature of clinical reasoning.
2. Training Matters: The format of supervision is crucial; decision-structured data is more effective for clinical decision tasks than QA-style data, even with matched knowledge budgets.
3. Balancing Act: While clinical decision training improves domain-specific reasoning, it risks eroding general assistant reliability. However, this tradeoff can be managed through lightweight replay strategies.

The authors explicitly state that ClinPivot is a research benchmark, not a certification of clinical safety. The labels are graph-derived and synthetic, intended to test model behavior rather than replace guideline-based evaluation or certify safety for patient care.