Technical Summary: Capabilities of Claude Fable 5 on Biomedical Challenge Problems

Problem Statement

The evaluation of frontier language models in biomedicine is currently undermined by two critical issues:

1. Benchmark Saturation: Legacy benchmarks (e.g., USMLE-style exams) have reached near-ceiling performance, failing to discriminate between state-of-the-art systems.
2. Unreliable Grading: Open-ended responses are frequently evaluated by other language models, introducing the judge's own biases and failure modes into the results.

Furthermore, standard evaluation metrics often conflate "refusal to answer" with "incorrect answers." In high-stakes domains like medicine, a model that declines to respond due to safety alignment is fundamentally different from a model that answers confidently but incorrectly. Treating these as identical outcomes obscures the true capability of a model and misrepresents its utility in clinical or research settings.

Methodology

This study evaluates Claude Fable 5 (Anthropic's most capable publicly available model as of mid-2026) against three baselines: its predecessors Claude Opus 4.6 and Opus 4.8, and GPT-5 (OpenAI).

Experimental Design:

• Benchmarks: Eight biomedical benchmarks were utilized, divided into text-only and multimodal categories:
  • Text: MedQA, PubMedQA, MedXpertQA (Text), and RareBench.
  • Multimodal: MedXpertQA MM, VQA-RAD, SLAKE, and PathVQA.
• Protocol: All evaluations were conducted under zero-shot, direct-answer prompting. No few-shot examples, chain-of-thought instructions, or retrieval augmentation were used.
• Scoring: A critical methodological constraint was the use of deterministic scoring against fixed answer keys. No benchmark relied on LLM-based judgment for grading.
• Refusal Tracking: Refusals were tracked as a distinct outcome. They were excluded from the denominator when calculating "scored-subset accuracy" (performance on items the model actually attempted) but included in "raw accuracy" to reflect total system behavior.
• Image Handling: Images were processed as base64 content blocks or data URIs, with multiple images in a single case submitted together.

Key Contributions

1. Comprehensive Evaluation: The first independent evaluation of Claude Fable 5's biomedical capabilities across eight diverse benchmarks.
2. Generational Trace: A controlled comparison of Fable 5 against Opus 4.6 and Opus 4.8 under identical protocols to isolate changes in capability versus alignment behavior.
3. Refusal-Centric Analysis: The introduction of refusal rates as a primary metric, separating safety-driven non-engagement from reasoning failures.
4. Benchmark Selection: The inclusion of MedXpertQA and RareBench, which remain challenging for frontier models, alongside legacy benchmarks to measure progress and saturation.

Results

1. The "Refusal Gap"

The central finding of the paper is that Claude Fable 5 exhibits a high rate of refusal (8.0% to 99.4% depending on the benchmark), a pattern absent in both its predecessors and GPT-5.

• Raw vs. Scored Accuracy: When refusals are counted as errors, Fable 5 often appears to underperform or regress compared to Opus 4.8 and GPT-5. However, when refusals are excluded from the denominator, Fable 5's accuracy meets or exceeds every other model on every benchmark.
• MedQA: Fable 5's raw score was 79.7% (17% refused), but its scored-subset accuracy was 96.6%, surpassing all other models.
• RareBench: Fable 5 refused 99.4% of items (1,115 of 1,122) under the primary clinical-role prompt. The few items it answered (6) yielded a Recall@1 of 0.36%, a metric the authors note reflects the refusal wall rather than diagnostic capability.

2. Distinct Refusal Patterns

The study identifies two distinct, non-unified patterns of refusal:

• Basic Science/Mechanism Concentration: On MedQA and MedXpertQA MM, refusals heavily concentrated on preclinical, basic-science content (e.g., Step 1 questions, anatomy, physiology, receptor pharmacology). On MedQA, 92.3% of refusals were Step 1 questions, despite Step 1 comprising only 45.1% of the answered questions.
• Disease Domain Specificity (RareBench): Refusals correlated strongly with the source dataset and disease domain. Inborn metabolic disease presentations (e.g., phenylketonuria, glutaric acidemia) were refused near-universally (99.7% in RAMEDIS), while adult-onset autoimmune presentations were answered at much higher rates.
• Prompt Framing: A "neutral framing" re-run on RareBench (removing the "clinical decision support system" persona) recovered only 9.3% of originally refused items, indicating that prompt engineering is not a reliable mitigation for this behavior.

3. Multimodal Performance

• MedXpertQA MM: Fable 5 achieved a scored-subset accuracy of 80.2%, decisively outperforming GPT-5 (67.2%) and its predecessors, with non-overlapping confidence intervals.
• PathVQA: Fable 5 refused 41.5% of items. On the items it answered, it achieved 29.3% accuracy, outperforming GPT-5 (5.3%) and Opus 4.6 (8.8%).
• VQA-RAD & SLAKE: Fable 5 led on VQA-RAD (45.8%) and was statistically indistinguishable from other models on SLAKE.

4. Generational Trends

The study concludes that the apparent "regression" in Fable 5's raw scores compared to Opus 4.6/4.8 is an artifact of increased refusal behavior, not a decline in underlying capability. Where Fable 5 engages, it demonstrates superior or equal reasoning capabilities.

Significance and Claims

The paper asserts that the primary constraint on Claude Fable 5's biomedical usefulness is willingness to engage, not competence once it engages.

• Misleading Raw Scores: Standard accuracy tables that fold refusals into "incorrect" results present a misleading narrative of capability decline. The authors argue that for deployment, the "scored-subset" accuracy is the more relevant metric for capability, while the refusal rate is the relevant metric for safety/alignment constraints.
• Safety vs. Utility: The refusal patterns observed (particularly on basic science and rare metabolic diseases) do not align with Anthropic's documented safeguards regarding dual-use content (e.g., pathogen design). This suggests either the classifier is over-triggering on benign content or the refusal mechanism is driven by an undocumented factor.
• Evaluation Rigor: The study emphasizes that future evaluations must track refusals as distinct outcomes and use deterministic scoring to avoid the "LLM-as-judge" bias, particularly as benchmarks move toward open-ended tasks.

The authors conclude that while Fable 5 is "selectively cautious," its underlying biomedical reasoning capabilities are among the strongest tested, provided the refusal barrier is navigated through case selection or fallback routing.