MedCalc-Bench Doesn't Measure What You Think: A Benchmark Audit and the Case for Open-Book Evaluation

This paper audits and corrects errors in the MedCalc-Bench dataset, demonstrating that providing models with calculator specifications ("open-book" prompting) significantly outperforms current state-of-the-art methods, thereby revealing that the benchmark primarily measures formula memorization and arithmetic precision rather than clinical reasoning.

The Gist

The Big Idea: The "Open-Book" Surprise

Imagine you are taking a difficult math test in a hospital. The test asks you to calculate a patient's risk score based on their blood work.

The Old Way (Closed-Book):
For years, researchers have been testing AI doctors by giving them these problems without letting them look up the formulas. It's like asking a student to memorize the entire periodic table and then solve complex chemistry equations on the spot.

• The Result: Even the smartest AI models were failing miserably, getting only about 35% of the answers right. The leaderboard looked like a disaster zone.

The New Discovery:
This paper argues that the test was unfair. In real life, doctors don't memorize formulas; they use calculators or apps. They look up the rules, plug in the numbers, and get the answer.

The authors tried a simple trick: They gave the AI the "cheat sheet" (the formula and rules) right in the prompt.

• The Result: Suddenly, the AI's score jumped from 35% to over 85%. It didn't need to be retrained or upgraded; it just needed to be allowed to "open the book."

---

The Three Main Findings (The "Plot Twist")

1. The Test Was Broken (The "Typos in the Answer Key")

Before testing the AI, the authors audited the test itself. They found that the "official" calculator code used to grade the answers was full of mistakes.

• The Analogy: Imagine a teacher grading a math test, but the answer key has typos. If the student writes "10" but the key says "100" because of a typo, the student gets marked wrong.
• What they found: They fixed over 20 errors in the test code. Some formulas had the wrong numbers, some had missing steps, and some had broken file paths. The "gold standard" wasn't actually gold; it was rusty.

2. The "Open-Book" Intervention

The authors realized the AI wasn't bad at clinical reasoning (figuring out what the patient needs); it was just bad at memorizing the specific math formula for that one test.

• The Analogy: It's like asking a chef to bake a cake without a recipe. They might know how to bake, but if they forget the exact ratio of sugar to flour, the cake fails. If you hand them the recipe card, they bake a perfect cake every time.
• The Outcome: By simply pasting the recipe (the calculator specification) into the AI's instructions, the AI outperformed all the complex, expensive, "reinforcement learning" systems that had been trained for months.

3. The Ceiling is Much Higher

The authors then asked: "If we give the AI the recipe, fix the broken answer key, and use the smartest AI available, how perfect can it get?"

• The Result: They found that the AI could get 95–97% of the answers right. The few mistakes left weren't because the AI was "dumb"; they were because the patient's story was vague or the test data itself was confusing.

---

Why Does This Matter? (The "So What?")

The paper concludes that MedCalc-Bench isn't actually testing if an AI can be a good doctor.

• What it was testing: Can the AI memorize a specific math equation and do decimal math perfectly without making a tiny rounding error?
• What it should test: Can the AI read a patient's chart, find the right numbers (like blood pressure or age), and use a tool to calculate the risk?

The Final Verdict:
The authors suggest we stop treating these calculators as a memory test. Instead, we should treat them as a tool-use test.

• Old Mindset: "Show me you know the formula."
• New Mindset: "Here is the formula. Now, look at this patient's chart, find the right numbers, and tell me the result."

The "Solo Researcher" Superpower

One last interesting note: The entire study was done by one person over a few weekends.

• The Analogy: In the past, auditing a dataset like this would have required a team of 10 people working for months.
• The Secret: The researcher used a "swarm" of different AI models to do the heavy lifting. One AI wrote the code, another checked the medical facts, another searched the internet, and another wrote the paper. It's like having a full research lab in your pocket for the price of a coffee.

Summary

The paper says: "Stop testing AI on memory. Start testing them on how well they use tools. And by the way, the test we were using was broken, and we fixed it."

Technical Summary

1. Problem Statement

MedCalc-Bench is a widely used benchmark designed to evaluate Large Language Models (LLMs) on clinical calculator tasks (e.g., APACHE-II, CKD-EPI, MELD). The task requires a model to read a clinical vignette, identify the relevant calculator, extract parameters, recall the specific formula, and compute the result.

Despite its importance, the benchmark has shown poor performance:

• State-of-the-art (SOTA) direct prompting scores plateau around 34.8%.
• Even advanced approaches using Reinforcement Learning (RL) with verifiable rewards or agentic tool frameworks only reach 61–74%.

The authors hypothesize that the benchmark's difficulty stems not from a lack of clinical reasoning, but from an artificial constraint: requiring models to memorize complex formulas from parametric memory and perform precise multi-step arithmetic (logarithms, exponents) without external aids. In real clinical practice, physicians use calculators rather than memorizing formulas.

2. Methodology

The authors employed a three-pronged approach: Benchmark Auditing, Open-Book Prompting, and Upper-Bound Analysis.

A. Benchmark Audit

The team conducted a systematic review of all 55 calculator implementations in the official MedCalc-Bench repository.

• Process: Cross-referenced implementations against primary clinical sources (original publications, MDCalc, guidelines) using automated testing, manual review, and LLM-assisted code analysis.
• Findings: Identified and fixed >20 errors, including:
  • Logic/Formula Errors: Incorrect coefficients (e.g., CKD-EPI, MELD-Na), missing scoring criteria (SIRS, SOFA), and unit conversion errors.
  • Runtime Bugs: Typos in variable keys (e.g., liver_diease), broken file paths, and argument mismatches.
  • Impact: Many errors propagated to the ground-truth labels, meaning the "correct" answers in the dataset were often mathematically incorrect based on the intended clinical rules.

B. Open-Book Prompting Intervention

The authors proposed a minimal intervention: Open-Book Prompting.

• Concept: Instead of forcing the model to recall the formula, the calculator specification (formula, parameter definitions, scoring rules, and unit expectations) is appended to the prompt.
• Rationale: This shifts the task from "Formula Recall + Arithmetic" to "Parameter Extraction + Tool Application," mirroring real-world clinical workflows.
• Implementation: Tested on GLM-4.6V and GLM-4.7 using three prompt variants:
  1. Vanilla: Original prompt (no formula).
  2. Open-Book: Prompt + Full calculator specification.
  3. Open-Book + Guidance: Specification + Explicit step-by-step extraction/computation instructions.

C. Upper-Bound Analysis

To determine the theoretical limit of the task, the authors used GPT-5.2-Thinking (with "xhigh" reasoning effort, search, and tools) on the subset of 198 rows where the mid-range models failed under open-book prompting. This helped distinguish between model limitations and dataset ambiguities.

3. Key Results

A. Performance Leap with Open-Book Prompting

Providing the calculator specification drastically improved performance without any fine-tuning or RL training:

• GLM-4.6V: Accuracy jumped from 51.9% (Vanilla) to 81.5% (Open-Book).
• GLM-4.7: Accuracy jumped from 36.0% (Vanilla) to 85.5% (Open-Book).
• Comparison: These results surpass all published SOTA results, including RL-trained systems (DeepSeek-R1 MedCalc-R1 at 74%) and agentic frameworks (RiskAgent at 67.7%).
• Insight: The gain comes almost entirely from eliminating the "Formula Recall" step; adding structured guidance provided only marginal benefits (+0.5%).

B. Error Analysis

• Vanilla Failure Modes: Dominated by arithmetic errors on real-valued operations (logarithms/exponents) and formula version conflicts (e.g., MELD vs. MELD 3.0).
• Open-Book Failure Modes: Shifted to parameter extraction (parsing the vignette correctly) and residual arithmetic precision issues. Formula recall errors were effectively eliminated.

C. Upper-Bound Estimates

Using GPT-5.2-Thinking on the residual errors:

• The model resolved 70.7% of the remaining errors.
• Conservative Upper Bound: 94.7% (assuming the model solves all rows the weaker model solved).
• Optimistic Upper Bound: 97.4% (accounting for ground-truth errors and ambiguities identified during the audit).
• Conclusion: The task is solvable by current frontier models if the "memorization" constraint is removed.

4. Significance and Contributions

1. Benchmark Audit: The paper exposes critical flaws in the MedCalc-Bench dataset, including over 20 implementation errors that corrupt ground-truth labels. The authors are releasing corrected implementations.
2. Redefining the Metric: The authors argue that MedCalc-Bench currently measures formula memorization and arithmetic precision rather than clinical reasoning. The difficulty is an artifact of the "closed-book" constraint.
3. Tool-Use Paradigm: The paper advocates for reframing clinical calculator benchmarks as Tool-Use evaluations. The core capability to test should be parameter extraction and calculator selection, not formula recall.
4. Efficiency: The study demonstrates that a simple prompt-level intervention (Open-Book) outperforms expensive, compute-intensive methods like RL training and extended thinking, making high-performance evaluation more accessible.
5. AI-Assisted Research Workflow: The paper documents a novel research workflow where a single researcher leveraged a "multi-model ecosystem" (using cheap models for bulk evaluation and frontier models for auditing/reasoning) to conduct a large-scale benchmark audit in a fraction of the time required for traditional manual methods.

Conclusion

MedCalc-Bench is "hard" for the wrong reasons. By removing the artificial constraint of formula memorization, LLM performance on clinical calculators skyrockets to near-perfect levels. The authors conclude that the benchmark should be repositioned to evaluate clinical parameter extraction and tool utilization, aligning the evaluation metric with actual clinical practice.