💬 NLP

HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats

The paper introduces HealthBench Professional, a rigorous open benchmark comprising physician-authored conversations and expert-adjudicated rubrics to evaluate and track the progress of large language models on real-world clinical tasks, demonstrating that the specialized GPT-5.4 model outperforms both base models and human physicians.

Original authors: Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K. Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, Akshay Jagadeesh, Khaled Saab, Johannes

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rebecca Soskin Hicks, Mikhail Trofimov, Dominick Lim, Rahul K. Arora, Foivos Tsimpourlas, Preston Bowman, Michael Sharman, Chi Tong, Kavin Karthik, Arnav Dugar, Akshay Jagadeesh, Khaled Saab, Johannes Heidecke, Ashley Alexander, Nate Gross, Karan Singhal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but sometimes overconfident, robot assistant how to help doctors. You want to know if the robot can actually help a doctor save a life, write a patient note, or find the latest medical research, or if it just sounds good but gets the details wrong.

This paper introduces HealthBench Professional, which is essentially a giant, real-world "driver's test" for AI assistants, but specifically designed for doctors.

Here is how the paper explains it, broken down into simple concepts:

1. The Problem: Old Tests Were Too Easy (or Fake)

Think of previous AI tests like a multiple-choice quiz where the answers are obvious, or a scripted role-play where the robot knows exactly what the "patient" is going to say.

The Issue: Real doctors don't talk in multiple-choice questions. They have messy, multi-turn conversations. They ask for help with difficult cases, they need notes written quickly, and they need to find specific research papers.
The Result: The old tests were like giving a robot a test on "how to drive in a parking lot," but then putting it on a real highway. The robot passed the parking lot test but crashed on the highway. Also, the smartest robots had already memorized the answers to the old tests, so the tests were "saturated" (useless for measuring improvement).

2. The Solution: A Real-World Simulation

The authors built a new test using 525 real conversations that actual doctors had with an AI tool called "ChatGPT for Clinicians."

The "Good Faith" Drivers: Some doctors just used the AI normally during their work (like a driver commuting to work). These represent everyday tasks.
The "Red Team" Drivers: Other doctors were hired to try to break the AI. They tried to trick it, confuse it, or ask it dangerous questions to see if it would fail. This is like a driving instructor intentionally throwing obstacles in the road to see if the robot can handle a crisis.

3. The Grading System: The "Human Referee"

In many AI tests, a computer grades the computer. In this paper, real doctors graded the AI.

The Process:
1. A doctor creates a conversation and writes a "rubric" (a checklist of what a perfect answer looks like).
2. Other doctors review that checklist to make sure it's fair and accurate.
3. A final group of doctors acts as the "referees" to resolve any arguments.
The Analogy: Imagine a sports game where the referees are all former professional players. They don't just look at the score; they look at how the play was made. They check for safety, accuracy, and clarity.

4. The Three Main Drills

The test focuses on the three things doctors actually do with AI:

Care Consult: "I have a patient with these weird symptoms. What could it be, and how do I treat it?" (Reasoning).
Writing & Documentation: "Here is a messy transcript of a patient visit; please turn it into a clean medical note." (Writing).
Medical Research: "Find me the latest studies on this specific drug interaction." (Searching).

5. The "Length Penalty" Rule

The authors noticed a trick: If an AI just talks a lot, it can accidentally get a higher score because it has more chances to say the right thing.

The Fix: They added a "length penalty." It's like a writing contest where if you write a 50-page essay to answer a simple question, you lose points for being wordy. They adjusted the scores so that concise, high-quality answers are rewarded, not just long rambling ones.

6. The Results: Who Won?

The paper compares different AI models and even real human doctors.

The Human Baseline: They hired expert doctors to answer the same questions. They gave them unlimited time and access to the internet. This is the "Gold Standard."
The Winner: The best-performing system was GPT-5.4 inside the "ChatGPT for Clinicians" tool.
- It scored 59.0.
- The human doctors scored 43.7.
- The standard version of GPT-5.4 (without the special doctor tools) scored 48.1.
The Takeaway: The specialized AI tool didn't just beat other AI models; it actually outperformed the human doctors in this specific test environment. The paper notes this is likely because the AI had instant access to all medical guidelines and could process information faster than a human could read and synthesize it in a test setting.

7. Why This Matters

The authors say this test is designed to be hard. They intentionally picked the hardest questions (the "Red Team" ones) to make sure the test doesn't get "too easy" for future, smarter robots.

The Goal: To give the healthcare community a reliable ruler to measure if AI is actually getting better at helping doctors, rather than just getting better at passing fake exams.

In short: The paper built a tough, real-world driving test for AI, graded by expert doctors, and found that a specialized AI tool is currently driving better than the human drivers in this specific simulation.

1. Problem Statement

While Large Language Models (LLMs) are increasingly used by millions of clinicians for real-time reasoning, documentation, and research, existing evaluation benchmarks fail to capture the complexity of these interactions.

Limitations of Current Benchmarks: Most existing health benchmarks rely on synthetic data, single-turn multiple-choice questions (e.g., MedQA), or structured patient vignettes. These do not reflect the multi-turn, open-ended nature of real clinician workflows.
Saturation: Many benchmarks are already saturated by frontier models, offering little signal for measuring future progress.
Lack of Trustworthiness: Evaluations often suffer from annotation noise or lack rigorous verification by multiple domain experts, making it difficult to distinguish between model failures and evaluation flaws.

Goal: Create a high-quality, unsaturated, and trustworthy benchmark that evaluates LLMs on the specific, real-world tasks clinicians perform daily.

2. Methodology

A. Data Collection & Construction

The benchmark consists of 525 physician-authored tasks derived from real interactions with "ChatGPT for Clinicians."

Sources: Data comes from two modes:
1. Good Faith (GF): Routine clinical, academic, or administrative use.
2. Red Teaming (RT): Deliberate adversarial stress-testing (e.g., false premises, emotional framing, role-play) to expose failure modes.
Three-Stage Adjudication Process: Every example and its grading rubric underwent a rigorous three-phase review by three or more physicians:
1. Authorship: A physician creates the conversation, writes an initial rubric, and assigns a difficulty rating (Likert 1–7).
2. Review: Independent physicians verify realism, difficulty, and rubric quality. "Difficult" examples (Likert 1–2) require dual verification of model errors.
3. Final Adjudication: Ambiguities are resolved, and rubrics are sharpened to ensure objectivity and alignment with clinical standards.
Stratified Sampling: From an initial pool of 15,079 examples, the final 525 were selected using difficulty-based stratified sampling. This enriched the proportion of "difficult" examples (Likert 1–2) by roughly 3.5x relative to the candidate pool, ensuring the benchmark remains challenging for frontier models.

B. Benchmark Structure

The tasks are categorized into three primary use cases central to clinical practice:

Care Consult: Reasoning through differentials, management, and treatment plans.
Writing and Documentation: Note generation, summarization, medical coding, and patient messaging.
Medical Research: Synthesizing evidence for clinical or scientific questions.

C. Scoring Mechanism

Rubric-Based Evaluation: Each example has a specific rubric with criteria ranging from specific facts to behavioral attributes (e.g., safety, uncertainty handling). Criteria are assigned point values between -10 and +10.
Model-Based Grading: A model (GPT-5.4) acts as the grader, evaluating whether the response meets each criterion.
Length Adjustment: To prevent models from gaming the system by generating excessively long responses, a length penalty is applied. The score is adjusted linearly based on response length (penalizing responses >2,000 characters and rewarding shorter, concise ones within a reasonable range).
Human Baseline: A strong baseline was established by collecting responses from specialty-matched physicians with unbounded time and web access, representing the "ideal" human performance.

3. Key Contributions

HealthBench Professional Benchmark: A novel, open benchmark of 525 real-world clinician tasks, moving beyond synthetic data to actual clinician-model conversations.
Rigorous Quality Control: A three-phase physician adjudication process ensures high trustworthiness, filtering out unrealistic examples and subjective criteria.
Adversarial & Difficulty Stratification: The deliberate inclusion of red-teaming data and the up-sampling of difficult examples (3.5x enrichment) create a benchmark that is not yet saturated, allowing for the measurement of future model progress.
Strong Human Baseline: The collection of unbounded, specialist-matched physician responses provides a rigorous upper-bound reference for model performance.
Length-Adjusted Scoring: A novel scoring adjustment that decouples performance from response verbosity, focusing on information density and utility.

4. Results

The paper evaluates OpenAI models (GPT-5 series), competitors (Claude, Gemini, Grok), and the human physician baseline.

Overall Performance: The GPT-5.4 model within the "ChatGPT for Clinicians" harness achieved the highest score (59.0), significantly outperforming:
- Human Physicians: 43.7 ( $p = 3.7 \times 10^{-10}$ ).
- Base GPT-5.4: 48.1 ( $p = 1.3 \times 10^{-7}$ ).
- Other Frontier Models: All other evaluated models scored lower.
Performance by Use Case:
- Writing & Documentation: The system showed the largest margin over humans (64.1 vs. 32.1) and other models.
- Care Consult: Outperformed humans (51.0 vs. 42.7) but was statistically similar to base GPT-5.4.
- Medical Research: Significantly outperformed humans (67.0 vs. 56.3).
Performance by Difficulty:
- On Red Teaming (Adversarial) Difficult cases, the system scored 55.8, vastly outperforming humans (30.0) and base models (26.2).
- On Good Faith Typical cases, the system remained ahead of humans but was statistically indistinguishable from some other strong models after correction.
System vs. Model: The "ChatGPT for Clinicians" harness (which includes tool use for evidence retrieval) significantly boosted performance over the base GPT-5.4 model, particularly in documentation and research tasks.
Reasoning Effort: Increasing reasoning effort (test-time compute) consistently improved scores, suggesting that models can utilize additional compute to produce higher-quality, not just longer, responses.

5. Significance and Implications

Real-World Relevance: Unlike exam-based benchmarks, HealthBench Professional measures performance on tasks that directly impact patient care delivery, documentation efficiency, and clinical decision-making.
Trust and Safety: The rigorous physician adjudication and adversarial testing provide a more reliable signal for safety and robustness, crucial for deploying AI in healthcare.
Future-Proofing: By intentionally enriching difficult examples, the benchmark avoids saturation, serving as a long-term metric for tracking the progress of frontier models in healthcare.
Clinical Utility: The results suggest that AI systems, when properly harnessed and evaluated, can surpass human physicians in specific high-volume tasks (like documentation and research synthesis), potentially alleviating clinician burnout and improving care quality.

Conclusion: HealthBench Professional establishes a new standard for evaluating LLMs in healthcare, shifting the focus from static knowledge recall to dynamic, multi-turn clinical reasoning and workflow integration. It demonstrates that current frontier models, when integrated with appropriate tools and safety guardrails, can outperform human physicians in specific professional contexts.