💬 NLP

DeVisE: Behavioral Testing of Medical Large Language Models

The paper introduces DeVisE, a behavioral testing framework that uses controlled counterfactual perturbations of demographic and vital sign attributes in ICU discharge notes to reveal how current medical large language models vary in their sensitivity and reasoning consistency, demonstrating that standard metrics often fail to capture clinically relevant behavioral differences.

Original authors: Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem, Iacer Calixto

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Camila Zurdo Tagliabue, Heloisa Oss Boll, Aykut Erdem, Erkut Erdem, Iacer Calixto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant new medical intern named "AI." This intern has read every medical textbook in existence and can recite symptoms and treatments faster than a human. But before you let them make real decisions about patients, you need to know: Do they actually understand medicine, or are they just memorizing patterns and guessing?

This is the problem the paper DeVisE tackles.

The Problem: The "Cheat Sheet" vs. True Understanding

Current tests for medical AI are like giving a student a multiple-choice quiz. They might get 90% right, but that doesn't tell us how they got there. Did they understand the heart rate, or did they just notice that "older patients" usually get "sicker" and guess based on that?

The authors wanted to see if the AI's reasoning is genuine or just superficial.

The Solution: The "What If?" Game

To test this, the researchers invented a game called DeVisE (Demographics and Vital signs Evaluation). Think of it as a "What If?" simulator for doctors.

Here is how they played the game:

The Original Patient: They took a real medical record from a hospital database (MIMIC-IV). Let's say, "Mr. Smith, 65 years old, heart rate 89 (normal)."
The Counterfactual (The Twist): They created a "twin" version of Mr. Smith where they changed only one tiny thing.
- Scenario A: Change the heart rate from 89 to 120 (high).
- Scenario B: Change the age from 65 to 25.
- Scenario C: Change the ethnicity.
The Test: They asked the AI: "What is the risk of death for the original Mr. Smith?" and then, "What is the risk for the twin Mr. Smith?"

The Rules of the Game

The researchers had two main rules for what a "smart" AI should do:

Rule 1: The Heartbeat Test (Vital Signs). If you change a patient's heart rate from normal to dangerously high, the AI should say, "Oh no, this patient is much sicker now!" The predicted risk of death should go up. If the AI says the risk stays the same, it's ignoring the medical reality.
Rule 2: The Demographic Test. If you change the patient's age or gender, the AI should adjust its prediction slightly based on real medical trends, but it shouldn't suddenly decide the patient is dying just because they are a different race or gender.

The Surprise Findings

The researchers tested 8 different "super-intelligent" AI models (some general ones like GPT, some medical-specific ones). Here is what they found:

1. The "Template" Trap

They tested the AI in two ways:

Raw Notes: Like reading a messy, handwritten doctor's note with lots of extra chatter.
Templates: Like a clean, fill-in-the-blank form with only the numbers.

The Result: When the AI read the clean templates, it became hyper-sensitive. It reacted wildly to small changes. It was like a car with the gas pedal stuck; a tiny bump in the road made it swerve. When reading the messy raw notes, the AI was calmer and more stable, likely because the extra context helped it "think" more like a human doctor who looks at the whole picture, not just one number.

2. The "Over-Reactors" vs. The "Under-Reactors"

Some models (like the medical-specific ones) were very conservative. Even when a patient's heart rate spiked, they barely changed their prediction. They were like a cautious doctor who needs to see a patient three times before changing their mind.
Other models (like the massive reasoning ones) were very sensitive. They changed their predictions drastically, sometimes too much. They were like a nervous doctor who panics at the first sign of a problem.

3. The Bias Problem

When they changed the patient's race or gender, the AI's predictions shifted in ways that mirrored real-world healthcare inequalities.

Example: Some models predicted that Black patients would stay in the hospital longer or have higher risks, even when their medical numbers were identical to White patients.
This proved that the AI had "learned" the biases present in the real world data, rather than just looking at the pure medical facts.

The Big Takeaway

The paper concludes that standard test scores are lying to us. An AI can get a high grade on a test but still fail the "What If?" test.

DeVisE is like a stress test for the AI's brain. It shows us:

Does the AI understand that a high heart rate is bad? (Yes, mostly).
Does the AI get confused by messy data? (Yes, sometimes).
Does the AI carry human biases? (Yes, unfortunately).

In short: Before we let AI doctors take over, we need to stop just asking them "What's the answer?" and start asking them, "What if I changed this one thing? Why did your answer change?" DeVisE is the tool that asks those questions.

1. Problem Statement

While Large Language Models (LLMs) are increasingly deployed in clinical decision support, standard evaluation benchmarks (e.g., accuracy, AUROC) fail to reveal whether models possess genuine medical reasoning or merely rely on superficial correlations, biases, and shortcuts.

The Gap: Conventional metrics do not assess if an LLM adjusts its predictions proportionally to clinically meaningful changes in patient data (e.g., if a patient's heart rate increases from normal to critical, does the model correctly predict a higher mortality risk?).
The Risk: Models may produce high accuracy scores while failing to understand the causal or correlational relationships between vital signs/demographics and clinical outcomes, leading to unsafe deployment in real-world settings.

2. Methodology: The DeVisE Framework

The authors introduce DeVisE (Demographics and Vital signs Evaluation), a behavioral testing framework that uses controlled counterfactuals to probe model reasoning. The framework operates in a zero-shot setting (no fine-tuning on the test data).

A. Data Construction

Source: MIMIC-IV database (ICU discharge notes from Beth Israel Deaconess Medical Center).
Dataset: 1,001 randomly selected adult patients.
Note Variants:
1. Raw Notes: Semi-structured clinical text containing full context.
2. Template-based Notes: Structured templates containing only the specific variables of interest (demographics and vital signs) to isolate variable effects from textual noise.
Variables:
- Demographics: Age, Gender, Ethnicity.
- Vital Signs: Heart Rate, Systolic/Diastolic BP, Respiratory Rate, Oxygen Saturation, Body Temperature.
Counterfactual Generation: For each patient, single-variable perturbations are created. Variables are mapped to clinically grounded severity scales (e.g., Normal, High, Very High) based on medical guidelines. For example, changing a heart rate from 89 bpm (Normal) to 120 bpm (High) creates a "counterfactual patient."
- Total generated: ~166,731 counterfactuals.

B. Evaluation Tasks

The framework evaluates models on two dimensions:

Task-Specific Evaluation:
- Downstream Tasks: Predicting Mortality (binary) and Length-of-Stay (LoS) (4-class ordinal) in the ICU.
- Metrics:
  - Label Probability Shift (KL Divergence): How much the predicted probability distribution changes between the original and counterfactual.
  - Label Flips (%): Frequency of categorical prediction changes.
  - Correct Direction (%CD): Whether the prediction shift aligns with clinical severity (e.g., higher severity $\rightarrow$ higher mortality risk).
  - Monotonicity (%Mono): Whether predictions change consistently as severity increases (e.g., severity +1 $\rightarrow$ risk increases; severity +2 $\rightarrow$ risk increases further).
Task-Independent Evaluation:
- Metric: Perplexity (PPL) shift. Measures how "surprised" the model is by the counterfactual text, independent of any downstream task. This tests if the model linguistically recognizes the clinical anomaly.

C. Models Evaluated

Eight state-of-the-art LLMs were tested across three categories:

General-purpose: Phi4, LLaMA-3.3, Qwen-2.5, GPT-OSS, GPT-4.1-mini.
Medical-domain: Meditron3-Phi4, OpenBioLLM.
Reasoning-oriented: DeepSeek-R1-Distill.

3. Key Contributions

DeVisE Benchmark: A novel evaluation framework applying behavioral testing to medical LLMs using minimally differing counterfactuals across demographics and vital signs.
Curated Dataset: Release of 1,001 raw and 1,001 template-based validated notes with corresponding counterfactuals, enabling controlled robustness testing.
Comprehensive Analysis: A comparative study of 8 LLMs across note modalities (raw vs. template), task types (specific vs. independent), and model archetypes (medical vs. general vs. reasoning).

4. Key Results

A. Task-Independent (Perplexity)

Models generally show increased perplexity (linguistic surprise) as counterfactual severity increases, indicating they recognize the text as less probable.
GPT-OSS-120B showed significantly higher sensitivity (instability) compared to others.
Template-based notes elicited stronger perplexity shifts than raw notes, suggesting models are more linguistically sensitive to isolated variable changes.

B. Task-Specific (Mortality & LoS)

Vital Signs:
- Most models exhibited monotonic and directionally correct shifts (e.g., worsening vitals $\rightarrow$ higher predicted mortality). This suggests models internalize physiological severity.
- Medical models (OpenBioLLM, Meditron) showed conservative behavior (lower KL shifts) on raw notes but higher stability.
- General-purpose models showed higher sensitivity but sometimes lacked monotonicity.
- DeepSeek-R1-Distill demonstrated a strong balance: modest sensitivity with high directionality and monotonicity, suggesting calibrated reasoning.
Demographics:
- Age had the strongest influence on predictions (older age $\rightarrow$ longer LoS).
- Gender and Race showed smaller but statistically significant effects.
- Bias Detection: Models predicted shorter LoS for White, Asian, and Female patients, and longer LoS for Black patients, aligning with known healthcare disparities. This indicates that pre-training biases persist even in zero-shot settings.
- Flip Rates: Demographic perturbations caused label flips up to 23% near decision boundaries, raising fairness concerns.

C. Impact of Note Modality (Raw vs. Template)

Templates amplify sensitivity: Using templates increased the percentage of correct directional responses (%CD) and label flips compared to raw notes.
Interpretation: Raw notes provide contextual information that may dilute the importance of specific vital signs, whereas templates force the model to rely explicitly on the provided variables.

5. Significance and Conclusion

Beyond Accuracy: The paper argues that standard accuracy metrics obscure critical behavioral differences. A model can be accurate on average but fail to reason correctly about specific clinical changes or exhibit hidden demographic biases.
Behavioral Testing: DeVisE provides a necessary "stress test" for clinical LLMs, revealing whether they rely on genuine medical reasoning or spurious correlations.
Safety Implications: The findings highlight that while models often follow clinical logic for vital signs, they inherit and amplify societal biases regarding demographics.
Future Directions: The authors call for behavioral testing to become a standard part of clinical LLM evaluation to ensure robustness, interpretability, and fairness before real-world deployment.

Limitations Noted: The study is limited to English and a single US hospital dataset (MIMIC-IV). It focuses on static vital signs and demographics, leaving temporal reasoning and other clinical variables for future work.