🤖 AI

T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph

This paper introduces T2D-Bench, a reproducible benchmark and evidence-gated evaluation framework utilizing a multi-layer clinical-lifestyle knowledge graph to demonstrate that computable evidence constraints can effectively detect, measure, and correct unsupported clinical omissions in large language model outputs for type 2 diabetes.

Original authors: Saba A. Farahani, Hung Cao, Ramesh Jain, Amir M. Rahmani

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saba A. Farahani, Hung Cao, Ramesh Jain, Amir M. Rahmani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant (a Large Language Model, or LLM) that can talk about diabetes like a seasoned doctor. It sounds confident, uses the right medical words, and gives advice that sounds perfect. But here's the catch: just because it sounds good doesn't mean it actually checked its homework or followed the strict rulebook.

This paper introduces T2D-Bench, a new way to test these robots to make sure they aren't just "making things up" or skipping important steps. Think of it as a strict teacher who doesn't just grade an essay on how pretty the handwriting is, but checks if the student actually cited their sources and followed every single rule in the textbook.

Here is how the system works, broken down into simple parts:

1. The "Master Rulebook" (The Knowledge Graph)

To test the robot, the researchers built a massive, digital map of facts called a Knowledge Graph. Imagine this map has three distinct layers:

Layer 1 (The Medical Backbone): This is the hard science. It contains official lists of diseases, drugs, and how they interact (like a giant, organized library of medical facts).
Layer 2 (The Rulebook): This is the "Law." It contains the official American Diabetes Association guidelines written in a language the computer can read. For example, "If a patient's kidney function is below X, do NOT give them Drug Y."
Layer 3 (The Lifestyle Bridge): This is the tricky part. It connects daily life (like sleep, diet, and exercise) to medical results (like blood sugar levels). It explains why sleeping poorly might raise blood sugar, creating a clear path from "I stayed up late" to "My sugar is high."

2. The "Exam" (The Vignettes)

The researchers created 100 specific test cases (called vignettes). These are like little stories about patients.

Some stories are simple: "The patient has high blood sugar; what's the diagnosis?"
Some are tricky "trap" questions: "The patient has high blood sugar, but they also have kidney issues and a weird sleep schedule. What do you recommend?"

For every single story, the researchers knew exactly what the robot must mention to pass. They called these "Gold Must-Trigger" items. If the robot didn't explicitly mention the specific rule or the specific lifestyle connection, it failed, even if the rest of the answer sounded great.

3. The "Evidence Gate" (The Teacher's Red Pen)

This is the core innovation. When the robot gives an answer, the Evidence Gate acts like a strict verifier:

Check: It scans the robot's answer to see if it included all the required "Gold" items from the map.
Fail & Correct: If the robot missed a step (like forgetting to mention that a specific drug is dangerous for kidney patients), the Gate says, "Stop! You missed a required piece of evidence."
Revision: The Gate asks the robot to rewrite the answer, forcing it to include the missing facts. It keeps doing this until the answer is 100% compliant with the rulebook.

What Did They Find?

The results were surprising and very clear:

The "Easy" Stuff: When the questions were about simple rules (like "What is the cutoff for diabetes?"), the robots were almost perfect. They knew the numbers.
The "Hard" Stuff: When the questions got complicated—mixing lifestyle factors like sleep and diet with medical rules—the robots failed 33% to 35% of the time on their first try.
- The Problem: The robots gave answers that sounded very plausible and confident, but they silently skipped the "mechanism." They might say, "Eat less sugar," without explaining how that connects to the patient's specific lab results, or they missed a drug interaction.
- The Fix: When the Evidence Gate stepped in and forced them to include the missing facts, the robots achieved 100% compliance. They could fix their mistakes when told exactly what was missing.

The Big Takeaway

The paper argues that in healthcare, "sounding smart" is not the same as "being safe."

A robot can write a beautiful, convincing paragraph about diabetes that is actually dangerous because it missed a tiny, crucial detail. T2D-Bench proves that we need a system that doesn't just ask, "Does this sound right?" but instead asks, "Did you check the map? Did you follow the rule? Did you connect the lifestyle dots?"

By using this "Evidence Gate," we can turn a robot that might confidently give bad advice into one that is forced to show its work and stick to the facts, making it much safer for real-world use.

Technical Summary of T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes

Problem Statement
While Large Language Models (LLMs) demonstrate the ability to generate clinically fluent recommendations for Type 2 Diabetes (T2D), they frequently fail to satisfy deterministic guideline constraints or explicitly justify lifestyle-related glycemic claims. Current evaluation methods often rely on lexical similarity, subjective expert judgment, or surface-level plausibility, which may fail to detect critical errors such as omitted thresholds, missed contraindications, or unsupported mechanistic claims. In safety-critical healthcare settings, these "plausible but ungrounded" outputs pose significant risks. Specifically, there is a lack of frameworks that can test whether LLM outputs are explicitly supported by authoritative evidence and can handle complex, multi-factor patient scenarios without silently inventing missing mechanistic steps.

Methodology
The authors introduce T2D-Bench, a reproducible benchmark and evidence-gated evaluation framework designed to test if diabetes-focused LLM outputs satisfy computable clinical and mechanistic evidence requirements. The methodology consists of three core components:

Multi-Layer Clinical–Lifestyle Knowledge Graph (KG):
The framework is built on a typed directed property graph ( $G$ ) with 2.1 million nodes and 495,000 edges, structured into three layers:
- Layer 1 (Biomedical Spine): Derived from UMLS, DrugBank, and SIDER, providing normalized identifiers, semantic typing, and pharmacovigilance structures (e.g., drug–drug interactions, adverse effects).
- Layer 2 (Computable Guideline Logic): Encodes selected statements from the American Diabetes Association (ADA) Standards of Care (2026) as machine-checkable primitives. This includes Rule nodes linked to Threshold nodes (operator, value, unit) to support deterministic checking of diagnostic criteria (e.g., HbA1c) and condition-dependent constraints (e.g., metformin contraindications based on eGFR).
- Layer 3 (Lifestyle Context & Mechanistic Bridge): Integrates lifestyle data (USDA, FooDB, Compendium of Physical Activities, Sleep Domain Ontology) and links them to glycemic physiology via an explicit "mechanistic bridge." This bridge maps lifestyle profiles (e.g., short sleep) to mechanisms (e.g., cortisol elevation) and subsequently to laboratory effects (e.g., increased fasting glucose), using typed edges (e.g., CAUSALLY_INCREASES) with provenance metadata.
Structured Benchmark Vignettes:
The benchmark comprises 100 structured JSONL vignettes covering four scenario families: diagnostic threshold compliance, medication safety under comorbidity constraints, single-factor lifestyle effects, and multi-factor lifestyle conflicts. Each vignette is paired with a "gold" specification defining required must-trigger identifiers:
- Rule IDs (Layer 2) for guideline compliance.
- Bridge/Mechanism tokens (Layer 3) for mechanistic attribution.
  A compact "benchmark slice" ( $G_{bench}$ ) containing 15,792 nodes is extracted for fast, deterministic verification.
Evidence Gate and Constrained Revision:
The evaluation workflow involves a deterministic verifier that checks if an LLM's output contains all required gold identifiers for a specific vignette.
- Baseline: The model generates a response without verification.
- Evidence-Gated: If the output lacks required identifiers, the system triggers a constrained revision loop. The model is prompted to revise its answer to explicitly incorporate the missing evidence identifiers as an "Evidence Trace." This loop continues until the output passes verification or a revision budget is exhausted.

Key Contributions

Multi-Layer Knowledge Graph: Development of a provenance-aware T2D knowledge graph that connects biomedical concepts, computable guideline logic, and lifestyle context through an explicit mechanistic bridge.
T2D-Bench Benchmark: Introduction of a 100-vignette benchmark paired with predefined evidence identifiers to test whether LLM outputs satisfy computable clinical and mechanistic requirements, moving beyond surface plausibility.
Evidence-Gated Evaluation Framework: A system that detects unsupported omissions and uses constrained revision to achieve verifier-level compliance with benchmark-defined evidence paths.

Results
The framework was evaluated on GPT-4o-mini and GPT-4o using deterministic decoding (temperature=0):

Baseline Performance: Without evidence gating, GPT-4o-mini satisfied all required identifiers in 65% of cases, and GPT-4o in 67%.
Scenario Stratification:
- Deterministic Guidelines: Both models achieved 100% compliance on diagnostic thresholds and renal medication safety vignettes.
- Lifestyle & Mechanisms: Performance dropped significantly in lifestyle scenarios. In single-factor lifestyle cases, compliance was 35–39%. Crucially, in multi-factor lifestyle conflicts (adversarial scenarios), baseline compliance was 0% for both models; they produced clinically plausible narratives that omitted required mechanistic bridge identifiers.
Evidence Gating Impact: After applying the Evidence Gate and constrained revision, both models achieved 100% evidence-path compliance across all 100 vignettes. The intervention rate was 35% overall, concentrated entirely in lifestyle and conflict scenarios.

Significance and Claims
The paper claims that T2D-Bench demonstrates that clinically plausible output should not be equated with evidence-grounded output. The primary significance lies in making hidden omissions—specifically unsupported mechanistic attributions in multi-factor settings—explicit, measurable, and correctable.

The authors emphasize that the Evidence Gate functions as a verification layer rather than a substitute for clinical reasoning. Its role is to enforce explicit graph-aligned evidence traces when baseline outputs are vulnerable to unsupported claims. The study concludes that computable evidence constraints can effectively intercept and correct diabetes-focused LLM outputs, providing a reproducible template for evaluating whether clinically plausible LLM outputs are explicitly supported by computable evidence paths. The authors modestly note that evidence-path compliance is an operational metric tied to the benchmark and does not replace prospective clinical validation.