Technical Summary: HealthCraft

Problem Statement

Frontier large language models (LLMs) are being deployed into clinical workflows faster than the infrastructure exists to evaluate them safely. Existing static medical-QA benchmarks fail to capture critical failure modes specific to emergency medicine: trajectory-level safety collapse, tool misuse against patient records, and capitulation under sustained clinical pressure. Static benchmarks measure the recall of guideline text but cannot detect whether a model will recommend home care after correctly identifying the need for emergency transfer, or whether it will place a lethal order (e.g., anticoagulants for a suspected aortic dissection) when interacting with a clinical data system.

Methodology

HealthCraft is a public reinforcement learning (RL) environment adapted from the Corecraft architecture, specifically designed for emergency medicine. It operates as a Docker-bundled system comprising three services: a PostgreSQL world-state store, a FastMCP tool server, and a task engine.

Environment Design

• World State: The environment is grounded in a FHIR R4 schema populated with 3,987 seed entities across 14 types (e.g., patients, encounters, conditions, medications). The state is deterministic (seed=42) and derived from the OpenEM clinical knowledge corpus, ensuring direct analogues to real clinical integrations.
• Action Space: Agents interact via 24 MCP tools organized into four waves: read-only search/get, computation, mutation (e.g., creating orders, updating records), and composite workflows (discharge, transfer). Tool calls are deterministic and logged in an append-only audit log.
• Noise Injection: To simulate realistic clinical data issues, the environment injects structured noise, including pagination limits, clock skew, incomplete records, stale data, ambiguous documentation, and red-herring findings.

Reward Function and Rubric

The core innovation is a dual-layer rubric with a hard safety gate.

• Criteria: The benchmark uses 2,255 binary criteria per task, of which 515 are safety-critical.
• Hard Safety Gate: The reward function (Eq. 1) zeros the reward if any safety-critical criterion is violated, regardless of performance on other criteria. This reflects the non-negotiable nature of clinical safety where a single lethal error invalidates a competent performance.
• Verification: Criteria are verified via three methods:
  1. world_state: Deterministic checks against the audit log (preferred for safety).
  2. llm_judge: Cross-vendor LLM judges (e.g., GPT judging Claude) for complex trajectory analysis.
  3. pattern: Regex matches for presentation checks.

Task Suite

The public benchmark consists of 195 tasks (expanded to 205 with a negative-class slate) across six categories:

1. Clinical Reasoning (disambiguating overlapping conditions).
2. Multi-step Workflows (coordinated actions across encounters).
3. Clinical Communication (discharge, consults).
4. Safety-Critical Judgment (EMTALA, capacity).
5. Information Retrieval.
6. Temporal Reasoning.

Tasks are generated from OpenEM's confusion pairs, decision rules, and evaluation properties, ensuring traceability to peer-reviewed sources rather than author intuition.

Key Contributions

1. First Public RL Environment for Emergency Medicine: Features a FHIR-R4 world state and a 24-tool MCP surface.
2. Hard Safety Gate Reward: A reward function that zeros out performance upon any safety-critical violation, operationalizing constrained MDPs for clinical safety.
3. Traceable Benchmark: A suite of 195 tasks graded against 2,255 binary criteria, grounded in OpenEM clinical sources.
4. Transparent Evaluation: A detailed account of six infrastructure bugs fixed between pilot versions (v2 to v8) that significantly altered model rankings, demonstrating that infrastructure fidelity is part of the measurement.
5. Audit Overlay: A deterministic LLM-judge audit overlay that quantifies evaluator noise, finding that 73% of judge disagreements stem from hallucinations rather than overlay faults.

Results (V8 Release)

Evaluated on two frontier models (Claude Opus 4.6 and GPT-5.4) across 195 tasks with 3 trials each:

• Overall Performance:
  • Claude Opus 4.6: Pass@1 of 24.8% (Safety failure rate: 27.5%).
  • GPT-5.4: Pass@1 of 12.6% (Safety failure rate: 34.0%).
• Multi-Step Workflow Collapse: Performance on complex, multi-step workflows collapsed to near zero (Claude: 1.0%, GPT-5.4: 0.0%). While agents often satisfied non-safety criteria, a single safety-critical slip in the workflow (e.g., missing a transfer check) zeroed the reward.
• Infrastructure Impact: Fixes to six infrastructure bugs between v7 and v8 caused a qualitative re-ordering of model performance. GPT-5.4's average reward increased by ~107% relative to v7, while Claude's dropped by ~13%, highlighting that tool-use benchmarks are highly sensitive to schema and evaluation fidelity.
• Negative-Class Slate: A post-hoc 10-task negative-class slate (restraint-focused) revealed that restraint criteria are satisfied at a high prevalence (0.929) in smoke pilots, suggesting that using the current reward signal for training could lead to pattern-matching specific over-actions rather than generalized clinical judgment.

Significance and Claims

The paper positions HealthCraft not as a tool for immediate deployment approval, but as an honest evaluation harness that makes failure modes visible.

• Measurement of Reality: The environment demonstrates that current frontier models are not ready for autonomous emergency care deployment; they fail one in four (Claude) or one in eight (GPT) tasks at Pass@1, with safety violations occurring in roughly one in three trials.
• Infrastructure Fidelity: The authors argue that for tool-use benchmarks, infrastructure fidelity is not a separate axis from model evaluation but an integral part of what is being measured. The re-ordering of models due to bug fixes serves as evidence that "stronger" model rankings can be artifacts of harness imperfections.
• Training vs. Evaluation Boundary: The paper explicitly states that the current reward signal (Eq. 1) is not drop-in training-safe. The high prevalence of satisfied restraint criteria in negative-class tasks suggests that training on this signal without ablation could lead to reward hacking.
• Open Science: All components (environment, tasks, rubrics, harness) are released under Apache 2.0 to encourage the community to break the environment, extend the task suite, and report results.

The authors conclude that HealthCraft's job is to provide confidence intervals and visible failure rates so that deployment conversations can be honest, rather than to provide triumphant metrics for immediate clinical integration.