OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models

1. Problem Statement

Current AI agent benchmarks suffer from a fundamental evaluation gap. While AI agents are expected to perform high-value professional work (e.g., emergency triage, nuclear safety monitoring, customs processing), existing benchmarks are limited to domains with public, accessible environments (e.g., web browsing, code repositories, retail APIs).

• The Untestable Majority: Critical professional domains (Healthcare, Finance, Legal, Manufacturing) are bound to private enterprise systems with no public APIs, making them impossible to evaluate with current tools.
• Scaling Limitations: Creating new benchmarks for these domains requires massive engineering overhead to deploy and maintain real-world environments or complex simulators.
• Lack of Robustness Testing: Existing benchmarks primarily test the "happy path" (ideal conditions) and fail to systematically evaluate how agents handle real-world environmental noise, such as API timeouts, data truncation, or service degradation.

2. Methodology: Language World Models (LWMs)

The authors propose Language World Models (LWMs) as a solution to simulate professional environments without requiring real infrastructure.

• Definition: An LWM is an LLM configured to act as a stateful, interactive environment. It takes an agent's action (tool call) and generates a structured observation (tool response) based on a specific configuration.
• Configuration Components:
  1. System Prompt: Encodes domain logic, rules, and error handling protocols.
  2. Tool Schema: Defines the action space (callable functions with typed parameters).
  3. Initial State: Structured JSON defining the starting conditions (e.g., inventory, patient queue).
  4. State Description: Semantic annotations guiding the LLM to maintain causal consistency (e.g., "inventory decreases after booking").
• Why it Works: LLMs leverage pre-trained knowledge of domain operational logic and format priors from API documentation to simulate tool responses, transforming environment construction from an engineering problem into a configuration problem.

3. The OCCUBENCH Benchmark

Built upon LWMs, OCCUBENCH is the first systematic cross-industry evaluation framework for professional tasks.

• Scale & Coverage:
  • 100 real-world professional task scenarios.
  • 10 industry categories (e.g., Business, Healthcare, Transportation, Public Service).
  • 65 specialized domains.
  • 382 evaluation instances.
• Multi-Agent Synthesis Pipeline:
  • Uses a multi-agent pipeline (powered by Gemini-3-Flash-Preview) to automatically generate environment configurations, task instructions, and verification rubrics.
  • Ensures solvability (valid solutions exist), verifiability (automated success criteria), discriminative difficulty, and structural diversity grounded in professional reference documents.
• Evaluation Dimensions:
  1. Task Completion: Measures multi-step decision-making across industries.
  2. Environmental Robustness: Evaluates performance under controlled fault injection:
     • E0 (Clean): Baseline.
     • E1 (Explicit Faults): Overt errors (HTTP 500, Timeouts) that signal failure.
     • E2 (Implicit Faults): Silent degradation (truncated data, missing fields, stale cache) with no error signals.
     • E3 (Mixed): Combination of explicit and implicit faults.

4. Key Results

The paper evaluates 15 frontier models across 8 model families (including GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro, Qwen 3.5, etc.).

• No Single Dominant Model:
  • Models exhibit distinct "occupational capability profiles."
  • Example: GPT-5.2 leads in Agriculture (84%), Business (86%), and Science (94%) but lags in Commerce (67%). Claude Opus 4.6 excels in Transportation (77%) but struggles in Commerce (53%).
  • Open-source models (Qwen 3.5 Plus, DeepSeek V3.2) are highly competitive, outperforming many closed-source variants.
• Implicit Faults are the Hardest Challenge:
  • Performance drops significantly under E2 (Implicit Faults) (Average: 53.4%) compared to E1 (Explicit Faults) (Average: 62.6%).
  • Implicit faults lack overt error signals, requiring agents to independently detect data degradation—a capability most current models lack.
• Scaling and Reasoning Effort:
  • Model Size: Larger models consistently outperform smaller variants within the same family (gaps up to 11%).
  • Generations: Newer generations show consistent improvement.
  • Reasoning Effort: Increasing reasoning effort yields significant gains. GPT-5.2 improved by 27.5 points when moving from "no reasoning" to "maximum reasoning" effort.
• Simulator Quality Matters:
  • Strong agents are not necessarily strong simulators. GPT-5.2 was the top agent (79.6%) but produced the worst simulation quality (average agent score dropped to 29.3% when evaluated by a GPT-5.2 simulator).
  • However, with a capable simulator (e.g., Gemini Flash or Qwen 3.5 Plus), pairwise ranking agreement reaches 85.7%, confirming LWM-based evaluation is reliable if the simulator is robust.

5. Significance and Contributions

• Bridging the Evaluation Gap: OCCUBENCH makes the "untestable majority" of professional domains evaluable without requiring real-world infrastructure, covering scenarios like nuclear safety and emergency triage that were previously impossible to benchmark.
• New Evaluation Paradigm: It shifts the focus from single-domain task completion to cross-industry specialization and environmental robustness, revealing that agents have distinct strengths and weaknesses depending on the domain.
• Insight into Agent Failure Modes: The study highlights that implicit data degradation is a critical failure point for current agents, as they struggle to detect silent errors without explicit signals.
• Practical Implications: Organizations should select AI agents based on specific industry needs rather than aggregate rankings. Furthermore, the reliability of LLM-based evaluation depends heavily on the quality of the simulator used.

Conclusion

OCCUBENCH establishes a new standard for evaluating AI agents in professional settings. By utilizing Language World Models, it provides a scalable, diverse, and robust framework that reveals the nuanced capabilities and limitations of current frontier models, particularly their vulnerability to silent environmental faults and their varying performance across different occupational domains.