Technical Summary: Agents' Last Exam (ALE)

Problem Statement

Despite recent AI systems achieving state-of-the-art results on diverse benchmarks (e.g., Olympiad mathematics, competitive programming), these gains have not translated into economically meaningful deployment across professional domains. The authors argue this gap is primarily an evaluation problem: widely used benchmarks lack sustained performance measurement on real, economically valuable workflows. Existing evaluations often trade off realism, breadth, or verifiability, relying on synthetic environments, short-horizon tasks, or human judgment. Consequently, benchmark victories accumulate faster than measurable transformation in core industries like finance, law, engineering, and healthcare.

Methodology

Benchmark Design and Taxonomy

Agents' Last Exam (ALE) is a benchmark designed to evaluate Generalist Computer-Use Agents (GCUA) on long-horizon, economically valuable, real-world tasks with verifiable outcomes.

• Taxonomy Grounding: Unlike benchmarks that select domains ad hoc, ALE anchors its taxonomy in the O*NET / SOC 2018 (U.S. federal occupational taxonomy). It clusters occupations with similar software-mediated workflows into 13 industry clusters spanning 55 subfields (e.g., Biomolecular Structure, CNC Toolpath Generation, Clinical Diagnostics).
• Task Construction: Tasks are not synthetic; they are sourced from 250+ industry experts who contribute projects they have already completed. The construction pipeline involves:
  1. Expert Sourcing & Submission: Experts upload past projects via a web portal.
  2. First-Pass Review: Conference-style screening (Major/Minor Revision, Accept, Strong Accept).
  3. Implementation: Engineers convert specifications into runnable assets (VM containers, codified evaluation logic) and perform dry-runs.
  4. Final QC: Expert committees verify reference outputs and evaluation bounds.
• Scale: The dataset comprises 1,490 task instances (960 external submissions, 530 commissioned). To prevent contamination, only 150 tasks (~10%) are public, with the rest held in a private pool for rolling evaluation.

Agent Architecture and Environment

ALE targets the Generalist Computer-Use Agent (GCUA), defined as an agent possessing full capability across five functional layers:

1. Brain: LLM reasoning and planning.
2. Eyes: GUI perception (screenshots).
3. Body: Orchestration and control flow.
4. Hands: Structured tool invocation (APIs, file ops).
5. Feet: Runtime substrate (VMs, Docker).

• Evaluation Pipeline: Tasks run on remote virtual machines (Linux/Windows) with a standardized directory layout (input/, software/, output/, reference/).
• Interaction: Agents interact via a unified action loop combining GUI-as-Tool (desktop actions like click, type, scroll) and CLI operations.
• Verification: Evaluation is deterministic and artifact-based. The system avoids "LLM-as-judge" unless unavoidable (e.g., for visual aesthetics). Instead, it uses:
  • Exact/hashed value matching.
  • Structured numeric/tabular checks with tolerances.
  • Geometric/spatial distance (e.g., STL mesh comparison).
  • Behavioral world state checks.
  • Narrow, evidence-anchored yes/no probes by vision-LLMs only when necessary.

Difficulty Tiers

Tasks are organized into three tiers to accommodate different evaluation budgets:

1. Near-Term (59 tasks): Workflows current frontier agents can partially complete.
2. Full-Spectrum (55 tasks): One task per subdomain for broad coverage.
3. Last-Exam (35 tasks): The hardest workflows, where most agents currently achieve 0% pass rates.

Key Results

Performance Overview

Current mainstream agents and backbone configurations are far from saturated on ALE.

• Overall Pass Rate: The average full pass rate across mainstream harness and backbone configurations is 2.6%.
• Top Performers: The strongest configuration (Codex with GPT-5.5) achieves a 26.2% overall pass rate. Even this top configuration scores below 50% on the "Near-Term" tier and under 10% on the "Last-Exam" tier.
• Comparison to CLI Benchmarks: On ALE-CLI (a Linux-only subset), the same top configuration (Codex + GPT-5.5) achieves only 25.2% overall pass rate, compared to 82% on Terminal-Bench, highlighting the increased difficulty of ALE's long-horizon, multi-step workflows.

Failure Analysis

Analysis of failed runs (specifically Claude Code + Opus 4.7) reveals the dominant bottlenecks:

• Understanding & Approach (74%): The majority of failures stem from Domain Knowledge Gaps (25%) and Wrong Strategies (30%), rather than execution errors. Agents often default to ad-hoc scripts instead of using specialized domain software.
• Execution (22%): Includes output format errors, implementation bugs, and GUI failures.
• Tool Usage: Agents underutilize GUI tools (34% of tasks require GUI as primary tool), often substituting with Bash/CLI commands, leading to incomplete solutions.

Model vs. Harness Impact

• Model Dominance: Varying the foundation model (under a fixed harness) produces an 18.0 percentage point spread in pass rates.
• Harness Impact: Varying the harness (under a fixed model) produces a much narrower spread (5.3–6.0 percentage points).
• Conclusion: The choice of foundation model accounts for roughly 3× the performance variance compared to the choice of agent harness.

Significance and Claims

The paper positions ALE not merely as a leaderboard, but as an instrument for closing the gap between benchmark success and GDP-relevant impact.

• Closing the Evaluation Gap: ALE is the first benchmark to cover all 55 SOC/O*NET industries with authentic, expert-authored tasks, moving beyond the limited scope of previous benchmarks (e.g., SWE-bench, OSWorld, GDPval).
• Verification Standard: By replacing human grading with deterministic, rubric-based automated verification, ALE aims to provide a scalable, reproducible metric for professional capability.
• Readiness Threshold: The name "Last Exam" signifies a dual aspiration:
  1. Competence Threshold: Passing these exams indicates readiness to perform sustained, economically valuable work in a profession.
  2. Difficulty Frontier: The benchmark sits at the boundary of what current systems can reliably accomplish.
• Living Benchmark: ALE is designed as a "living" benchmark where the task pool grows continuously as new workflows and industries are onboarded, ensuring it remains a relevant measure of progress toward real industrial adoption.

The authors conclude that while frontier agents currently clear only a small fraction of these tasks, saturation of ALE would signal that AI systems have become capable of performing the underlying professional workflows at a level sufficient for real industrial adoption.