🤖 AI

ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks

The paper introduces ChainWorld, a framework that composes atomic OSWorld tasks into long-horizon desktop workloads to evaluate computer use agents, revealing that current models struggle with multi-step completion and exhibit distinct failure profiles depending on whether tasks are presented in single-turn or multi-turn formats.

Original authors: Vincent Siu, Manasi Sharma, Dawn Song, Daniel Yue Zhang, Chenguang Wang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vincent Siu, Manasi Sharma, Dawn Song, Daniel Yue Zhang, Chenguang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are testing a new robot assistant. So far, everyone has only tested it by asking it to do one simple thing at a time, like "open the toaster" or "turn on the light." The robot is great at these single tasks.

But in real life, a human doesn't just do one thing; they do a chain of things to get a job done. You don't just "open the toaster"; you open the toaster, put bread in, wait for it to pop, grab a plate, and then butter the toast. If the robot forgets the plate after the toast pops, the whole job fails, even if it opened the toaster perfectly.

This paper, called ChainWorld, is about testing computer agents (AI robots) on these longer chains of tasks instead of just single ones.

The Problem: The "One-Task" Trap

Current tests are like a driving test where you only have to parallel park once. You might pass that, but if you can't drive through a whole city without getting lost, you aren't ready for the real world. The authors found that just because an AI is good at single tasks doesn't mean it can handle a sequence of them.

The Solution: Building "Task Chains"

The researchers took a huge library of existing, simple computer tasks (called OSWorld) and tried to link them together like beads on a string.

The Challenge: You can't just randomly glue any two tasks together. If Task A deletes a file that Task B needs, the chain breaks. It's like trying to bake a cake where the first step is "throw away the flour."
The Method: They built a smart "compatibility map." They checked every possible pair of tasks to see if they could logically follow each other without crashing the computer or deleting necessary files. They then searched this map to build chains of 2, 3, or 4 tasks that made sense as a single work session.

They created 347 of these "chains" to use as a new, harder test.

The Experiment: Two Ways to Ask

They tested four different AI models on these chains using two different "rules of engagement":

The "All-at-Once" Test (Single Turn): The AI is given the whole list of tasks in one giant prompt. "Do Task A, then Task B, then Task C." It has to plan the whole journey in its head before moving a finger.
The "Step-by-Step" Test (Multi Turn): The AI is given Task A. Once it finishes, it gets Task B. Then Task C. It's like a human boss giving you instructions one by one, letting you finish one before moving to the next.

The Results: The Gap is Real

The results were surprising and a bit sobering:

The Success Rate is Low: Even the best AI models only completed the full chain of tasks about 31% of the time. That means they failed more than two-thirds of the time.
Step-by-Step Helps (A Little): Giving the AI tasks one at a time (Multi Turn) helped three of the four models perform better. It's easier to focus on one step than to hold the whole plan in your head. But even with this help, they still failed often.
Different Kinds of Failure:
- In the "All-at-Once" test: The AI usually got the idea right but messed up the details. It was like a student who understood the math problem but wrote the wrong number at the very end. They made "near-miss" errors.
- In the "Step-by-Step" test: The AI had trouble managing the session. They would get distracted, forget what they were supposed to do next, or give up halfway through. It was like a worker who starts a project, gets bored, and walks away before finishing.

The Big Takeaway

The paper concludes that we can't just look at how well an AI does single tasks and assume it's ready for real work. There is a huge gap between "doing a task" and "managing a workflow."

The authors built a new test (ChainWorld) that acts like a stress test for an AI's attention span and memory. It shows that while AI is getting better at seeing and clicking, it still struggles to stay on track when a job requires multiple steps and remembering what happened five minutes ago.

In short: AI is great at doing single tricks, but it's still learning how to juggle a whole routine without dropping the balls.

Technical Summary: ChainWorld

Problem Statement

Current evaluations of computer use agents rely almost exclusively on atomic desktop tasks, measuring perception, interface control, and short-horizon decision-making. However, realistic desktop work requires sustaining state across multiple related objectives and applications. Atomic task performance does not guarantee reliable long-horizon execution, as agents must preserve state, track unfinished objectives, and navigate across applications without losing context. Existing alternatives for creating long-horizon benchmarks—manual authoring and freeform synthesis—either lack scalability or shift the burden of coherence and validity into the generation pipeline itself, complicating evaluation.

Methodology: ChainWorld

The authors introduce ChainWorld, a framework that composes existing atomic tasks from the OSWorld benchmark into long-horizon desktop workloads while preserving the original source evaluators. The methodology avoids generating new tasks from scratch, instead relying on a constrained search over existing, verified tasks.

1. Composition Engine

The core of ChainWorld is a directional compatibility graph built from atomic OSWorld tasks.

Task Representation: Tasks are annotated with semantic fields (e.g., required applications, file writes, destructive operations, state assertions) to determine dependencies.
Compatibility Graph: A pairwise compatibility score determines if Task B can validly follow Task A. This involves:
- Hard Rules: Rejecting pairs with snapshot mismatches, terminal-only conflicts, or unrecovered process kills.
- Soft Rules: Rewarding state reuse (e.g., Task A writes a file Task B reads) and penalizing interference.
Constrained Search: The engine performs a beam search over the graph to find valid ordered chains of length 2 to 4, varying the underlying application footprint.
Coverage-Aware Selection: To ensure diversity, the selection process allocates a limited judging budget to maximize coverage across chain lengths, application counts, and source applications.

2. Quality Control Stack

A three-stage process ensures the composed workloads are executable and coherent:

Executable Continuity: Enforced by the compatibility graph rules.
Evaluator Validity: Full chain validation ensures all source task evaluators can still be applied to the final state.
Session Coherence: A tuple-level LLM judge (Opus 4.7) reviews the chain for "identity consistency" (whether tasks plausibly belong to one user session) and naturalness. Only 9 of 389 judged candidates were rejected for inconsistency, indicating the graph and search effectively pre-filter incoherent chains.

3. Evaluation Protocols

The resulting workload (347 chains) is evaluated under two protocols to isolate the effects of task presentation:

Single Turn: The entire chain is presented in one prompt with a concatenated instruction and a total step budget scaled by chain length ( $75 \times n$ ).
Multi Turn: Tasks are revealed one at a time. State persists across turns, and each task receives a fixed budget of 75 steps. This protocol tests session management and the ability to re-engage with new objectives.

Key Results

The study evaluated four computer use agents (GPT 5.5, Opus 4.7, Kimi 2.5, Gemini 3.1 Pro) on the ChainWorld workload.

Low Chain Completion: Across all models, the maximum chain completion rate is 31% (GPT 5.5 in multi-turn). This highlights a significant gap between atomic task success and coordinated long-horizon execution.
Protocol Impact: Multi-turn evaluation generally improved completion rates for GPT 5.5, Opus 4.7, and Kimi 2.5 compared to single-turn, suggesting that sequential revelation aids performance. However, completion rates remained low even with this advantage.
Failure Modes:
- Single Turn: Failures concentrate on artifact precision (near-miss errors where the agent produces the correct file type but wrong content/format).
- Multi Turn: Failures are more heterogeneous, frequently involving session management problems such as "fragmented progress" (attempting multiple tasks but completing none) and "later-turn disengagement" (stopping early in the sequence).
Infrastructure Validity: Runtime and setup failures accounted for only 3% of failures, confirming that the composed workloads are mechanically executable and that failures are primarily due to agent behavior rather than infrastructure breakdown.

Significance and Claims

The paper claims that ChainWorld demonstrates that long-horizon desktop evaluation can be constructed from existing, evaluator-backed atomic tasks without re-authoring workflows. By preserving the source evaluators, the framework maintains a direct link to established benchmarks while exposing coordination failures that atomic metrics obscure.

The authors argue that:

Atomic success does not imply long-horizon reliability: Agents often succeed locally but fail to sustain a multi-objective session.
Evaluation protocols matter: Single-turn and multi-turn settings reveal different failure profiles (precision vs. management), suggesting that a single aggregate score is insufficient for characterizing agent capabilities.
Structured Workload Analysis: Treating chained evaluation as a structured workload (analyzing by length and protocol) provides more actionable insights for training and system design than aggregate scores.

The study concludes that while current agents show promise in atomic tasks, they struggle significantly with the state preservation and session management required for realistic desktop workloads.