🤖 AI

Debugging code world models

This paper analyzes the limitations of Code World Models (CWMs) by identifying that their primary failure modes stem from token-budget exhaustion due to dense runtime states and subword tokenization issues with strings, while demonstrating that long-horizon state tracking errors are primarily caused by incorrect action generation rather than inherent state propagation weaknesses.

Original authors: Babak Rahmani

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Babak Rahmani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🎬 The Big Idea: The "Movie Director" AI

Imagine you have a super-smart AI assistant that doesn't just write code; it acts like a movie director who can watch a movie before it's even filmed.

In the world of computer science, this is called a Code World Model (CWM). Instead of just guessing the final answer to a math problem, this AI simulates the entire movie scene-by-scene. After every single line of code (every "action"), it pauses and writes down exactly what the computer's memory looks like (the "state").

Normal AI: "I think the answer is 42."
CWM AI: "Okay, line 1: variable X is 5. Line 2: X becomes 10. Line 3: X becomes 15. Therefore, the answer is 15."

The researchers wanted to know: Is this "movie director" perfect? If it makes a mistake, why?

🚨 The Two Big Problems

The researchers found that while these AI directors are great, they crash in two specific ways.

1. The "Too Long, Didn't Read" Problem (Token Budget)

Imagine you are trying to tell a story to a friend, but your friend only has a tiny notepad that fits 8 pages. If your story is short, no problem. But if the story involves a character walking through a forest, stopping at every tree to write down the leaf count, your notepad fills up instantly.

The Issue: When the code runs for a long time (like a loop that repeats 100 times), the AI has to write a "state report" after every single step. This creates a massive amount of text.
The Result: The AI runs out of space on its "notepad" (token budget) before it finishes the story. It gets cut off mid-sentence, and the answer is lost.
Analogy: It's like trying to read a 500-page book, but the library only lets you borrow the first 50 pages.

2. The "Confused Translator" Problem (String Brittleness)

This is the more interesting bug. The AI is amazing at math, lists, and logic. But it gets terrible at handling strings (text like "Hello World").

The Issue: Computers don't read text letter-by-letter like humans. They chop text into chunks called "tokens." Sometimes, a chunk like "-." is one single token. But if you put it inside a longer word, the computer chops it up differently.
The Result: The AI gets confused. It sees the text "-." in isolation and knows what it means. But when it sees it inside a sentence, the "chopping" changes, and the AI loses track of where the separator is.
Analogy: Imagine you are a translator who knows the word "Cat." But if you see "Cat" inside the word "Catastrophe," you suddenly forget what "Cat" means and start hallucinating. The AI fails not because the logic is hard, but because the text representation is shaky.

🧪 The Experiments: How They Found the Truth

The researchers didn't just guess; they set up controlled experiments to isolate the problems.

Experiment A: The "Lego Tower" Test (Composition)

They built towers of functions (like stacking Lego blocks) to see if the AI could handle deep logic.

Non-String Blocks (Math, Lists): The AI built 50-block towers perfectly. It was a genius at logic.
String Blocks (Text): As soon as they started stacking text operations, the tower collapsed. The deeper the stack, the more likely the AI was to drop a block.
Conclusion: The AI's logic engine is fine; its "text handling" is the weak link.

Experiment B: The "Long Walk" Test (Long-Horizon Tracking)

They asked the AI to track a permutation (swapping items around) for 128 steps.

The Baseline: The AI tried to do it all at once. It failed quickly.
The "Teacher" Test: The researchers forced the AI to use the correct next step (like a teacher holding its hand) and only asked the AI to predict the result of that step.
The Surprise: When the AI didn't have to guess the next move, it could track the state perfectly for 128 steps!
Conclusion: The AI isn't bad at remembering the state; it's bad at guessing the next command. Once it guesses wrong, the whole movie script is ruined.

💡 The Takeaways: What Does This Mean?

Don't Blame the Brain, Blame the Eyes: The AI's logic is strong. The problem is how it "sees" text (tokenization). If we can teach it to read text more like a human (character-by-character) rather than in chunks, it will get much better at strings.
The "State" is Heavy: Asking an AI to write a full report after every single step is too expensive and slow. It fills up the memory too fast.
The Future: We need AI that can run code without needing to write a report after every single step. We need models that can "hold the state" in their head (like a human does) rather than writing it down on a notepad every time.

🏁 Summary in One Sentence

Code World Models are like brilliant movie directors who can simulate code perfectly, but they keep running out of paper to write their notes, and they get confused whenever the script involves tricky text formatting.

1. Problem Statement

Code World Models (CWMs) are a class of language models trained to simulate program execution by predicting explicit runtime states (variable bindings and control flow) after every executed command. Unlike traditional Chain-of-Thought (CoT) reasoning, CWMs use dense state supervision, interleaving code actions with full snapshots of the runtime environment. While CWMs have shown significant gains on software engineering benchmarks, their failure modes and the fundamental reasons for their limitations remain poorly understood.

The paper addresses two critical questions:

Local Semantic Execution: Do CWMs correctly apply deterministic operations across different data types, or do they fail due to representation issues?
Long-Horizon State Tracking: Can CWMs maintain accurate state over extended execution histories, or do they suffer from compounding errors typical of Transformer architectures?

2. Methodology

The authors employ a dual-perspective evaluation strategy combining real-world benchmarks with controlled, synthetic experiments to isolate specific failure mechanisms.

A. Real-Code Benchmark Analysis

Datasets: CruxEval-O and HumanEval.
Approach: The models are tasked with simulating execution traces to predict final outputs. The authors analyze failure distributions by data type (strings, integers, lists, etc.) and categorize errors into semantic failures (incorrect logic) and truncation failures (token budget exhaustion).

B. Controlled Composition Experiments

Goal: To isolate the effect of data types from program structure.
Setup: The authors constructed a "Composition Zoo" where deterministic, single-argument functions are nested to varying depths ( $d=1$ to $5$).
Variables: They tested seven non-string data types (Boolean, Bitwise, Math, Character, List, Set, Dictionary) and compared them against String-valued compositions.
Intervention: A "Code Decomposition" intervention was applied to failing samples, rewriting complex expressions into explicit intermediate variables to test if errors stem from hidden state visibility.

C. Long-Horizon State Tracking (Code S5 Benchmark)

Goal: To evaluate state propagation over long sequences independent of local semantic errors.
Setup: A permutation-tracking task on the symmetric group $S_5$ . Models must track the state of 5 variables through $N$ swap operations ( $N \in \{8, 16, 32, 64, 128\}$ ).
Intervention (Teacher Forcing): The authors compared the baseline (model generates both action and state) against a "Teacher Forcing" setting where the ground-truth action is injected at every step, isolating the model's ability to update the state correctly.

3. Key Findings & Results

A. Two Dominant Failure Regimes

Analysis of CruxEval-O and HumanEval revealed that failures concentrate in two specific areas:

Token-Budget Exhaustion: Dense state supervision requires revealing the full runtime state after every command. For programs with deep iteration or in-place mutations (e.g., list insertion during iteration), the resulting traces exceed the token context window, leading to truncation.
String-Valued State Brittleness: Failures disproportionately occur in string operations.
- Statistic: In CruxEval-O, strings account for 73% of failures despite being only 46% of outputs. In HumanEval, strings are 44% of failures but only 17% of outputs.
- Cause: The paper attributes this to subword tokenization discontinuities (e.g., Byte-Pair Encoding). Specific character sequences (like separators in rsplit) may tokenize as a single token in isolation but split into multiple tokens within a larger string context. This prevents the model from "seeing" the target pattern at the token level, causing semantic hallucinations.

B. Compositionality Results

Non-String Data: CWMs achieved 100% accuracy across all non-string data types (Boolean, List, Dict, etc.) even at depth 5. This proves the model architecture can reliably compose deterministic logic when state representation is stable.
String Data: Accuracy degraded sharply with depth (from 75% at depth 2 to 25% at depth 5).
Conclusion: The failure is not in the logic of composition but in the representation of string states under tokenization.

C. Long-Horizon Tracking (Action Hallucination vs. State Propagation)

Baseline Performance: Accuracy dropped significantly as sequence length increased (e.g., near 0% at 128 steps for GPT-5 and standard CWM).
Root Cause: Inspection revealed that errors were primarily caused by Action Hallucination (generating an incorrect next command). Once a command is wrong, the subsequent state is deterministically corrupted.
Teacher Forcing Result: When ground-truth actions were injected (forcing the model to only predict the state update), the CWM maintained ~90% accuracy even at 128 steps.
Implication: Transformer-based CWMs can faithfully propagate state over long horizons if the action sequence is correct. The dense state supervision effectively reduces long-horizon tracking to a sequence of locally supervised updates, mitigating the known limitations of Transformers in state tracking.

4. Key Contributions

Diagnosis of Failure Modes: Identified that CWM limitations are not due to a lack of reasoning capability but rather tokenization-induced representation brittleness (specifically for strings) and efficiency bottlenecks (token budget).
Decoupling Action and State: Demonstrated that long-horizon failures in CWMs are driven by action generation errors, not state-update errors. When actions are correct, Transformers can track state for hundreds of steps.
Controlled Benchmarks: Introduced the "Composition Zoo" and "Code S5" benchmarks to isolate data-type effects and long-horizon dynamics, providing a rigorous framework for future CWM evaluation.
Intervention Analysis: Showed that code decomposition (making intermediate states explicit) improves performance slightly but is not scalable due to token explosion, highlighting the need for better state representations.

5. Significance and Future Directions

Representation Matters: The paper argues that for Code World Models to be robust, the community must move beyond standard subword tokenization. Byte-level or tokenizer-free approaches (e.g., ByT5, Byte Latent Transformers) are suggested to resolve the string brittleness issue.
Efficiency vs. Accuracy: Dense state supervision is powerful but computationally expensive. The results suggest a trade-off: while dense supervision enables accurate state tracking, it is unsustainable for long programs.
Architectural Shifts: The findings imply that future CWMs might benefit from linear recurrent architectures (like Mamba or Linear RNNs) which are better suited for sparse observation and long-horizon state propagation, potentially reducing the need for the token-heavy dense traces currently used.

In summary, the paper provides a "debugging" framework for Code World Models, revealing that their current limitations are largely solvable through better tokenization strategies and a shift in how execution traces are supervised, rather than requiring fundamentally new reasoning paradigms.