🤖 machine learning

When a Verified World Model Still Loses: Play-Adequacy vs Prediction-Accuracy in LLM-Synthesized Code World Models

This paper demonstrates that Large Language Model-synthesized Code World Models can achieve near-perfect transition accuracy on sampled data yet fail systematically in planning because they miss rare but pivotal rules, revealing that prediction accuracy is an inadequate metric for planning-oriented world models compared to play performance or search-distribution coverage.

Original authors: Javier Aguilar Martín

Published 2026-07-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Javier Aguilar Martín

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Magic Box and the Blind Spot

Imagine you are teaching a super-smart robot how to play a new board game. You don't want the robot to just memorize every single move it has ever seen; you want it to understand the rules so it can think ahead and win. In the world of artificial intelligence, there is a clever trick called a "Code World Model." Instead of the robot guessing, we ask a Large Language Model (like the ones that write stories or code) to read the rulebook and write a computer program that simulates the game perfectly. Once the robot has this program, it can run millions of imaginary games in its head to find the best move.

But here is the tricky part: How do we know the robot wrote the program correctly? Usually, we test it by playing a few random games against it. If the robot's program predicts the outcome of those random games correctly, we say, "Great! The model is verified!" and we let it play for real. This paper asks a scary question: What if the robot's program is perfect at predicting random games, but completely wrong at the specific moments that actually matter for winning? It turns out, just because a model passes a random test doesn't mean it's ready for the big leagues.

The Tale of the Missing Rule

The researchers in this paper discovered a hidden trap in how we test these AI game players. They found that an AI can pass a "verification gate" with flying colors—getting 100% of the random test games right—yet still lose every single real game against a skilled opponent.

To understand how this happens, imagine you are testing a new video game engine. You play 40 random matches where you just mash buttons and move your character in circles. The engine predicts exactly what happens in those 40 matches. You give it a gold star and say, "Perfect!" But suppose there is a secret rule in the game: "If the game lasts longer than 100 turns, the player with the most gold wins." In those 40 random, button-mashing games, the game never lasts that long, so the engine never gets tested on this rule. It passes the test.

However, when a smart player (the AI's opponent) plays, they know how to drag the game out to 100 turns to win. The AI, using its "verified" engine, doesn't know this secret rule exists. It thinks the game should end in a tie. Because it's missing this one tiny, rare rule, it makes a fatal mistake and loses. The researchers call this the "verified-vs-correct gap." The model is verified (it passed the test) but not correct (it fails in the real world).

The Law of the Rare Mistake

The paper doesn't just say this happens; it gives us a mathematical formula to predict exactly when it will happen. They call it the "Danger Law."

Think of the verification test as a fishing net. The "rarity" of the secret rule is how hard it is to catch a fish that triggers that rule. If the rule happens in 50% of games, the net (even a small one) will catch it easily. But if the rule only happens in 2.5% of games (like the "100-turn" rule in their experiment), and you only cast your net 40 times, there is a very high chance you will miss it completely.

The formula says: Danger = (How bad the mistake is) × (Chance the test missed the rule).

In their experiments, they created a game called army5x5a with a hidden rule about what happens when the game hits a time limit. They found that when the rule was rare (happening in only about 2.5% of random games), the AI passed the test but lost the real games with a win rate of only 0.404 (about 40%), compared to a fair baseline of 0.495 (about 50%). That might not sound like a huge difference, but in the world of competitive gaming, losing 1.6 games for every one you win is a disaster. The "verified" model was systematically outsmarted because it was blind to the one thing that mattered most.

Why More Examples Don't Help

You might think, "Okay, so the test was too small. Let's just give the AI more examples of that rare rule!" The researchers tried this, too. They fed the AI thousands of examples of the game ending in a tie-breaker, hoping it would "learn" the rule.

But here is the surprising twist: It didn't work. The AI acted like a translator, not a detective. If you told it, "Here is the rule," it wrote the code perfectly. But if you just showed it examples and said, "Figure out the rule," it failed. Even with hundreds of examples, the AI couldn't infer the missing rule from the data alone. It simply couldn't guess what wasn't explicitly written in the instructions. This suggests that for these AI systems, "specification completeness" (giving them the full rulebook) is far more important than "learning from examples."

The Poker Problem: When You Can't See the Cards

The paper also looked at games where you can't see everything, like Poker. In these games, the AI has to guess what cards the opponent is holding. This is called the "inference" part of the model.

The researchers proved that for simple poker games, a random test is actually safe because the game is too shallow; you can't hide a secret rule in a short game. But they built a tiny, custom game called Beacon to prove that in deeper, more complex games, the same trap exists. In Beacon, the AI had to guess an opponent's hidden type based on their moves. The AI passed the test perfectly (0 errors in 8,156 test cases) but lost every single real game (0% win rate).

Why? Because the test used random moves that almost never reached the deep part of the game where the secret was hidden. The AI's "guessing brain" was wrong, but the test never saw it because the test was too shallow. It's like testing a detective's ability to solve a murder by asking them to solve a missing sock case. They might get the sock case right, but fail the murder.

The Bottom Line

The main takeaway from this paper is a warning for anyone building AI that plays games or makes plans: Don't trust a model just because it passes a random test.

If the AI is going to be used by a smart planner that looks for deep, strategic moves, you cannot verify it with random, shallow tests. You have to either:

Test it on the actual strategy: Run the AI against a smart opponent and see if it wins, not just if it predicts random moves correctly.
Give it the full rulebook: Make sure the instructions are 100% complete before asking the AI to write the code.

The paper shows that a model can be "verified" and still be dangerously wrong, simply because the test missed the one rare, pivotal moment where the game is actually won or lost. It's a reminder that in the world of AI, passing a test isn't the same as being ready for the real thing.

Technical Summary: Verified World Models vs. Play-Adequacy in LLM-Synthesized Code World Models

1. Problem Statement

The paper investigates the Code World Model (CWM) paradigm, where Large Language Models (LLMs) synthesize executable code representing a game's transition dynamics, which is then used by a classical planner (e.g., MCTS) to play. The standard verification protocol involves refining the synthesized code until it achieves 100% transition accuracy on a set of randomly sampled trajectories (the "gate").

The central problem is whether passing this gate certifies the model's adequacy for planning. The authors argue that high prediction accuracy on random samples is the wrong metric for planning adequacy. They posit a structural gap: a planner concentrates search on strategically significant (often deep or rare) states, whereas random sampling concentrates on shallow, common histories. Consequently, a CWM can pass the gate with 100% accuracy yet fail systematically in play if it omits a "rare-but-pivotal" rule that random play rarely triggers but competent play reliably exploits.

2. Methodology

The study employs a rigorous experimental and theoretical framework across perfect-information and imperfect-information games:

Synthesis Pipeline: LLMs (GPT-5.x family: mini, nano, large) are prompted with game rules and random trajectories to synthesize Python modules implementing a standard CWM contract (initial_state, legal_actions, apply_action, etc.).
The Gate: A refinement loop checks the synthesized code against ground-truth trajectories. The model is accepted only if it achieves 100% transition accuracy on the sampled set.
Instrumentation:
- Perfect Information: The authors constructed a variant of the game army5x5a with a "material-at-cap" tiebreak rule. This rule is rare under random play (~~2.5% incidence) but decisive under competent play (~~50% incidence).
- Imperfect Information: They introduced Beacon, a minimal hand-instrumented game designed to isolate inference errors. It features a "deep region" reachable only by optimal survival strategies, which random play almost never reaches.
Metrics:
- Gate Lens: Transition accuracy, state accuracy, observation rate, and inference rate.
- Decision Lens: Win rate in arena play against a ground-truth opponent, measured with Wilson 95% confidence intervals.
- Gap Measurement: The difference between state accuracy on the gate distribution versus the planner's search distribution.
Repair Attempts: The authors tested whether providing discriminating examples (via DAgger, targeted on-manifold transitions, or artificial states) allows the LLM to infer the missing rule.

3. Key Contributions and Results

3.1 The Verified-vs-Correct Gap

The paper demonstrates that a CWM can pass the transition gate (100% accuracy) and be $\ge$ 98% state-accurate on the planner's search distribution, yet lose systematically at play.

Evidence: In army5x5a with the omitted material-at-cap rule, the "rule-blind" CWM achieved a win rate of 0.404 (95% CI [0.384, 0.424]) against a fair baseline of 0.495. The Wilson intervals are separated, confirming a statistically significant "play cost" of 0.091.
Cause: The error lies in the <1% of states where the rule applies. Random sampling dilutes this error, making state accuracy appear high, while the planner's search concentrates on these pivotal states.

3.2 The Quantitative Danger Law

The authors derive a closed-form law governing the expected harm from sampling verification:
$\text{danger} = \text{play\_cost} \times (1 - \text{rarity})^N$

$(1 - \text{rarity})^N$ : The exact probability that a gate of size $N$ (i.i.d. Bernoulli samples) fails to observe a rule with rarity $r$ .
$\text{play\_cost}$ : The empirical performance deficit conditional on the rule being omitted.
Threshold Behavior: The law predicts a threshold: harm is negligible when rules are common enough to be caught by the gate, rises sharply as rules become rare, and saturates at the full play cost when the rule almost always escapes detection.

3.3 Translation, Not Inference

The paper investigates whether providing example transitions can repair the missing rule.

Finding: LLM synthesis behaves as rule translation, not rule inference.
Evidence: Across multiple data regimes (naive DAgger, proper DAgger, targeted on-manifold examples) and model sizes, the LLM failed to infer the omitted rule even when discriminating transitions were present in the sample.
Provable Core: A theoretical proposition shows that if a rule is absent from the sample (probability $(1-r)^N$ ), no learner can infer it from those transitions alone. Even when the rule is present, the LLMs tested (GPT-5.x) consistently failed to encode it, suggesting a limitation in inferring rules from examples rather than just translating specifications.

3.4 Imperfect Information and the Inference Gap

The mechanism extends to the inference function (reconstructing hidden states) in imperfect-information games.

Coverage Bound: The authors prove that a random inference gate is identifying (covers all relevant info-sets) when $N \gtrsim b^{d_{max}} \cdot p_{chance}^{-1} \cdot \log |I|$ . This explains why shallow games like Kuhn poker show no gap.
Beacon Experiment: In the constructed game Beacon, a "verified-but-wrong" inference function passed the gate (0/8156 mismatches) but lost every single game (win rate 0.000 vs. 0.500 baseline). The error was confined to a deep region reachable by optimal play but invisible to random sampling.
Belief Blindness: A transition-accuracy gate is structurally blind to incorrect belief models (observation/inference functions) because belief partitions do not appear in transition tuples $(s, a, s')$ .

4. Significance and Claims

The paper claims to provide a rigorous negative answer to the question of whether transition accuracy on random trajectories certifies planning adequacy.

Structural Diagnosis: The failure is not a model capacity issue but a reach-distribution shift. The verification policy (uniform random) and the deployment policy (competent planner) concentrate mass on disjoint regions of the state space. This is analogous to the problem of unconstrained off-equilibrium-path beliefs in game theory.
Practical Implications:
1. Verification: Adequacy must be measured on the search distribution or by direct play, not by prediction accuracy on sampled transitions.
2. Specification: Completing the specification is more effective than attempting to repair gaps via examples. The model translates what is given; it does not reliably infer what is omitted.
Scope: The findings are established for the GPT-5.x family under specific regimes. While the "translation vs. inference" behavior is observed across model sizes and repair strategies, the authors frame the universal claim ("no model can infer an omitted rule") as a conjecture supported by their data, noting that stronger code-reasoning models or agentic scaffolds might alter this outcome.

The paper concludes that the "verified-vs-correct" gap is a fundamental limitation of sampling-based verification for synthesized world models in planning, driven by the statistical impossibility of catching rare-but-pivotal dynamics with uniform random sampling.