The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom

The Detective's Dilemma: When the Map Doesn't Match the Territory

Imagine you are teaching a robot to navigate a maze. You don't just want the robot to memorize the path; you want it to build a "world model" in its brain—a mental map that predicts what will happen if it turns left or right. If the robot can predict the future accurately, it can plan ahead and solve complex puzzles. This is the dream of latent world models: AI systems that learn the hidden rules of their environment so they can act intelligently without needing a human to hold their hand.

But here is the tricky part: how do you know the robot's mental map is actually good? Usually, scientists check by seeing how well the robot predicts the next step. If it can guess the next frame of a video perfectly, we assume it understands the world. However, this paper asks a dangerous question: What if the robot is a genius at predicting the next second, but a total disaster at planning for the next minute?

This story comes from the world of machine learning, specifically the study of how AI learns to simulate reality. It's a field where researchers build digital brains to understand physics, movement, and cause-and-effect. The stakes are high because if we build robots or self-driving cars based on models that look good on paper but fail in the real world, the consequences could be messy. This paper is a story about a team of independent researchers who decided to play "detective" with a famous new AI method called LeWorldModel. They wanted to see if the magic trick actually worked, or if the magician was just hiding a few crucial steps in the instructions.

---

The Case of the Missing Instructions

The researchers set out to reproduce a result from a paper claiming that their new AI, LeWorldModel, could solve a simple two-room maze about 87% of the time. This seemed low compared to other methods that hit 97–100%, but the author claimed their method was special because it was simpler and didn't need fancy tricks to work.

The independent team, led by Joyjeet Singh, rented some computer power (about $24 worth of GPU time) to rebuild the AI from scratch. They expected to see the same 87% success rate. Instead, they found a mess of hidden secrets.

The Four Hidden Rules
The team discovered that the original paper's instructions were like a recipe that forgot to mention the most important ingredients. If you followed the published configuration files alone, the AI wouldn't learn anything. The researchers found four "undocumented conventions" buried in the code that determined everything:

1. Dense Action Gathering: The AI needed to see every tiny movement the robot made, not just a few snapshots.
2. Programmatic Width: The size of a specific part of the brain was set by a hidden math formula, not a number in the settings file.
3. ImageNet Normalization: The pictures fed to the AI had to be processed in a very specific way (using "ImageNet" standards), not just by dividing colors by 255.
4. Z-Scored Actions: The robot's movements had to be mathematically adjusted based on the whole dataset, with weird "NaN" (not a number) errors removed first.

Without these four hidden fixes, the AI's "predictor" (the part that guesses the future) couldn't even converge. It was like trying to bake a cake with a recipe that forgot to say "add flour" or "preheat the oven."

The Ghost in the Machine
Even after fixing the recipe, the team hit a wall. For three of their training runs, the AI looked like it was failing. The "validation loss" (a score measuring how bad the AI was) was bouncing around wildly, suggesting the model was broken. But the "training loss" (how bad it was on the data it was learning from) was actually getting better!

They found the culprit: a Batch Normalization layer. Think of this as a thermostat in the AI's brain. In the wrong mode, this thermostat was broken, amplifying tiny errors by a factor of 300. It made the AI look like it was failing when it was actually learning perfectly. Once they "recalibrated" this thermostat, the true performance emerged.

The Two Different Maps
The team also found that the original paper had published two different sets of rules for testing the AI, and they gave totally different results:

• The "Short" Test: The code's configuration file said to test the AI with a goal 25 steps away and a budget of 50 steps. Under this rule, the original authors' own AI hit 84.0%.
• The "Long" Test: The paper's appendix said to test with a goal 100 steps away and a budget of 150 steps. Under this rule, the same AI only hit 14.0%.

The researchers realized that the "87%" success rate the paper claimed was likely based on the "Short" test, but the "Long" test (which the appendix described) was actually much harder and revealed the AI's weakness.

The Big Surprise: Accuracy Doesn't Mean Skill

The most shocking discovery wasn't about the hidden rules; it was about what "good" actually means.

The team trained three different versions of the AI:

1. A "bad" one with high prediction errors.
2. A "medium" one (the original authors' version).
3. A "great" one with very low prediction errors (the team's corrected version).

The Short-Term Win:
When the goal was close (25 steps away), the "great" AI was the best, hitting 94.0% success. The "bad" one was the worst at 78.0%. This made sense: better prediction = better planning.

The Long-Term Disaster:
But when the goal was far away (100 steps away), the rules flipped completely.

• The "great" AI (the most accurate predictor) became the worst planner, hitting only 20.0% success.
• The "bad" AI (the least accurate predictor) became the best planner, hitting 80.0% success.

The most accurate AI was so confident in its perfect predictions that it made huge, reckless moves and overshot the goal, ending up 116.6 units away (farther than a random guess!). The "bad" AI, unsure of its predictions, moved more cautiously and actually reached the goal.

What This Means for the Future

This paper doesn't say the AI method is useless. In fact, once the hidden rules were fixed, the team's AI reached 94.0% on the short test, beating the original claim. The representation (the mental map) was solid; the robot knew where it was.

However, the paper delivers a strong warning: You cannot judge a planner by how well it predicts the next step.

In this specific maze, being a "perfect predictor" actually made the AI worse at long-term planning. The researchers found that the most accurate models were so sure of their path that they committed to extreme actions that led them astray.

The team also found that a pre-registered experiment (a test planned in advance to see if the AI was better at navigating same-room vs. cross-room goals) worked on one version of the AI but failed completely on another. This suggests that the "magic" of the AI might be specific to the exact version of the model, not a general property of the method.

The Takeaway
The paper concludes that if you want to build an AI that can plan for the long haul, you shouldn't just pick the one that predicts the future most accurately. In fact, on this task, picking the most accurate model would have meant picking the worst planner. The lesson for the future is that we need new ways to test AI that look beyond simple prediction scores, because a perfect map doesn't always mean you know how to drive the car.