An Omitted Mode Is a Rare Rule: The Sampling-Verification Danger Law in Continuous Code World Models

In the modern world of artificial intelligence, a popular strategy for teaching machines to control physical systems—like robots or self-driving cars—is to have the machine first build a mental map of how the world works. This map, often called a "world model," predicts what will happen next if a specific action is taken. Once the machine has this map, it uses a planning algorithm to simulate thousands of possible futures, looking for the sequence of actions that leads to the best outcome. The assumption has long been that if this mental map is accurate enough on the data it has seen, it will be safe to use for planning. If the map predicts the future correctly in a thousand test runs, it is assumed to be a reliable guide.

However, a new study challenges this assumption by looking at what happens when the world contains rare, sudden events that the machine has never encountered in its training. Imagine a car that has learned to drive on a smooth road but has never seen a wall. If the wall is rare enough that the car never drove into it during its training, the machine's mental map might simply ignore the wall's existence, assuming the road continues forever. The danger is that when the machine finally tries to drive in the real world, it might plan a path straight into that unseen wall, believing it can pass through. The question researchers asked was whether a standard safety check, which tests the machine's map against random samples of the world, would catch this specific kind of blindness before it causes a crash.

The researchers set up a series of controlled experiments to answer this, using simple physical simulations like a cart moving along a track or a pendulum swinging. In these simulations, they introduced a "rare rule": a hard stop, like a wall or a floor, that would instantly halt the object's motion if it touched it. They then asked a large language model to write the code for the machine's mental map, but they deliberately hid the existence of this wall from the model's training data. The model was only shown random movements of the cart or pendulum that happened to avoid the wall. Because the wall was rare, the model often never saw it during its training.

The results were stark. When the researchers tested the models using a standard safety check—running the model through thousands of random scenarios to see if it matched the real physics—the models frequently passed. They were accepted as "correct" because, in the thousands of random tests, the wall was never triggered. Yet, when these "verified" models were handed over to a planner to control the system, the planner would confidently steer the cart or pendulum directly into the hidden wall. The model, having no concept of the wall, would predict the object passing right through it. The planner, trusting this false prediction, would drive the object into the wall, where it would get stuck. The machine would then be unable to reach its goal, losing almost all of its potential success. The study found that this failure was not a mistake in calculation but a fundamental gap in knowledge: the model was blind to a rule that existed but was never sampled.

The researchers discovered that the likelihood of this disaster follows a precise mathematical pattern based on rarity. If a dangerous event happens in one out of a hundred random trials, and the safety check only runs a hundred trials, there is a significant chance the check will miss the event entirely. If the check runs a thousand trials, the chance of missing it drops, but it never reaches zero. The study proved that as long as the training data does not include the rare event, no amount of clever planning or model refinement can discover it. The model remains blind, and the safety check, by its very nature of using random sampling, cannot guarantee that the blind spot has been found.

The study also explored whether the machine could learn the rule if it was shown the wall during training. On simple, one-dimensional tasks, like a cart moving in a straight line, the answer was surprisingly positive. When the model was shown even a few examples of the cart hitting the wall, the language model was able to write the exact code to stop the cart, effectively "repairing" its mental map. It learned the rule perfectly. However, the researchers then moved to more complex, two-dimensional environments, where the wall was a circular patch on a flat surface. In these cases, even when the model was shown the wall hitting the patch, it failed to infer the correct shape and location of the boundary from the evidence. Instead of learning the circular rule, the model would invent incorrect shapes, like a straight line or a square, or it would simply freeze the object in place without understanding why. The model could not generalize from the few examples it saw to the full shape of the obstacle.

This finding highlights a critical limitation in how these systems learn. The study showed that while the machine can be excellent at translating a rule it is explicitly told or shown in a simple context, it struggles to infer the shape and location of a complex rule from scattered evidence. The machine's ability to "fix" itself depends heavily on the geometry of the problem. In simple cases, it works; in complex, two-dimensional cases, it fails. The researchers tested various ways to help the model, such as giving it more examples, changing the way it was asked to think, or providing hints about the shape of the obstacle. None of these interventions worked. The model consistently failed to induce the correct rule for the two-dimensional patch, often substituting a simpler, incorrect shape.

The implications of this work are significant for anyone relying on AI to control physical systems. It suggests that safety checks based on random sampling are insufficient for catching rare but catastrophic failures. A model can pass every test it is given and still be dangerously wrong about a specific, rare event. The study proves that if a dangerous event is rare enough to be missed by the sampling process, the model will remain blind to it, and the planner will exploit that blindness, leading to failure. The only way to ensure safety is to guarantee that the training data covers the boundaries of these rare events, or to use a different kind of verification that does not rely on random chance. The research concludes that while AI can be remarkably good at learning from data, it cannot learn what it has never seen, and in the physical world, what it has never seen can be exactly what causes the most damage.