💻 computer science

CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations

The paper introduces CALIPER, a calibration-based benchmark demonstrating that standard "clean scene" evaluations fail to distinguish whether pretrained visual encoders truly infer physical properties like mass and friction, as their apparent competence often relies on direct pixel-level cues rather than genuine physical reasoning.

Original authors: Aman Mehta, Riya Baviskar

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Mehta, Riya Baviskar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To build machines that can move through the real world, scientists are teaching computers to see not just shapes and colors, but the hidden rules that govern how things move. When a robot pushes a box, the distance it slides depends on invisible factors like how heavy the box is and how much friction exists between the box and the floor. A computer cannot see these properties directly in a single photograph; a light box and a heavy box look exactly the same until something moves them. To bridge this gap, researchers use artificial intelligence models trained on vast libraries of video to act as the eyes of these robots. These models are supposed to learn the physics of the world so they can predict what will happen next. But for years, the tests used to grade these models have been too simple to tell if they are actually learning physics or just memorizing the camera angles.

A team of independent researchers recently ran a new experiment to see if these digital eyes could truly understand the physical world. They set up a simulation where a standard puck strikes a box of unknown weight and friction. First, the computer watches two practice strikes where the puck hits the box at known speeds, and the box slides a specific distance. These are the calibration moments, giving the computer a chance to learn the box's secrets. Then, the computer sees a third strike at a new speed, but the video cuts off the moment the puck touches the box. The computer must now predict how far the box will slide based only on what it learned from the first two hits. The researchers tested eight different types of vision systems, ranging from highly advanced models trained on millions of videos to a computer vision system with completely random, untrained weights.

The results revealed a startling flaw in how we currently judge these machines. When the researchers ran the test in a perfectly clean, static room with a fixed camera, every single system performed almost perfectly, including the one with random weights. The computer did not need to understand physics to get the right answer; it simply learned that in a fixed camera view, the distance an object moves creates a specific pattern of pixels. Because the camera never moved, the computer could measure the slide directly from the screen coordinates without ever needing to know the box's mass or friction. It was like a student who memorized the answer key for a specific test but could not solve the problem if the numbers changed. The researchers found that in this clean environment, the test could not tell a smart model from a dumb one.

To fix this, the researchers changed the environment for every single video clip. They randomly shifted the camera angle, changed the lighting, altered the floor color, and added distracting objects to the scene. Suddenly, the pattern of pixels no longer gave away the answer. The computer could no longer rely on looking at the screen coordinates. In this messy, unpredictable world, the results split dramatically. The most advanced models, trained to predict video, still performed well, correctly predicting the slide distance with high accuracy. However, the models that relied on looking at single frames or had no training at all failed completely. The untrained system, which had scored nearly perfect in the clean room, now performed no better than a system that simply ignored the object and guessed based on the push speed alone.

The study also examined how these models are usually tested by other scientists. A common method involves changing one property in a scene, like making an object slightly heavier, and seeing if the computer's internal representation changes. The researchers found that in many existing tests, the change was so tiny that the computer's reaction was just random noise, meaning the test was measuring nothing real. They also looked at "probes," which are simple tests that try to read a specific property, like mass, directly from the computer's brain. They discovered that a model could pass a probe test and appear to know the mass, yet fail to use that knowledge when actually making a prediction. Conversely, a model could fail a probe test but still use the information effectively if the scene provided other clues. This proved that simply being able to read a property from a model's memory does not mean the model is actually using it to understand the world.

The final measure of success was a practical task: asking the computer to choose the exact speed needed to push the box to a specific goal distance. In the messy, changing environment, the best model missed the target by only 4.0 millimeters. The untrained model, however, missed by 19.6 millimeters, a failure rate identical to ignoring the object entirely. The researchers concluded that the ability of a test to separate good models from bad ones must be proven, not assumed. If a test cannot distinguish between a sophisticated AI and a random guesser, the test itself is broken. By introducing real-world chaos and requiring the computer to use evidence from multiple interactions, the new method successfully revealed which models truly understand the physics of motion and which are just looking at the pixels.

Technical Summary: CALIPER

Problem Statement
Pretrained visual encoders (e.g., V-JEPA 2, VC-1, R3M) serve as the perception front-end for world models in robotic manipulation. Their ability to infer latent physical properties, such as mass and friction, is typically evaluated using perturbation benchmarks and linear probes in "clean" scenes characterized by a fixed camera, a single object, and no clutter. The authors argue that these standard evaluations are fundamentally flawed: they fail to distinguish between encoders that genuinely infer physics and those that do not. In fixed-camera settings, the displacement of an object becomes a deterministic function of pixel coordinates. Consequently, any injective feature map (including raw pixels or randomly initialized networks) allows a linear readout to recover the physical outcome without inferring the underlying physics, rendering the benchmarks incapable of ranking representations.

Methodology: CALIPER
The paper introduces CALIPER (Calibrate, then Predict), a benchmark designed to ensure the answer is not contained within the input frame itself. The task involves a simulated environment where a standard puck strikes a target box of unknown mass ( $m$ ) and friction ( $\mu$ ).

Calibration: The model observes two full clips of the target being struck at known speeds ($0.45$ and $0.75$ m/s), revealing the resulting slide distances.
Query: The model is shown a third strike at a new speed, but the input is truncated exactly at the moment of contact. The model must predict the subsequent slide distance.
Swap Test: To verify that the model uses the specific object's interaction history rather than spurious correlations, the calibration clips are swapped with those from a different object. If accuracy drops, the model was using the correct evidence.

The evaluation is conducted across two distinct scene configurations:

Clean Scene: Fixed camera, fixed lighting, empty table.
Nuisance Scene: Camera pose, lighting, floor color, and distractor objects are resampled for every clip, ensuring clips within an episode do not share a frame of reference.

Eight representations are tested under a unified linear readout (ridge regression on frozen features): V-JEPA 2, VideoMAE, VC-1, DINOv2, R3M, a randomly initialized ViT, raw pixels, and frame differences. The study also employs "erasure" tests (iterative nullspace projection) to determine if a property is necessary for prediction and analyzes probe accuracy under different frame aggregation strategies (mean-pooled vs. concatenated).

Key Results

Failure of Clean Scenes: In the fixed-camera clean scene, all representations, including the random ViT and raw pixels, achieve nearly identical performance ( $R^2 \approx 0.99$ ), within $0.02$ of the simulator-state ceiling. The calibration clips provide a $+0.50$ $R^2$ boost for all models, but the swap removes this gain entirely. This confirms that the high scores are due to the scene geometry (pixel coordinates mapping directly to displacement) rather than physical inference.
Success of Nuisance Scenes: When camera, lighting, and clutter vary per clip, the representations separate significantly. The spread in performance grows from $0.020$ to $0.498$ $R^2$ $R^{2}$ .
- Top Performers: V-JEPA 2 ($0.974$), Frame Differences ($0.944$), and VideoMAE ($0.912$) maintain high performance.
- Mid-Tier: VC-1, R3M, and DINOv2 drop to the $0.67–0.71$ range.
- Bottom Tier: The random ViT drops to $0.476$, performing no better than a baseline that ignores the object ($0.514$), indicating it cannot distinguish object displacement from camera motion under variation.
Control Task Performance: When asked to select a push speed to reach a specific goal distance, V-JEPA 2 misses by $4.0$ mm, while the random ViT misses by $19.6$ mm (matching the "ignore object" baseline).
Probes vs. Use: Linear probes fail to track these differences.
- Aggregation Sensitivity: Concatenating frames instead of mean-pooling raises the random ViT's friction probe score from $0.361$ to $0.672$ (surpassing VC-1) without changing its prediction accuracy.
- Scene Dependence: Erasing the mass direction from raw pixels costs $-0.001$ $R^2$ in the clean scene but $-0.350$ in the nuisance scene. This demonstrates that probe decodability is not a reliable indicator of whether a property is being used for prediction, as redundancy in the scene can mask the necessity of specific features.

Significance and Claims
The paper concludes that a benchmark's ability to rank models must be empirically demonstrated, not assumed. The authors propose three checks to validate benchmarks:

Effect-Size Audit: Verify that perturbations actually change the simulated outcome (the authors found a prior benchmark where 13 of 20 cells changed the outcome by less than $10^{-4}$ ).
Spread Analysis: Ensure the performance gap between a random baseline, raw pixels, and the best model is significant; a narrow spread indicates the scene, not the encoder, is driving the score.
Swap/Erasure Verification: Confirm that claims of physical inference are supported by evidence that the model relies on the specific interaction history, as decodability is neither necessary nor sufficient for use.

The authors assert that the "clean scene" results previously interpreted as "all representations are equivalent" were actually a statement about the scene's lack of invariance requirements. The CALIPER benchmark reveals that while many models can learn to predict outcomes in static, fixed-camera environments, only specific pretraining objectives (like video prediction in V-JEPA 2) confer the invariance necessary for robust physical inference in dynamic, variable environments.

Technical Summary: CALIPER

More like this