Inferring Dynamic Physical Properties from Video Foundation Models

This paper introduces new synthetic and real-world video datasets for predicting dynamic physical properties like elasticity, viscosity, and friction, and evaluates various inference methods—including classical computer vision, prompt-based adaptation of video foundation models, and multi-modal large language models—demonstrating that pre-trained generative and self-supervised video models achieve performance comparable to each other and approaching an oracle baseline, while currently outperforming MLLMs.

The Gist

Imagine you are watching a video of a rubber ball bouncing, honey dripping, or a toy car sliding across a table. You don't need a ruler or a stopwatch to know that the ball is bouncy, the honey is thick, or the table is slippery. Your brain intuitively "feels" the physics just by watching the motion.

This paper asks a big question: Can computers learn to do the same thing?

The researchers from Oxford University and Shanghai Jiao Tong University wanted to see if modern AI models (the "smart" computers that can generate videos or answer questions) actually understand the rules of physics hidden inside a video, or if they are just guessing based on what things look like.

Here is a simple breakdown of what they did and what they found.

1. The Three "Physics Tests"

To test the AI, they created a new dataset called PhysVid. They didn't just throw random videos at the AI; they designed three specific "exams" based on how things move over time:

• The Bouncy Ball (Elasticity): They watched balls drop and bounce. A super-bouncy ball (like a superball) bounces high; a sad ball (like a lump of clay) barely bounces. The AI had to guess how "bouncy" the ball was just by watching the height of the jumps.
• The Sticky Liquid (Viscosity): They watched liquids pour onto a plate. Water spreads out fast; honey spreads out slowly. The AI had to guess how "thick" or "sticky" the liquid was based on how quickly it spread.
• The Sliding Toy (Friction): They watched objects slide across different surfaces. A toy sliding on ice goes far; the same toy sliding on a rug stops quickly. The AI had to guess how "slippery" the surface was based on how fast the object slowed down.

2. The Three AI "Students"

They tested three different types of AI models to see who could pass the exam:

• The "Generative" Artist (DynamiCrafter): This AI is trained to make videos. It knows how things should move because it has tried to create millions of realistic videos.
  • Analogy: Like a movie director who knows how a ball should bounce because they've directed thousands of action scenes.
• The "Self-Supervised" Observer (V-JEPA-2): This AI is trained by watching videos and trying to guess what happens next. It learns the "grammar" of motion without being told the rules.
  • Analogy: Like a baby watching the world, learning that if you drop a cup, it falls, without anyone explaining gravity.
• The "Multilingual" Chatbot (MLLMs like Gemini, GPT-4o): These are the famous chatbots that can see images and talk. The researchers asked them questions like, "How bouncy is this ball?" using different ways of phrasing the question (prompts).
  • Analogy: Like a very smart librarian who has read every book on physics but has never actually seen a ball bounce in real life.

3. The "Oracle" (The Cheat Sheet)

Before testing the AI, the researchers built a perfect "Oracle" system. This isn't an AI; it's a set of math rules and computer vision tools that can measure the exact physics (like measuring the exact height of a bounce) with 100% accuracy.

• Analogy: This is the teacher with the answer key and a laser ruler. It sets the "perfect score" that the AI students are trying to reach.

4. The Results: Who Passed?

• The Oracle: Got an A+ (obviously). It could measure the physics perfectly.
• The Generative & Self-Supervised Models: These did surprisingly well! They got mostly As and Bs.
  • They were great at predicting how bouncy a ball was or how thick a liquid was.
  • They struggled a bit with "friction" (sliding objects) because that requires understanding complex angles and how the camera moves, which is harder to guess.
  • Key Takeaway: These models have actually "learned" some physical intuition just by watching videos. They aren't just guessing; they understand the flow of time.
• The Chatbots (MLLMs): These did the worst. Even with special instructions (prompts) telling them how to look at the video, they often failed.
  • Analogy: The chatbots were like the librarian who knows the theory of friction but gets confused when looking at a real video. They often focused on what the object was (e.g., "It's a red ball") rather than how it moved.
  • However, when the researchers gave them "few-shot" examples (showing them a few solved problems first), they improved, but still couldn't beat the other models.

5. The Big Picture

The paper concludes that while AI is getting very good at understanding what is in a video (identifying objects, people, and scenes), it is still learning how to understand how things move physically.

• The Good News: Video generation models (like the ones that make deepfakes or AI movies) have accidentally learned a lot about physics. They know that water spreads and balls bounce because they've practiced making those videos.
• The Bad News: The "smart" chatbots that can talk and see are currently worse at physics than the video generators. They are great at language, but their "eyes" aren't quite tuned to the laws of physics yet.

In summary: If you want a robot to know how slippery a floor is so it doesn't fall, you should probably ask the video-generation AI, not the chatbot. The video AI has "seen" the world move; the chatbot has only "read" about it.

Technical Summary

1. Problem Definition

The paper addresses the challenge of inferring dynamic physical properties from raw video data. Unlike static appearance recognition, this task requires temporal reasoning to estimate properties that are not directly observable in a single frame but emerge through motion. The authors focus on three specific properties:

1. Elasticity: The coefficient of restitution of a bouncing object (ratio of rebound velocity to impact velocity).
2. Viscosity: The resistance to flow of a liquid, inferred from the rate of area expansion as it spreads on a surface.
3. Dynamic Friction: The coefficient of friction between a sliding object and a surface, inferred from the rate of deceleration.

The core difficulty lies in the need for the model to be sensitive to subtle temporal cues (deformation, deceleration, oscillation) while being robust to "nuisance parameters" such as camera viewpoint, lighting, object appearance, and initial conditions.

2. The PhysVid Dataset

To evaluate these capabilities, the authors introduce PhysVid, a new benchmark dataset containing both synthetic and real-world videos.

• Composition:
  • Synthetic Data: Generated using the Genesis physics simulator. It includes a training split and two test splits:
    • Test-1: In-distribution (same nuisance parameter distribution as training).
    • Test-2: Out-of-distribution (different camera viewpoints, lighting, and object colors).
  • Real-World Data (Test-3): Collected from the internet (YouTube) and in-house captures (iPhone slow-motion).
    • Elasticity: Various balls (tennis, rubber, balloons) dropped.
    • Viscosity: Liquids (coffee, syrup, oil) poured from a controlled funnel.
    • Friction: Objects (LEGO, wood, metal) sliding on various surfaces (towel, wood, cardboard).
• Ground Truth:
  • Synthetic videos have exact ground truth derived from simulation parameters.
  • Real-world videos have ground truth estimated via manual annotation (e.g., measuring bounce heights, liquid spread areas, or using spring dynamometers for friction).
• Task Formulations:
  • Absolute Prediction: Predicting the numerical value of the property ($y = \Phi(v)$).
  • Relative Comparison: Determining which of two videos exhibits a higher property value ($y = \Phi(v_1, v_2)$).

3. Methodology

The authors evaluate three distinct approaches to inferring these properties:

A. Oracle Method (Classical Computer Vision)

This serves as an upper bound, utilizing privileged access to visual cues.

• Elasticity: Tracks the ball's centroid trajectory, normalizes drop/bounce heights, and uses a GRU to regress the elasticity coefficient.
• Viscosity: Segments the liquid area over time, normalizes the growth rate, and calculates viscosity as the inverse of the area growth slope.
• Friction: Maps the sliding object to a bird's-eye view using homography, fits a parabola to the trajectory to estimate acceleration, and derives the friction coefficient ($\mu_k = a/g$).

B. Video Foundation Models (Generative & Self-Supervised)

The authors propose a lightweight readout mechanism to extract properties from frozen pre-trained models without retraining the backbone.

• Backbones:
  • Generative: DynamiCrafter (Video Diffusion Model).
  • Self-Supervised: V-JEPA-2 (Video Joint Embedding Predictive Architecture).
• Mechanism:
  1. Extract spatiotemporal features from the frozen backbone.
  2. Introduce a learnable query vector ($q$) that attends to feature tokens via cross-attention.
  3. Aggregate attended features via an MLP to produce a video representation ($P$).
  4. Train a simple MLP head on top of $P$ to predict the physical property (using L1 loss for absolute values or Cross-Entropy for relative comparisons).

C. Multimodal Large Language Models (MLLMs)

The authors explore prompting strategies for off-the-shelf MLLMs (Qwen2.5-VL, GPT-4o, Gemini 2.5 Pro) to perform the task directly.

• Strategies Tested:
  • Baseline: Simple query.
  • Oracle Estimation Teaching: Providing step-by-step instructions on how to calculate the property (e.g., "find the bounce peak, calculate the ratio").
  • Few-Shot Examples: Providing video-ground truth pairs.
  • Frame Indexing: Explicitly labeling frame numbers to aid temporal understanding.
  • Black Frames: Inserting black frames between video pairs in relative tasks to clearly separate inputs.

4. Key Results

Experiments were conducted on the PhysVid splits (Test-1, Test-2, Test-3).

• Oracle Performance: The classical CV oracle achieved near-perfect performance (ROC AUC $\approx$ 1.0, Pearson Corr $\approx$ 0.99) on synthetic data and strong performance on real data, confirming the task is solvable with visual cues.
• Foundation Models (Generative vs. Self-Supervised):
  • DynamiCrafter and V-JEPA-2 achieved generally similar, strong performance on synthetic data (Test-1/Test-2).
  • Generalization: Both models generalized well to real-world data for Elasticity and Viscosity. However, Friction proved significantly harder, especially for V-JEPA-2, likely due to the complexity of projective geometry and the lack of visual reference grids in real videos.
  • Absolute vs. Relative: Relative comparison was easier than absolute value prediction. The gap between foundation models and the oracle was larger in absolute prediction tasks.
• MLLM Performance:
  • MLLMs generally underperformed compared to the specialized video foundation models, particularly on synthetic data.
  • Prompting Impact: Performance improved significantly with Oracle Estimation Teaching (for relative tasks) and Few-Shot Examples (for absolute tasks).
  • MLLMs performed relatively better on real-world data (Test-3) than synthetic data, suggesting they rely more on semantic priors (e.g., "honey is thick") than visual motion dynamics.

5. Significance and Contributions

1. New Benchmark: PhysVid fills a gap in existing literature by providing quantitative ground truth for dynamic physical properties across both synthetic and real domains, enabling rigorous evaluation of generalization.
2. Efficient Readout: The proposed cross-attention readout mechanism demonstrates that pre-trained video foundation models already encode significant physical understanding, which can be unlocked with minimal trainable parameters.
3. Insight into Physical Reasoning: The study reveals that while current video foundation models can infer simple dynamic properties (elasticity, viscosity) effectively, they struggle with complex interactions involving friction and projective geometry.
4. MLLM Limitations: The results highlight that current MLLMs, despite their semantic strength, lack robust physical reasoning capabilities when relying solely on visual motion, often failing to outperform specialized vision models unless heavily guided by prompts.

Conclusion: The paper establishes that while video foundation models have acquired a "physical intuition" sufficient for relative comparisons and simple property estimation, achieving precise, absolute physical reasoning comparable to classical computer vision methods remains a significant challenge for future research.