Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models

1. Problem Statement

Despite significant advancements in image and video understanding, Multimodal Large Language Models (MLLMs) struggle with intuitive physics understanding, particularly regarding the dynamics of continuum objects (e.g., fluids, cloth, smoke).

• Root Causes: Current MLLMs are trained on datasets focused on human actions or treated as static frame sequences, failing to capture low-level temporal dynamics and physical laws governing continuous deformation.
• Evaluation Gap: Existing benchmarks (e.g., ContPhy, PhysBench) evaluate high-level reasoning that entangles physics with vision, language, and common sense. This "entanglement" makes it difficult to isolate whether a model fails due to a lack of physical understanding or deficiencies in other cognitive abilities.
• Empirical Findings: State-of-the-art (SOTA) MLLMs perform poorly on foundational physics tasks, often barely exceeding random guessing. Furthermore, simply scaling model parameters or using Chain-of-Thought (CoT) prompting yields only marginal improvements, suggesting a fundamental representational gap.

2. Methodology

The authors propose a two-pronged approach: first, isolating the problem through new benchmarks, and second, solving it via a novel training framework.

A. Benchmark Construction: Disentangling Physics

To isolate intuitive physics understanding, the authors introduce two low-level diagnostic tasks:

1. Next Frame Selection (NFS): Given a sequence of frames, the model must select the correct next frame from multiple distractors. This tests the ability to predict temporal evolution.
2. Temporal Coherence Verification (TCV): Given a video sequence with potential corrupted frames, the model must verify if the sequence is temporally consistent (Yes/No).

Data Preparation:

• The dataset combines synthetic simulations (from ContPhy, PhysBench) and real-world web-mined videos.
• A robust filtering pipeline uses an ensemble of MLLMs and human annotators to ensure high-quality fluid interaction clips.
• Distractors are generated via similarity-based pruning to ensure they are semantically distinct but plausible.

B. Proposed Solution: Scene Dynamic Field (SDF)

The core innovation is Scene Dynamic Field (SDF), a lightweight, representation-level bridge that injects physical knowledge into MLLMs without architectural overhaul.

• Concept: SDF leverages physics simulators (Blender with Flip Fluids) to generate visual prompts that explicitly visualize motion dynamics.
• Visual Representation: Instead of raw RGB, SDF maps particle velocity vectors to a blue-channel gradient.
  • Higher velocity magnitudes are encoded as deeper blue shades.
  • This creates a "perceptually calibrated" visual cue that highlights flow direction and speed, abstracting complex physics into a format MLLMs can process.
• Multi-Task Fine-Tuning Strategy: The authors propose a three-stage training framework:
  1. Dynamic Perception: The model learns to identify the correct SDF representation corresponding to an RGB video sequence (matching velocity fields to visual patterns).
  2. SDF-Guided Chain-of-Thought (CoT): SDF frames are inserted as visual prompts into the reasoning chain. The model is guided to analyze fluid dynamics, integrate the SDF visual cue, and then predict the next frame.
  3. Self-Distillation: To optimize reasoning, the model is fine-tuned using a mix of expert-generated reasoning data (from strong models like Gemini-2.5-Pro) and self-distilled data (1:10 ratio), encouraging the model to internalize the physical reasoning process.

3. Key Contributions

1. Novel Benchmarks (NFS & TCV): The first set of tasks specifically designed to disentangle and evaluate low-level intuitive physics understanding in MLLMs, revealing that current models fail at fundamental dynamic perception.
2. Scene Dynamic Field (SDF): A novel method that bridges the gap between physics simulators and MLLMs. It transforms abstract physical dynamics (velocity fields) into explicit visual prompts, enabling models to "see" physics rather than just infer it linguistically.
3. Comprehensive Empirical Analysis:
   • Demonstrated that model scaling (increasing parameters from 3B to 72B) yields diminishing returns on physics tasks.
   • Showed that language-only reasoning (CoT/Thinking modes) improves performance slightly but remains insufficient for capturing complex spatio-temporal relationships.
   • Validated that SDF provides a scalable, cost-effective path to physically grounded MLLMs.

4. Experimental Results

• Baseline Performance: SOTA open-source models (e.g., Qwen2.5-VL, InternVL) achieved only ~30% accuracy on NFS (4-choice) and ~55% on TCV (binary), barely above random chance.
• SDF Performance:
  • Fluid Tasks: SDF achieved up to 20.7% absolute improvement on NFS for fluid tasks compared to zero-shot baselines.
  • Generalization: Crucially, models fine-tuned only on fluid data with SDF showed strong zero-shot transfer to unseen physical domains (cloth, sand, smoke, plasticine). Standard fine-tuning failed to generalize, whereas SDF maintained significant gains, proving it learns underlying physical dynamics rather than domain-specific patterns.
• Ablation Studies:
  • Input Frames: Performance stabilizes after 5 frames; more frames do not significantly help.
  • Prompting: Detailed physical prompts help slightly, but SDF (visual prompting) is far superior.
  • Data Mixture: A 1:10 ratio of expert data to self-distilled data yielded optimal results.

5. Significance and Impact

• Paradigm Shift: The paper argues that intuitive physics is a foundational capability that must be established before higher-level reasoning can be effective. It advocates for "curriculum benchmarking" to evaluate models step-by-step.
• Cost-Effective Physics Grounding: By using physics simulators to generate synthetic visual cues (SDF), the authors provide a scalable alternative to collecting massive amounts of real-world physics data, which is expensive and difficult to label.
• Future Direction: The work highlights that current MLLM architectures are capable of learning physics if exposed to the right type of data (dynamic, physics-grounded representations), suggesting a shift from purely end-to-end training to data-centric approaches that explicitly model temporal dynamics.

In conclusion, Scene Dynamic Field successfully unlocks intuitive physics understanding in MLLMs by transforming abstract physical laws into explicit visual reasoning cues, significantly outperforming current scaling and prompting strategies.