Technical Summary: TailOR: Trimming the Long-Tail of Visual World Modeling Evaluation

Problem Statement

Physical interactions in the real world follow a long-tailed distribution. While a small set of common, regular interactions (e.g., hammering a nail, cutting bread) dominates human experience and visual training data, a vast spectrum of rare, irregular, yet physically valid interactions remains underrepresented. Recent world models (image and video generation models) have achieved impressive realism on existing benchmarks, but these evaluations largely operate under "head scenario" assumptions where objects behave in familiar ways and tools are used for their intended purposes.

This raises a critical question: Do current world models truly internalize the underlying physical principles of object interactions, or do they primarily rely on statistical regularities and co-occurrence patterns observed in training data? Existing benchmarks often fail to probe whether models can reason about object attributes (e.g., rigidity, torque, structural strength) when faced with unconventional or impossible tool-task pairings.

Methodology

To address this gap, the authors introduce TailOR, a benchmark designed to systematically evaluate world models under irregular physical interactions. The framework consists of three core components:

1. Benchmark Design and Scenario Modes

TailOR decomposes physical interactions into three progressively challenging scenario modes:

• Regular Scenarios (Head): Standard tool-task pairs frequently seen in training data (e.g., using a screwdriver to loosen a screw). These test in-distribution performance.
• Unconventional Scenarios (Long-Tail): Canonical tools are replaced with attribute-compatible substitutes that are atypical but physically valid (e.g., using a coin to loosen a screw because it shares rigidity and a flat edge). This tests affordance generalization beyond surface-level associations.
• Impossible Scenarios (Long-Tail): Tools are selected that violate critical physical attributes required for the task (e.g., using dry spaghetti to loosen a screw). These test constraint awareness and the model's ability to recognize and depict failure.

2. Evaluation Settings

To disentangle outcome prediction from instruction following, each scenario is evaluated under two complementary settings:

• Predictive Generation: The model must infer the outcome of an interaction without guidance. This tests implicit physical reasoning and the ability to predict success or failure based on object properties.
• Descriptive Generation: The desired outcome is explicitly specified in the prompt. This tests controllability and whether the model can faithfully realize a target state, even if it contradicts common training patterns.

3. Data Curation and Metrics

The dataset is constructed via a scalable pipeline involving:

• Sources: Action ontologies (HICO-DET) and object-affordance graphs (ConceptNet).
• Generation: LLMs generate tasks, unconventional substitutes, and impossible tools, followed by human verification to ensure physical plausibility and visual clarity.
• Metrics: Evaluation is conducted across four dimensions:
  • Instruction Adherence: Entity completeness, attribute fidelity, and scene validity.
  • Interaction Accuracy: State change correctness, affordance grounding, and motion plausibility (for video).
  • Physical Realism: Consistency with fundamental physical laws (gravity, contact mechanics).
  • Perceptual Quality: Visual fidelity and temporal consistency.
• Protocol: Both human annotation (by experts in visual content and physics) and automatic evaluation (using a VLM judge, gemini-2.5-pro) are employed.

Key Results

The authors evaluated state-of-the-art image models (Z-Image, Qwen-Image, GPT-Image-1, Nano-Banana-2) and video models (HunyuanVideo-1.5, Wan-2.2, Sora-2, Veo-3.1) on TailOR.

1. Systematic Long-Tail Gap: Performance consistently degrades from Regular to Unconventional and Impossible scenarios across all models and metrics. The most significant drops occur in Interaction Accuracy and Physical Realism, indicating that failures stem from weak affordance-level reasoning rather than mere perceptual issues.
2. Video vs. Image: Video generation models consistently underperform image models. The added complexity of modeling temporal dynamics (motion trajectories, force propagation) exacerbates failures, often leading to cascading inconsistencies where early errors propagate through the sequence.
3. Predictive vs. Descriptive:
   • In Predictive settings, models struggle to infer correct outcomes without guidance, often reverting to familiar patterns.
   • In Descriptive settings, even when the outcome is explicitly specified, models frequently ignore instructions to revert to high-frequency training patterns (e.g., generating a successful interaction for an impossible tool). This suggests a bias toward perceptual coherence over causal consistency.
4. Failure Modes:
   • Image Models: Failures often involve incorrect outcomes, inaccurate object attributes (e.g., wrong material properties), and physical violations (e.g., objects intersecting).
   • Video Models: Failures are dominated by implausible dynamics, temporal inconsistencies, and affordance misgeneralization (e.g., a soft object behaving rigidly).

Significance and Claims

The paper claims that TailOR exposes a fundamental limitation in current visual generative models: they largely memorize interaction templates rather than performing compositional physical reasoning.

• Internalization vs. Memorization: The results suggest that high performance on standard benchmarks does not equate to a genuine understanding of physical principles. Models rely on statistical co-occurrence rather than reasoning about why an interaction works (e.g., torque transmission, structural integrity).
• Need for New Architectures: The authors argue that future progress requires models that understand object attributes, causal dynamics, and constraint-aware interactions. They suggest that training pipelines need stronger inductive biases for physical primitives (rigidity, leverage) rather than holistic visual patterns.
• Benchmark Utility: TailOR serves as a diagnostic testbed to distinguish between models that have truly internalized world physics and those that merely reproduce the visual statistics of common experiences.

The paper concludes that while current models excel at simulating the "head" of the physical interaction distribution, they remain brittle when faced with the "long tail," highlighting the need for research into physically grounded world modeling.