OSCBench: Benchmarking Object State Change in Text-to-Video Generation

This paper introduces OSCBench, a novel benchmark derived from instructional cooking data that evaluates the ability of text-to-video models to generate accurate and temporally consistent object state changes, revealing that current models struggle significantly with this capability despite their progress in other areas.

The Gist

Imagine you are teaching a robot chef how to cook. You give it a simple instruction: "Peel and slice this potato."

If the robot chef is really good, it will show you a video where the potato starts whole, gets peeled, and ends up as neat slices. But what if the robot chef is just "pretending"? It might show a potato that looks like it's being peeled, but the skin never actually comes off, or the potato magically turns into a slice without ever being cut. Or worse, it might peel a carrot instead of a potato because it's confused.

This is exactly the problem the paper OSCBench is trying to solve.

The Big Idea: The "Magic Trick" Test

Current AI video generators (Text-to-Video models) are amazing at making things look pretty. If you ask for a video of a chef in a kitchen, the lighting, the chef's apron, and the kitchen background usually look perfect.

However, these AIs are terrible at Object State Change (OSC).

• State Change means an object actually transforming from one thing to another (e.g., Whole Lemon $\rightarrow$ Sliced Lemon).
• The Problem: The AIs are great at the "setup" (the chef, the knife, the kitchen) but fail at the "magic trick" (the actual cutting). They often produce videos where the object stays whole, disappears, or changes into something weird.

The authors built a new test called OSCBench to specifically check if these AIs can handle the "magic trick" of changing an object's state.

How They Built the Test (The Recipe Book)

To test this, the researchers didn't just make up random ideas. They looked at thousands of real cooking videos (like "HowTo100M") because cooking is full of clear state changes: chopping, peeling, frying, and mixing.

They organized their test into three levels of difficulty, like a video game:

1. Regular Level (The Basics):
   • Example: "Slice a lemon."
   • Why: This is common. The AI has probably seen this a million times in its training data. It's like asking a human to tie their shoes.
2. Novel Level (The Twist):
   • Example: "Peel a berry."
   • Why: You don't usually peel berries! The AI can't just "memorize" the answer. It has to understand what "peeling" means and apply it to a new object. This is like asking a human to tie a knot they've never seen before.
3. Compositional Level (The Combo):
   • Example: "Peel and then slice a pear."
   • Why: This requires doing two things in a row, where the second step depends on the first one being done right. It's like asking a human to peel a potato, then slice it, without dropping it or forgetting the knife.

The Results: The "Uncanny Valley" of Cooking

The researchers tested six top-tier AI video models (some free and open, some paid and secret). Here is what they found:

• The Good News: The AIs are fantastic at the "decor." They get the chef, the kitchen, and the lighting right.
• The Bad News: They are terrible at the "cooking."
  • In many videos, the knife passes through the fruit without cutting it.
  • Sometimes the fruit turns into slices instantly without the action happening.
  • In the "Combo" level, the AI often forgets the first step (peeling) and just does the second (slicing), or vice versa.

The Analogy: Imagine a movie special effects team. They can build a perfect set and dress the actors perfectly. But when the actor is supposed to eat an apple, the AI makes the apple disappear or turn into a rock. The movie looks great, but the physics are broken.

How Did They Grade the AIs?

They used two methods:

1. Human Judges: Real people watched the videos and gave them scores.
2. AI Judges (MLLMs): They used a super-smart AI (like GPT-5) to watch the videos. But they didn't just ask the AI "Is this good?" They gave it a Checklist (Chain-of-Thought).
   • Step 1: "Look at the lemon. Is it whole?"
   • Step 2: "Look at the next frame. Is it sliced?"
   • Step 3: "Did the transition happen smoothly, or did it jump?"
   • Step 4: "Give a score."

They found that the AI judges were surprisingly good at spotting these errors, almost as good as the humans. This is huge because it means we can automatically test future AIs without hiring hundreds of people.

The Takeaway

The paper concludes that while AI video generation is getting visually stunning, it still lacks a deep understanding of cause and effect. It knows what a "knife" looks like, but it doesn't fully understand that a knife cuts things.

OSCBench is like a new driving test for AI. Before, we just checked if the car looked nice. Now, we are checking if the car can actually stop at a red light and turn left without crashing. Until AIs pass this test, we can't fully trust them to simulate real-world actions, like helping robots in factories or creating realistic instructional videos.

In short: The AI can draw a beautiful picture of a chef, but it can't yet make the chef actually cook the meal.

Technical Summary

1. Problem Statement

While Text-to-Video (T2V) generation models have achieved significant progress in visual fidelity and temporal coherence, existing benchmarks primarily focus on perceptual quality, text-video alignment, and physical plausibility (e.g., gravity, collisions). They largely overlook a critical dimension of action understanding: Object State Change (OSC).

OSC refers to the transformation of an object's state induced by a specific action (e.g., a whole lemon becoming sliced, or dough being kneaded). Current T2V models often fail to faithfully realize these state transitions, producing videos that look realistic at a glance but exhibit:

• Incorrect state changes: Objects transform into implausible states.
• Inconsistency: State transitions are not smooth over time (e.g., an object reverts to its original state).
• Misunderstanding of action semantics: The model performs the wrong action or fails to infer the intended outcome from the prompt.

This gap hinders the deployment of T2V models in downstream applications like robotics, embodied AI, and instructional video synthesis, where accurate state modeling is essential.

2. Methodology: OSCBench Construction

The authors introduce OSCBench, a benchmark specifically designed to evaluate OSC performance. It is constructed from instructional cooking data (based on the HowToChange dataset) and organized into three distinct evaluation regimes to test different cognitive capabilities:

• Regular Scenarios (108 scenarios): Covers common, frequent action-object pairs (e.g., "slicing a lemon"). These test in-distribution performance and memorization.
• Novel Scenarios (20 scenarios): Features uncommon but feasible action-object pairs (e.g., "peeling a berry"). These test the model's ability to generalize and reason about state changes without relying on frequent training patterns.
• Compositional Scenarios (12 scenarios): Involves multiple sequential actions (e.g., "peeling and slicing a pear"). These test the model's ability to maintain coherent intermediate and final states over time.

Dataset Statistics:

• Total Prompts: 1,120 prompts across 140 unique object-state scenarios.
• Prompt Structure: <subject> <action> <object> <scene>.
• Variations: Includes minimal prompts (<action> <object>) to test reliance on contextual cues.

Evaluation Framework:
The benchmark employs a dual-evaluation approach:

1. Human User Study: Three human experts rate videos on a 1-5 Likert scale across four dimensions:
   • Semantic Adherence: Subject, Object, and Action alignment.
   • Object State Change: Accuracy (reaching the target state) and Consistency (smooth temporal evolution).
   • Scene Alignment: Background context.
   • Perceptual Quality: Realism and Aesthetics.
2. MLLM-Based Automatic Evaluation: Uses Multimodal Large Language Models (specifically GPT-5.2, Qwen3-VL-30B, and GPT-5-mini) with a Chain-of-Thought (CoT) strategy.
   • Process: The MLLM is guided to (1) restate criteria, (2) extract frame-level visual evidence, and (3) assign scores with justification. This avoids treating MLLMs as black-box scorers and improves reliability.

3. Key Contributions

1. First OSC-Specific Benchmark: OSCBench is the first benchmark explicitly designed to evaluate object state changes in T2V generation across regular, novel, and compositional scenarios.
2. Comprehensive Evaluation Protocol: It introduces a multi-dimensional evaluation framework combining human studies and CoT-guided MLLM assessment, covering semantic adherence, OSC performance, scene alignment, and perceptual quality.
3. Systematic Benchmarking of SOTA Models: The paper evaluates six state-of-the-art models (4 open-source: Open-Sora-2.0, HunyuanVideo, HunyuanVideo-1.5, Wan-2.2; and 2 proprietary: Kling-2.5-Turbo, Veo-3.1-Fast), providing a diagnostic analysis of their strengths and weaknesses.

4. Key Results and Analysis

The evaluation reveals a significant discrepancy between high-level semantic alignment and low-level state change modeling:

• Performance Gap: While models perform well on Semantic Adherence (Subject/Object presence) and Scene Alignment, they consistently struggle with Object State Change Accuracy and Consistency.
  • Example: In human evaluations, proprietary models like Veo-3.1-Fast scored 0.79 on OSC accuracy, while open-source models scored significantly lower (0.38–0.57).
• Generalization Challenges:
  • Novel Scenarios: Models show the most severe performance degradation here, indicating an inability to reason about unseen action-object combinations.
  • Compositional Scenarios: Performance is better than novel but worse than regular, highlighting difficulties in maintaining temporal coherence across multiple actions.
• Action Complexity: Models handle simple, visually salient actions (e.g., heating, rolling) better than complex hand-object interactions (e.g., peeling, mashing) which require precise manipulation and subtle visual transitions.
• Evaluation Reliability:
  • MLLMs using Chain-of-Thought (specifically GPT-5.2) show strong correlation with human judgments, particularly for OSC accuracy.
  • Standard similarity models (like ViCLIP) fail to capture fine-grained state changes.
  • MLLMs without CoT tend to over-score or miss subtle inconsistencies (e.g., a lemon dripping juice without deformation).

5. Significance

• Diagnostic Tool: OSCBench identifies Object State Change as a critical bottleneck in current T2V generation. It reveals that models are currently "motion generators" rather than "state-aware simulators."
• Guidance for Future Research: The findings suggest that future models need architectures or training objectives that explicitly model causal relationships between actions and object states, rather than just optimizing for visual coherence.
• Automated Evaluation Standard: The paper validates that CoT-guided MLLMs can serve as reliable, scalable proxies for human evaluation in assessing complex video generation tasks, reducing the cost of benchmarking.
• Application Impact: By highlighting failures in state modeling, the work provides a roadmap for improving T2V models in fields requiring precise physical simulation, such as robotic instruction and educational content creation.

In conclusion, OSCBench establishes that while T2V models have mastered the "look" of video, they have not yet mastered the "logic" of physical transformation, marking a new frontier for the next generation of video generation models.