💻 computer science

ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos

This paper introduces ProcObject-10K, the first benchmark designed to evaluate object-centric reasoning and temporal grounding in instructional videos, revealing that current multimodal models rely heavily on linguistic priors rather than fine-grained object dynamics while demonstrating that object-centric fine-tuning can effectively bridge this gap.

Original authors: Wenliang Guo, Yu Kong

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenliang Guo, Yu Kong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a cooking show. A standard AI might look at the video and say, "The chef is chopping onions, then frying them." It sees the actions (chopping, frying) but doesn't really understand what is happening to the ingredients. It doesn't realize that the onion changed from a solid, crunchy state to a soft, translucent one, or that if the chef had forgotten to add oil, the onions would burn instead of fry.

This paper, ProcObject-10K, argues that current AI is too focused on the "dance moves" (actions) and not enough on the "dancers" (objects) and how they change.

Here is a simple breakdown of what the researchers did:

1. The Problem: The "Blind" Chef

The authors say that existing AI benchmarks for instructional videos are like asking a student to describe a movie by only listing the plot points, without noticing how the characters' emotions or the setting changed.

The Gap: AI models can often guess the right answer to a question like "What happens next?" because they have read millions of recipes on the internet (linguistic priors). They know onions usually get fried.
The Failure: However, if you ask the AI to point to the exact moment in the video where the onion changed from raw to cooked, or to explain why a mistake happened (e.g., "The butter didn't melt because it was too cold"), the AI fails. It can't find the visual evidence. It's like a student who memorized the answer key but didn't read the textbook.

2. The Solution: A New "Gym" for AI (ProcObject-10K)

To fix this, the researchers built a new testing ground called ProcObject-10K. Think of this as a specialized gym designed to train AI to pay attention to objects and their state changes.

The Dataset: They gathered over 1,700 video clips (cooking, assembly, lab work) and created 10,500 questions.
The Questions: Instead of just asking "What did the chef do?", they ask:
- Precondition: "What had to happen to the egg before it could be whisked?" (It had to be cracked).
- State Evolution: "How did the egg change from the bowl to the microwave?" (It went from liquid to solid curd).
- Counterfactuals: "What would have happened if the garlic wasn't peeled?" (It would be chunky and bitter).
- Mistakes: "What went wrong with the butter?" (It stayed in chunks instead of melting).
The Evidence: Crucially, every answer must be backed up by specific time stamps in the video. The AI must prove where it saw the change.

3. The Test Results: The "Answering-Grounding Gap"

The researchers tested 13 of the smartest AI models available (like GPT-4, Gemini, and others) on this new gym.

The Result: The models were great at writing plausible answers (scoring well on language), but terrible at pointing to the video evidence.
The Metaphor: Imagine a detective who can write a perfect story about who committed the crime based on the news, but when asked to point to the security camera footage that proves it, they point to the wrong room.
The Stat: The models' ability to find the right video segment was below 45%. They were relying on their "common sense" (what they read on the internet) rather than actually watching the video.

4. The Fix: Teaching the AI to "Look"

To help the AI learn, the researchers created a special training method (Supervised Fine-Tuning).

The Method: They didn't just tell the AI the answer. They gave it "pseudo-labels"—fake but helpful hints showing exactly which objects (like the egg or the knife) were important and when they appeared.
The Analogy: It's like a teacher putting a highlighter on the specific sentences in a textbook that contain the answer, rather than just giving the student the answer key.
The Outcome: When they trained the AI with these "highlighters," the AI got much better at both answering the questions and finding the video evidence.

5. The Bonus: It Works Everywhere

The best part is that this new skill wasn't just for this specific test.

When they took the AI trained on this "object-centric" method and tested it on other video tasks it had never seen before, it performed better than models that were just trained on general video data.
It even helped the AI act as a "planner" for robots (embodied agents), helping them understand how to manipulate objects in the real world, not just watch videos.

Summary

The paper claims that to truly understand instructional videos, AI needs to stop just watching the "actions" and start tracking the "objects" and how they change over time. They built a new test (ProcObject-10K) to prove that current AI is "blind" to these changes, and they showed that a specific training method can fix this, making AI smarter at understanding cause-and-effect in the real world.

Technical Summary: ProcObject-10K

Problem Statement

Current benchmarks for procedural understanding in instructional videos are predominantly action-centric. They model procedural structure as sequences of actions or events, often overlooking the fundamental role of object state transitions. In real-world procedural tasks (e.g., cooking, assembly), the goal is achieved through the progressive transformation of object states under human interaction. The same action can yield different outcomes depending on the object's initial state, which in turn dictates valid subsequent actions.

Existing datasets fail to evaluate whether models can reason about these object-level dynamics. While some datasets focus on object tracking, they lack goal-driven procedural structure; conversely, instructional video datasets focus on action sequences without modeling the causal dependencies induced by object state changes. Consequently, current benchmarks allow models to achieve high performance by relying on linguistic priors and action patterns rather than performing fine-grained reasoning about how objects evolve to complete a task.

Methodology

1. The PROCOBJECT-10K Benchmark

The authors introduce PROCOBJECT-10K, the first benchmark designed to jointly evaluate object-centric reasoning and temporal evidence grounding in instructional videos.

Dataset Composition: The dataset comprises 10,522 open-ended VideoQA pairs grounded in 1,799 video clips spanning 137 tasks across 9 domains (including egocentric and exocentric views).
Data Generation Pipeline: A four-stage semi-automated pipeline was employed:
1. Collection: Videos were curated from existing datasets (CaptainCook4D, EgoPER, HoloAssist, COIN) known for complex hand-object interactions.
2. Sampling: Temporally contiguous action sequences were sampled to ensure causal coherence and consistent object appearance.
3. QA Generation: Vision-Language Models (VLMs) were prompted with dense object-centric captions and sparse frames to generate questions, answers, and supporting temporal evidence spans.
4. Verification: A rigorous filtering process removed instances solvable by commonsense alone (text-only), verified answer-evidence alignment via LLMs, and included human review to refine ambiguity and ensure temporal grounding.
Question Taxonomy: The benchmark defines five reasoning types centered on object dynamics:
1. Precondition Grounding: Identifying necessary object states before an action.
2. Object State Evolution: Tracking how an object changes between steps.
3. Counterfactual Reasoning: Predicting outcomes if specific object states were altered.
4. Mistake Recognition: Identifying errors in object handling or state.
5. Readiness Assessment: Determining if an object is in the correct state for the next step.

2. Object-Centric Supervised Fine-Tuning (SFT)

To address the identified reasoning gaps, the authors propose an SFT framework that explicitly encourages models to attend to the dynamic evolution of action-relevant objects.

Pseudo-Label Construction: Instead of manual annotation, the framework uses a VLM to extract key object phrases from QA pairs, which are then localized in video frames using an object detection model. This generates:
- Soft Patch-level Masks: Spatial supervision indicating overlap between image patches and detected objects.
- Object Presence Indicators: Temporal supervision indicating which frames contain relevant objects.
Training Objectives: The standard generative loss ( $L_{gen}$ $L_{g e n}$ ) is augmented with two auxiliary constraints:
- Spatial Constraint ( $L_{spl}$ ): Encourages the model to focus on image patches overlapping with supporting objects.
- Temporal Constraint ( $L_{tmp}$ ): Encourages the model to focus on specific frames where key object dynamics occur.
Inference: The auxiliary heads are removed during inference to maintain efficiency, leaving a model with learned object-centric visual representations.

Key Results

1. The Answering-Grounding Gap

Benchmarking 13 leading Multimodal Large Language Models (MLLMs) revealed a substantial discrepancy:

Plausible Answers, Poor Grounding: Models achieved reasonable scores in open-ended question answering (relying on linguistic priors) but failed to localize the supporting temporal evidence.
Low Grounding Performance: The mean Intersection-over-Union (mIoU) for temporal grounding remained consistently below 45% across all models, even for large-scale proprietary and open-source models.
Blind LLM Performance: Language-only models (e.g., GPT-5.4-Mini) achieved competitive answering scores (LLM-as-Judge score ~3.14) without video input, confirming that current models often bypass visual evidence.

2. Impact of Object-Centric SFT

The model fine-tuned with the proposed object-centric SFT framework (Qwen3-VL-4B) demonstrated significant improvements:

Benchmark Performance: It achieved the highest performance across most metrics, surpassing much larger baselines (e.g., 38B parameter models) in both answering and grounding accuracy.
Generalization:
- VideoQA: The fine-tuned model transferred effectively to the MultiHop-EgoQA benchmark in a zero-shot setting, outperforming models specifically fine-tuned on that dataset.
- Embodied Planning: When integrated as a zero-shot planner in the ALFRED framework, the fine-tuned model improved Success Rate (SR) and Goal Conditioned (GC) scores compared to zero-shot baselines, indicating that object-centric reasoning aids embodied task execution.

Significance and Claims

The paper claims that PROCOBJECT-10K serves as a critical diagnostic tool, exposing the reliance of current MLLMs on linguistic priors rather than fine-grained object dynamics. The authors argue that:

Procedural understanding requires object-centric reasoning: Action sequences alone are insufficient; understanding the causal dependencies of object state transitions is fundamental for intelligent systems assisting in procedural tasks.
Explicit supervision is necessary: The proposed SFT framework demonstrates that explicitly supervising models on object state dynamics (via pseudo-labels and spatial-temporal constraints) is an effective strategy to close the answering-grounding gap.
Transferability: The object-centric representations learned from this benchmark are not limited to the dataset itself but generalize to other grounded VideoQA tasks and embodied planning scenarios, suggesting broader value for procedural understanding in AI.

The authors conclude that PROCOBJECT-10K aims to catalyze future research into object-centric procedural reasoning, tighter integration with embodied agents, and the development of world models that better understand physical state transitions.