EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

This paper introduces EXPLORE-Bench, a benchmark derived from real first-person videos to evaluate the ability of multimodal large language models to perform long-horizon egocentric scene prediction, revealing significant performance gaps compared to humans and demonstrating that stepwise reasoning offers partial improvements at a computational cost.

The Gist

Imagine you are watching a cooking show, but instead of just watching, you are the chef. You have a clear view of your kitchen counter (the initial scene). Then, someone hands you a long, detailed list of instructions: "Crack the egg, whisk it, pour it into the pan, turn on the stove, add salt..." (the atomic actions).

Now, imagine you have to close your eyes and mentally run through every single one of those steps in your head. When you open your eyes, you must describe exactly what the kitchen looks like. Is the pan smoking? Is the egg in the bowl or the pan? Did the salt spill?

This is exactly what the paper EXPLORE-Bench is testing, but with robots and AI instead of humans.

Here is a breakdown of the paper in simple terms:

1. The Big Problem: AI is Bad at "What If?"

Current AI models (Multimodal Large Language Models) are great at looking at a picture and saying, "That's a cat." They are also good at answering questions like, "What is the cat doing?"

But they struggle with Long-Horizon Reasoning. This is a fancy way of saying: "If I do A, then B, then C, what does the world look like after all of that?"

Think of it like a game of Jenga.

• Short-term AI: Can tell you, "That block is red."
• Long-term AI (The Goal): Needs to know that if you pull the bottom block out (Action A), the tower will wobble (Action B), and eventually, the whole thing will crash down (Final Scene).
• The Reality: Most current AIs are like a person who sees the block being pulled but thinks, "Oh, the tower is still standing!" They fail to predict the crash.

2. The Solution: EXPLORE-Bench

The researchers built a new "exam" called EXPLORE-Bench to test how good AI is at this specific skill.

• The Test: They give the AI a photo of a starting scene (like a kitchen counter) and a long list of actions (like a recipe or a chore list).
• The Task: The AI must describe the final scene after all those actions happen.
• The Twist: They don't just ask for a vague description. They check the AI's answer against a "Gold Standard" list that includes:
  • Objects: Is the egg there? Is the pan there?
  • Attributes: Is the egg broken? Is the pan hot?
  • Relations: Is the egg inside the pan, or is it on the floor?

They collected over 1,000 real-life videos (from cooking to bike repair) to make sure the test is realistic, not just made-up computer graphics.

3. The Results: AI is Still a Novice Chef

The researchers tested many of the smartest AI models available today (including big names like GPT-5, Gemini, and open-source models).

• The Score: Humans scored about 59/100. The best AI models scored around 49/100.
• The Gap: While 10 points might not sound like much, in the world of AI, it's a huge canyon. The AI often gets the "big picture" wrong. It might forget that an object was moved, or it might imagine an object disappearing when it should have stayed.
• The "Abnormal" Trap: The test included tricky scenarios, like a faucet left running or a tower of bottles about to fall. Humans are great at spotting these dangers because of common sense. The AI, however, often missed them completely, describing a calm scene when chaos was actually happening.

4. The "Step-by-Step" Strategy

The researchers tried to help the AI by breaking the long list of actions into smaller chunks (like telling a story one chapter at a time instead of all at once).

• The Analogy: Imagine trying to remember a 100-page story. It's hard. But if you read 10 pages, summarize them, then read the next 10, it's easier.
• The Result: This "step-by-step" thinking did help the AI a little bit, but it came with a cost. It made the AI much slower and required much more computer power (like asking a friend to help you solve a puzzle, but they take three times as long to do it).

5. Why Does This Matter?

You might ask, "Why do we care if an AI can predict a messy kitchen?"

Because this is the foundation of Embodied AI—robots that live in our world.

• If a robot is going to help you cook, it needs to know that if it knocks over the milk, the floor will be wet and slippery (a safety hazard).
• If a robot is going to clean your house, it needs to know that moving a heavy box might knock over a vase.

The Bottom Line:
Today's AI is like a very smart tourist who can describe a city they are looking at right now. But it is not yet a good planner who can predict how the city will change if they build a new road or move a building. EXPLORE-Bench is the map that shows us exactly where these robots need to get smarter before we can trust them to handle our real-world tasks.

Technical Summary

Here is a detailed technical summary of the paper "EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning".

1. Problem Definition

The paper addresses a critical gap in the capabilities of Multimodal Large Language Models (MLLMs) regarding embodied agents. While MLLMs are increasingly used as foundations for agents, their ability to perform long-horizon reasoning from an egocentric (first-person) viewpoint remains unverified.

• Core Task: Egocentric Scene Prediction with Long-Horizon Reasoning.
  • Input: An initial-scene image and a long sequence of atomic action descriptions (e.g., "C cracks the egg," "C places the skillet on the stove").
  • Output: A detailed prediction of the final scene state after all actions are executed.
• The Challenge: Current models struggle to track causal chains over long sequences, maintain coherent object states, and anticipate complex physical consequences (e.g., a stack of objects collapsing or a faucet left running). Existing benchmarks focus on short-term state changes or multiple-choice questions, lacking systematic evaluation for holistic, long-term scene evolution.

2. Methodology

A. Dataset Construction: EXPLORE-Bench

The authors constructed EXPLORE-Bench, a benchmark containing 1,157 instances derived from real first-person videos (sourced from Ego4D, Ego-Exo4D, and self-recorded data).

• Data Characteristics:
  • Action Length: Sequences range from 11 to 694 atomic actions (average 113), significantly longer than existing benchmarks.
  • Scenarios: Diverse daily activities (cooking, repair, cleaning) covering both normal and abnormal (safety hazards, environmental damage) outcomes.
• Annotation Pipeline:
  • A scalable, multi-step pipeline was developed to generate structured ground truth for the final scene.
  • Components: Object categories, visual attributes (color, texture, state), and inter-object relations (spatial and interaction).
  • Quality Control: Utilized a combination of LLMs (RAM++, Grounding DINO, Qwen3-VL, GPT-5.2) for initial tagging and grounding, followed by rigorous human-in-the-loop verification and correction to ensure 99% accuracy.

B. Evaluation Protocol

To enable fine-grained, quantitative assessment, the authors proposed a three-dimensional evaluation metric:

1. Object-level Coverage ($S_{obj}$): Measures the recall of objects present in the predicted description vs. ground truth using Sentence-BERT embeddings.
2. Attribute Accuracy ($S_{att}$): Scores the correctness of visual attributes (shape, color, state) on a 0–5 scale via LLM-based scoring.
3. Relation Accuracy ($S_{rel}$): Scores the correctness of spatial and interaction relations.

• Unified Score ($S_{uni}$): A weighted average of the three metrics (weights: 0.25, 0.35, 0.40 respectively) scaled to 0–100.

C. Inference Strategies (Test-Time Scaling)

The paper investigates stepwise reasoning to improve performance on long sequences:

• Single-turn Inference: Decomposing actions into segments and predicting intermediate states in a single pass. Results showed this often led to performance drops as models ignored unchanged parts of the scene.
• Multi-turn Inference: Iteratively processing action segments, feeding the previous scene description back into the model. This approach yielded better results for long sequences but incurred significant computational overhead.

3. Key Contributions

1. New Task Formulation: Defined and formalized "Egocentric Scene Prediction with Long-Horizon Reasoning," shifting focus from short-term state changes to holistic, long-term consequence anticipation.
2. EXPLORE-Bench: Introduced a high-quality benchmark with 1,157 instances featuring structured, fine-grained annotations (objects, attributes, relations) derived from real-world videos.
3. Comprehensive Benchmarking: Evaluated a wide range of proprietary (GPT-5.2, Gemini-3) and open-source (Qwen, InternVL, LLaVA) MLLMs, revealing a substantial performance gap compared to humans.
4. Analysis of Reasoning Strategies: Demonstrated that while decomposing long action sequences (stepwise reasoning) can improve performance, it introduces non-trivial computational costs and does not always guarantee better results without careful tuning.
5. Abnormal Case Analysis: Highlighted that current models perform particularly poorly on "abnormal" scenarios (e.g., safety hazards), failing to detect critical states that humans perceive easily.

4. Experimental Results

• Human vs. Model Performance:
  • Human participants achieved a unified score ($S_{uni}$) of 59.08 on the full dataset.
  • The best-performing proprietary model (Gemini-3-Pro) scored 49.66, and the best open-source model (Qwen3-VL-8B-Thinking) scored 50.96.
  • Gap: There is a significant gap (~7–10 points) between the best models and human performance, indicating that long-horizon reasoning remains a major unsolved challenge.
• Model Trends:
  • Proprietary vs. Open-source: Proprietary models generally outperformed open-source ones, though Qwen3-VL-8B showed competitive results.
  • Thinking Models: "Thinking" (Chain-of-Thought) variants did not consistently outperform their non-thinking counterparts across all model families, suggesting that reasoning capabilities are not automatically transferred to long-horizon tasks.
  • Abnormal Cases: Models struggled significantly with abnormal outcomes (e.g., a bottle falling, a faucet running). While humans scored 91.64 on abnormal cases, the best model (GPT-5.2-Chat) only reached 62.79.
• Inference Strategy Findings:
  • Multi-turn inference with small window sizes (e.g., 10 actions per segment) improved performance on long sequences by ~3.4 points compared to default settings but increased inference time multiplicatively.
  • Single-turn inference with decomposition generally degraded performance, as models failed to maintain context for unchanged objects.

5. Significance and Future Work

• Embodied AI Safety: The benchmark highlights a critical safety gap: current MLLMs cannot reliably predict negative consequences of long action sequences (e.g., knocking over a stack, leaving water running). This is a prerequisite for deploying safe embodied agents in the real world.
• Research Direction: The paper establishes a principled testbed for advancing consequence-aware reasoning. It suggests that future work must focus on improving long-term state tracking and developing more efficient test-time scaling methods that do not rely solely on brute-force multi-turn inference.
• Community Impact: By providing structured annotations and a clear evaluation protocol, EXPLORE-Bench enables systematic progress tracking in egocentric vision and planning, moving beyond coarse textual similarity metrics.

In summary, EXPLORE-Bench exposes the limitations of current MLLMs in simulating the physical world over long horizons and provides the necessary tools and metrics to drive the development of safer, more capable embodied agents.