💻 computer science

Target-Bench: Can Video World Models Achieve Mapless Path Planning with Semantic Targets?

This paper introduces Target-Bench, a novel benchmark comprising 450 robot-collected scenarios designed to evaluate video world models' semantic reasoning and mapless path planning capabilities, revealing a significant gap between visual realism and planning performance while demonstrating that fine-tuning on real-world data can substantially improve task-level results.

Original authors: Dingrui Wang, Zhihao Liang, Hongyuan Ye, Zhexiao Sun, Zhaowei Lu, Yuchen Zhang, Yuyu Zhao, Yuan Gao, Marvin Seegert, Finn Schäfer, Haotong Qin, Wei Li, Luigi Palmieri, Felix Jahncke, Mattia Piccinini

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dingrui Wang, Zhihao Liang, Hongyuan Ye, Zhexiao Sun, Zhaowei Lu, Yuchen Zhang, Yuyu Zhao, Yuan Gao, Marvin Seegert, Finn Schäfer, Haotong Qin, Wei Li, Luigi Palmieri, Felix Jahncke, Mattia Piccinini, Johannes Betz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented artist who can paint incredibly realistic pictures of the future. If you show them a photo of a living room and say, "Draw what happens next," they might paint a cat jumping onto a sofa, a ball rolling across the floor, and a person walking through the door. The picture looks perfect, the lighting is right, and the physics seem real.

But here's the catch: If you asked that artist to actually walk through the room without bumping into the furniture, could they do it?

This is the exact problem researchers are tackling with a new project called Target-Bench.

The Big Idea: From "Watching" to "Walking"

For a long time, AI video models (like the ones that make deepfakes or generate movie scenes) have gotten amazing at looking real. They can predict how a scene evolves visually. But the robotics world wants to use these models to help robots act.

The researchers asked: If an AI can predict a video of a robot walking toward a specific object (like a red chair), can we actually use that video to tell a real robot how to move?

To find out, they built Target-Bench, which is like a driving test for AI video models.

The Test Drive: Target-Bench

Think of Target-Bench as a giant obstacle course with 450 different scenarios.

The Setup: They used a real robot dog (a quadruped) to walk through real houses and parks.
The Goal: The robot was told to go to specific things, like "the blue car," "the open door," or even tricky things like "the place where you'd sit to rest" (an implicit goal).
The Challenge: The researchers took these real videos and fed them into various AI video models. The models tried to guess what the next few seconds of the video would look like if the robot kept moving toward that goal.

The "Decoder" Translator

Here is the clever part. The AI models just output a video file. A robot can't drive on a video file; it needs coordinates (turn left, go forward 2 meters).

The researchers built a special tool called a World-Decoder. Imagine this as a translator that watches the AI's generated video and says, "Okay, in this frame, the camera moved 1 meter forward. In the next, it turned slightly right." It turns the visual prediction back into a physical path.

The Scorecard: It's Not About Perfect Lines

In the past, people judged these models by asking, "Does the generated video look exactly like the real video?"

Target-Bench changes the rules. It asks: "Did the robot head in the right direction?"

They use a scoring system that cares more about tendency than perfection.

The Analogy: Imagine you are trying to walk to a coffee shop.
- Old Way: Did you walk on the exact same sidewalk tiles as the person before you? (Too strict!)
- Target-Bench Way: Did you generally walk toward the coffee shop without walking into a wall or turning around? (Practical!)

They measure things like:

Did you get close? (Endpoint Score)
Did you stay on the right path? (Approach Consistency)
Did you wander off too far? (Miss Rate)

The Results: The "Uncanny Valley" of Planning

The results were a bit of a reality check.

The Visuals: The best AI models (like Sora 2 or Wan2.2) generated beautiful, realistic videos.
The Planning: When they tried to turn those videos into a path for a robot, the scores were surprisingly low. The best model only got a 34% score.

What does this mean? It means the AI is great at imagining the future, but it's still bad at planning for it. It's like a movie director who can visualize a perfect chase scene but doesn't know how to drive a car.

The Good News: A Little Training Goes a Long Way

The researchers then tried something cool. They took one of the open-source AI models and gave it a "crash course" using just a small amount of real robot data (325 videos).

The Result: After this tiny bit of training, the model's planning score jumped from 8% to nearly 40%.

The Metaphor: It's like taking a person who has only watched thousands of driving movies and giving them 10 hours of actual driving lessons. Suddenly, they aren't just a movie critic; they can actually drive the car.

Why This Matters

This paper is a wake-up call and a roadmap.

Wake-up Call: Just because an AI can make a pretty video doesn't mean it understands how to navigate the real world.
Roadmap: We don't need massive supercomputers to fix this. A little bit of real-world data can teach these "movie makers" how to become "drivers."

In the future, this could mean robots that can look at a messy room, understand where the "clean spot" is, and figure out how to walk there without a pre-made map, just by "imagining" the path first.

1. Problem Statement

While recent Video World Models (WMs) have demonstrated remarkable capabilities in generating photorealistic videos and simulating world evolution (e.g., Sora, Veo), their ability to perform semantic reasoning and task-level planning remains unquantified.

The Gap: Existing benchmarks focus primarily on visual fidelity, spatio-temporal consistency, or physics simulation. They lack a framework to evaluate whether a model can interpret a natural language semantic goal (e.g., "go to the red chair") and generate a future visual sequence that implies a valid, metric-scale path toward that goal in an unstructured, mapless environment.
The Challenge: Assessing planning requires more than just visual quality; it requires extracting a geometric trajectory from generated video and verifying if that trajectory aligns with the semantic intent and physical reality.

2. Methodology: Target-Bench

The authors introduce Target-Bench, the first benchmark designed to evaluate video WMs on mapless path planning toward semantic targets. The framework consists of three core components:

A. The Target Dataset

Collection: 450 real-world scenarios collected using a quadruped robot (DEEP Robotics Lite 3) equipped with LiDAR, stereo RGB cameras, and IMU.
Scale: 112,500 frames spanning 47 semantic categories (e.g., cars, bicycles) in diverse indoor and outdoor environments.
Annotations:
- Semantic Targets: Defined via natural language prompts, categorized as Explicit (direct object names) or Implicit (describing attributes/functions).
- Ground Truth: SLAM-based metric-scale trajectories and point cloud maps.
- Data Split: 125 samples for benchmarking (diverse scenes/motion trends) and 325 for training/fine-tuning.

B. The Evaluation Pipeline

The benchmark operates in two stages:

World-Decoder (Spatio-Temporal Reconstruction):
- Since WMs generate video, not trajectories, the system uses state-of-the-art 3D reconstruction methods (VGGT, SpaTracker, ViPE) to extract camera poses from the generated future frames.
- Scale Recovery: Monocular methods (VGGT, SpaTracker) output scale-ambiguous trajectories. The authors implement a lightweight mechanism to recover metric scale by anchoring predictions to the real displacement of the first and last frames of the ground truth segment.
Path Evaluation Metrics:
The system compares the reconstructed path against the ground truth using five complementary metrics:
- ADE (Average Displacement Error): Average L2 distance over time.
- FDE (Final Displacement Error): Distance of the final predicted position from the target.
- MR (Miss Rate): Percentage of points deviating beyond a threshold (2.0m).
- SE (Soft Endpoint): A Gaussian penalty measuring proximity to the target (focuses on goal-reaching).
- AC (Approach Consistency): Evaluates if the trajectory stays within a "corridor" around the ground truth path, allowing for non-unique valid paths (focuses on directional tendency rather than exact overlap).
- WO (Weighted Overall Score): A unified score (0–1) prioritizing SE and AC to emphasize task success and directional consistency.

3. Key Contributions

First Systematic Benchmark: Target-Bench is the first framework to holistically evaluate video WMs on mapless path planning with textual semantic goals, moving beyond visual quality to functional utility.
Novel Dataset: An open-source dataset of 450 real-robot scenarios with metric-scale ground truth, point clouds, and both explicit/implicit semantic targets.
Evaluation Framework: A modular "World-Decoder" pipeline that converts video generation into quantitative path planning metrics, focusing on tendency rather than strict trajectory overlap.
Fine-Tuning Demonstration: The first demonstration that fine-tuning an open-source video model on a small real-world robot dataset significantly boosts planning performance.

4. Experimental Results

The authors evaluated state-of-the-art models (Sora 2, Veo 3.1, Wan series) and fine-tuned variants.

Off-the-Shelf Performance:
- The best performing proprietary model, Wan2.2-Flash, achieved a Weighted Overall (WO) score of only 0.341.
- Most models struggled with metric accuracy (high FDE/ADE) and reliability (high Miss Rates), despite generating visually plausible videos.
- Implicit vs. Explicit: Performance was similar for both target types, suggesting models can understand implicit semantic reasoning, but the geometric execution remains poor.
Fine-Tuning Impact:
- Fine-tuning the open-source Wan2.2-5B model on the 325 training samples dramatically improved performance.
- The fine-tuned model with data augmentation (Wan2.2-5B-FT-DA) achieved a WO score of 0.394, surpassing the best off-the-shelf model by ~15% and showing a 469% improvement over the base open-source model (0.084).
Reconstruction Tools:
- VGGT performed best as a world-decoder for ground truth videos (WO 0.862), validating the evaluation pipeline.
- ViPE (SLAM-based) and SpaTracker showed lower performance, highlighting the difficulty of extracting accurate metric paths from generated videos.
Planning Horizon:
- Performance improved as the prediction horizon decreased (8s $\to$ 4s), indicating that WMs are more reliable for short-term planning.

5. Significance and Future Work

Bridging the Gap: The study reveals a significant disconnect between "visual realism" and "planning capability" in current video world models. High-quality video generation does not guarantee the ability to reason about spatial navigation.
Data Efficiency: The results prove that video WMs can learn complex navigation tasks from limited real-world data (325 samples), suggesting a viable path for adapting foundation models to robotics without massive datasets.
Future Directions:
- Moving toward closed-loop re-planning where the robot continuously updates its plan based on new observations.
- Investigating how latent memory in WMs supports long-horizon navigation.
- Improving 3D reconstruction tools to further reduce the gap between generated video and geometric reality.

In conclusion, Target-Bench establishes a rigorous standard for evaluating the "embodied intelligence" of video world models, demonstrating that while current models are not yet ready for autonomous mapless navigation, they hold significant promise when adapted via targeted fine-tuning.