Technical Summary: Thinking Ahead: Foresight Intelligence in MLLMs and World Model

1. Problem Definition

Current Vision-Language Models (VLMs) and autonomous driving benchmarks predominantly focus on perception (understanding the present state) and reactive planning (immediate control decisions). There is a significant gap in evaluating Foresight Intelligence: the ability to anticipate and reason about complex, long-horizon future events, particularly under uncertainty and counterfactual conditions.

Existing benchmarks (e.g., DriveLM, OmniDrive) often restrict queries to immediate actions or geometric path generation (L2 displacement errors). They fail to address:

• Long-horizon reasoning: Anticipating semantic consequences over extended timeframes (e.g., 12 seconds).
• Counterfactual simulation: Reasoning about "what-if" scenarios (e.g., "If the ego vehicle turns left, what happens?").
• Semantic consistency of World Models (WMs): While WMs can generate visually realistic future videos, it remains unclear if these generations are semantically coherent and useful for downstream decision-making.

2. Methodology

2.1 FSU-QA Dataset

The authors introduce FSU-QA, a Visual Question Answering (VQA) dataset derived from the nuScenes dataset, specifically designed to train and evaluate Foresight Intelligence.

• Data Construction: Utilizes front-view videos (~15s) split into a 3-second historical observation window and a 12-second future window.
• Task Hierarchy: The dataset organizes 21,000+ QA pairs into three cognitive levels:
  1. Low-level (Spatio-temporal Dynamic Reasoning): Predicting ego-motion states (speed, turn, lane changes) and relative agent positions/distance over 12 seconds.
  2. Mid-level (VRU-centric Risk Assessment): Inferring pedestrian intent, ego-VRU relative position changes, and identifying risk areas within 3 seconds.
  3. High-level (Causal Reasoning): Counterfactual Prediction (CFP). Models must predict outcomes of hypothetical actions (e.g., "If the ego vehicle turns left with a constant radius...") based on geometric constraints and HD maps.
• Annotation Protocol: Ground truth for counterfactuals is generated via rule-based geometric intersection checks using 3D trajectories and HD maps, validated by human experts (89% agreement rate).

2.2 FSU-Bench Evaluation Protocol

The benchmark evaluates two distinct capabilities:

1. VLM-oriented Evaluation: Assesses a VLM's ability to predict future semantic outcomes given historical video ($V_{-T_h:0}$) and trajectories ($Traj_{-T_h:0}$).
   $$\hat{a} = \text{VLM}(q, V_{-T_h:0}, Traj_{-T_h:0})$$
2. World Model (WM) Evaluation: A proxy-evaluation paradigm to test if WM-generated futures ($\hat{V}_{1:T_f}, \hat{Traj}_{1:T_f}$) provide semantically meaningful cues. The VLM is augmented with predicted future data:
   $$\hat{a} = \text{VLM}(q, V_{-T_h:0}, Traj_{-T_h:0}, \hat{V}_{1:T_f}, \hat{Traj}_{1:T_f})$$
   • Control Experiment: To ensure gains are not due to mere token augmentation, the authors perform shuffled control experiments where WM predictions from random, unrelated scenes are used. A drop in accuracy confirms that performance gains rely on semantic coherence.

2.3 Models Evaluated

• VLMs: 13 models including closed-source (GPT-5, Claude-Sonnet-4.5, Gemini-2.5 Pro) and open-source (Qwen2.5/3-VL, Llama 4 series).
• World Models: Epona (autoregressive + DiT) and DrivingWorld (purely autoregressive).
• Fine-tuning: Qwen3-VL-8B was fine-tuned on FSU-QA (Qwen3-VL-8B-FI) to test the dataset's training efficacy.

3. Key Results

3.1 VLM Performance (Baseline)

• General Struggle: State-of-the-art VLMs face significant challenges in foresight intelligence. Even the best model (GPT-5) achieved only ~48.66% overall accuracy.
• Task Disparity: Models perform well on low-level geometric tasks (e.g., Lane Change: ~90-97%) but struggle with mid-level risk assessment and high-level counterfactual reasoning (often <20% on CFP tasks).
• Scale vs. Capability: Performance is not strictly correlated with parameter size. Open-source models (e.g., Qwen2.5-VL-72B) outperformed some smaller closed-source models, but all models exhibited systematic weaknesses in multi-agent interaction and safety-critical counterfactuals.
• Fine-tuning Impact: Fine-tuning Qwen3-VL-8B on FSU-QA resulted in substantial improvements (Overall: 59.59%), outperforming all zero-shot baselines, including larger models like GPT-5.

3.2 World Model Evaluation

• Semantic Utility: Augmenting VLMs with WM-generated predictions generally improved performance, but the gains were architecture-dependent.
• DrivingWorld vs. Epona: DrivingWorld provided larger and more consistent gains, particularly in relational reasoning (Relative Position, Pedestrian Intent). Epona's gains were more modest and task-selective.
• Validation: The shuffled control experiments confirmed that performance improvements were due to the semantic coherence of the predicted futures, not just the addition of visual tokens.

4. Key Contributions

1. FSU-QA Dataset: A novel VQA dataset targeting Foresight Intelligence with 9 task types spanning low-level perception to high-level counterfactual reasoning, addressing the lack of long-horizon, multi-agent, and "what-if" benchmarks in autonomous driving.
2. FSU-Bench & Proxy Protocol: A unified benchmark that evaluates both VLM foresight capabilities and the semantic utility of World Models via a proxy-evaluation paradigm, validated by shuffled control experiments.
3. Empirical Findings:
   • Current VLMs systematically underperform on high-level counterfactual reasoning, a gap not solely explained by model scale.
   • World Model-generated data can enhance foresight reasoning, but the quality of semantic coherence varies significantly between architectures.
   • Targeted fine-tuning on FSU-QA effectively bridges the foresight gap within the nuScenes domain.

5. Significance and Claims

The paper claims that Foresight Intelligence is a distinct capability from reactive planning, requiring simulation-level cognition rather than just pattern recognition. The authors posit that:

• Existing benchmarks are insufficient for evaluating the safety-critical, long-horizon reasoning required for autonomous driving.
• FSU-QA provides a principled foundation for training models to anticipate complex futures.
• The evaluation of World Models via downstream QA performance offers a more diagnostic metric for real-world utility than standard pixel-level generation metrics (e.g., FVD, FID).

The authors conclude that while current models struggle with high-level foresight, the combination of targeted datasets (FSU-QA) and improved World Models offers a viable path forward. They note that cross-domain generalization (e.g., to Waymo or nuPlan) remains an open challenge for future work.