Technical Summary: Zero-Label Driving Scenario Complexity Detection via Joint Embedding Predictive Architecture

Problem Statement

Real-world autonomous vehicle (AV) datasets are heavily skewed toward uneventful, routine driving scenarios, making the identification of rare, complex, and safety-critical events labor-intensive and expensive. Existing solutions rely on human annotation, supervised classifiers, or hand-crafted rule sets, all of which require substantial prior knowledge or costly data collection. The authors propose a solution to discover scenario complexity entirely without labels, asking whether a model can autonomously identify demanding scenarios based solely on its ability to predict future agent dynamics.

Methodology

Data and Representation

The study utilizes the nuPlan mini dataset, extracting 1,322 scenarios (5 seconds each at 10 Hz). Unlike approaches using raw pixels, this method operates on structured agent state data. Each scenario is represented as a tensor of shape $[50, 21, 6]$, encoding 50 timesteps for up to 21 agents (ego + 20 surrounding). The six features per agent include position ($x, y$), velocity ($v_x, v_y$), heading ($\theta$), and agent type. Data is normalized to ensure typical values fall near $[-1, 1]$.

Architecture: Joint Embedding Predictive Architecture (JEPA)

The core of the approach is a minimal Joint Embedding Predictive Architecture (JEPA) trained on structured agent states. The architecture consists of four components:

1. Context Encoder: A 4-layer Transformer that processes the first 25 timesteps (context window) to produce a latent representation $z_{ctx}$.
2. Predictor: A 2-layer Transformer that maps $z_{ctx}$ to a predicted future latent $\hat{z}_{tgt}$, conditioned on a learnable horizon embedding.
3. Target Encoder: An identical copy of the context encoder that processes the future 25 timesteps (target window) to produce the ground-truth latent $z_{tgt}$. Crucially, this encoder is never updated via backpropagation; its weights are updated solely via an Exponential Moving Average (EMA) of the context encoder's weights.
4. Position Decoder: A small MLP that decodes the predicted latent into future positions, serving as an auxiliary supervision signal.

Training Objective

The model is trained with a self-supervised loss function combining two terms:
$$\mathcal{L} = \underbrace{\|\hat{z}_{tgt} - z_{tgt}\|_2^2}_{\text{JEPA Loss}} + \lambda \underbrace{\|\hat{p} - p_{tgt}\|_2^2}_{\text{Position Loss}}$$
where $\lambda = 0.1$. The gradients flow through the context encoder and predictor but are stopped at the target encoder. The EMA update ($\alpha = 0.996$) prevents representation collapse by ensuring the target encoder remains a smoothed historical average of the context encoder.

Complexity Metric: Surprise Score

At inference, no labels are used. The surprise score ($s$) for a scenario is defined as the L2 distance between the predicted future latent and the actual future latent:
$$s = \|\hat{z}_{tgt} - z_{tgt}\|_2$$
A high score indicates the model's prediction of the future state was inaccurate, implying the scenario contained complex, unexpected interactions.

Key Contributions

1. Zero-Label Complexity Scoring: The authors demonstrate that a JEPA model trained on structured driving data without any labels produces surprise scores that correlate strongly with ground-truth scenario tags (e.g., "unprotected turn," "pedestrian proximity").
2. Ablation Analysis: Four experiments isolate the source of the signal:
   • Shuffled Scores: Confirm the correlation is not an artifact of the analysis pipeline.
   • Random Encoder: Proves that trained weights are necessary (random initialization yields no correlation).
   • Constant-Velocity Baseline: Shows that simple kinematic prediction fails to rank complex scenarios correctly, often assigning low surprise to slow-approach events preceding complex maneuvers.
   • No-EMA Training: Demonstrates that without EMA, the model collapses (score spread drops 44-fold), proving EMA is the specific mechanism enabling the learning of temporal structure.
3. Downstream Anomaly Detection: The surprise score serves as an effective zero-shot anomaly detector, achieving an Average Precision (AP) of 0.512 against a chance baseline of 0.436.

Results

• Tag Correlation: The model assigns significantly higher surprise scores to complex tags such as "starting unprotected cross turn" (1.978) and "near pedestrian on crosswalk" (1.834), while assigning lower scores to routine tags like "following lane with lead" (1.406).
• Ablation Findings: The constant-velocity baseline showed a moderate correlation ($\rho=0.314$) but failed to distinguish complex scenarios that begin with controlled deceleration. The no-EMA model failed completely, with all scenarios receiving nearly identical scores.
• Anomaly Detection: The trained JEPA model was the only method to remain consistently above the chance baseline across the full ranking curve. The constant-velocity baseline fell below chance after the top 5% of ranked scenarios.
• Qualitative Validation: Visual inspection of the top 9 most surprising scenarios revealed dense agent clustering, intersections, and pedestrian crossings, while the bottom 9 showed sparse, parallel lane-following.

Significance and Claims

The paper claims that temporal prediction error in a self-supervised latent world model is a practical proxy for driving scenario complexity.

• Mechanism: The model learns to predict the "compressed essence" of future agent interactions. It fires on scenarios with unusual multi-agent dynamics rather than visual noise or static properties.
• Advantage over Kinematics: Unlike constant-velocity baselines, the JEPA model recognizes that controlled deceleration toward an intersection is a precursor to complex events, assigning higher surprise scores even before the critical maneuver occurs.
• Practicality: The approach requires no annotation cost. A single model trained end-to-end with a standard self-supervised objective can automatically curate datasets by surfacing complex scenarios, addressing the data imbalance problem in autonomous driving development.

The authors maintain a modest scope, noting limitations such as the use of the mini subset of nuPlan (1,322 scenarios) and the current coupling to the nuPlan data pipeline, suggesting that calibration may improve with larger-scale data.