⚡ electrical engineering

Mind the Privileged-to-Camera Gap: Actor-Centric Sidecar Supervision for Camera-First Open-Loop Waypoint Prediction

This paper proposes an actor-centric sidecar supervision method that leverages privileged simulator-derived labels during training to significantly improve camera-first open-loop waypoint prediction accuracy by explicitly grounding actor representations, achieving a 32.6% reduction in final displacement error compared to baseline models.

Original authors: Feeza Khan Khanzada, Jaerock Kwon

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Feeza Khan Khanzada, Jaerock Kwon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car using only a video camera. The robot's goal is to look at the road ahead and predict exactly where the car should be in the next few seconds (these predictions are called "waypoints").

The Problem: The "Blind Spot" in Learning
Most current robot drivers are like students who are only graded on their final destination. If the car ends up in the right spot, the student gets an A. But this doesn't tell the student how they got there. Did they ignore a pedestrian? Did they miss a car turning left? Because the robot isn't explicitly taught to notice other road users (actors) during training, it might learn to drive well by accident, but it lacks a deep understanding of who is around it and why they matter.

The Solution: The "Sidecar" Tutor
The authors of this paper introduced a clever training trick they call "Sidecar Supervision."

Think of the robot driver as a student taking a test.

The Real Test (Inference): When the robot is actually driving, it only has its camera. It sees the road, knows its own speed, and has a destination. It has no idea what other cars or pedestrians are doing unless it figures it out from the picture.
The Training Session (The Sidecar): During practice, the robot is given a "Sidecar" tutor. This tutor has a magical, perfect view of the entire simulation. The tutor knows exactly where every car, pedestrian, and cyclist is, how fast they are moving, and which ones are most important for the robot's path.

The tutor doesn't drive the car. Instead, it whispers to the robot: "Hey, look at that red car on the left; it's moving fast and might cut you off. Also, that pedestrian is crossing; pay attention to them."

The robot uses this extra information only while practicing. It learns to build a mental map of the road that includes these other actors. Once the training is over, the "Sidecar" tutor is removed. The robot goes back to driving with just its camera, but now it has learned to "see" the importance of other road users on its own.

The Results: A Big Leap Forward
The researchers tested this method against standard robots that only learned from the final destination.

The Old Way: The standard robot made a final error of about 1.8 meters (roughly the length of a small car) when predicting where it would end up.
The New Way: The robot trained with the "Sidecar" tutor reduced that error to 1.2 meters.

This is a 32% improvement. The paper notes that this new method worked better on almost every single test route (1,445 out of 1,494). It was especially good in tricky situations with many cars or when vulnerable road users (like cyclists or pedestrians) were present.

The "Privileged Gap"
The paper also ran a "cheat code" test. They asked: "What if the robot could see the perfect simulation data during the actual drive?"
The answer was: It would be even better (error dropping to 0.17 meters). This proves that while the "Sidecar" training helps the camera-based robot get much smarter, there is still a gap between what a camera can see and what a perfect, all-knowing system knows. The camera-based robot is still guessing a little bit, but the "Sidecar" training helps it guess much more accurately.

In Summary
The paper shows that you can make a camera-only self-driving car much smarter by giving it a "cheat sheet" (the Sidecar) during training that teaches it to notice and understand other road users. Even though the robot doesn't use that cheat sheet when it's actually driving, the lessons stick, leading to safer and more accurate predictions of where the car should go.

Note: The authors emphasize that these results are from a computer simulation (open-loop testing). They have not yet proven this works in real-world traffic or guarantees safety in a live crash scenario.

Technical Summary: Mind the Privileged-to-Camera Gap

Problem Statement

Camera-first autonomous driving models predict future ego-vehicle waypoints using only RGB images, ego-state features, and route commands. While end-to-end systems mapping these inputs directly to controls are scalable, the authors argue that standard waypoint supervision alone often leaves the representation of nearby road users (actors) weakly supervised. In complex urban scenes, the ego trajectory frequently depends on a small set of specific vehicles, pedestrians, or cyclists. A pure waypoint objective constrains the final path but may fail to force intermediate image features to explicitly encode which actors exist, which are relevant, or how they move.

The core problem addressed is whether training-only, actor-level supervision can improve RGB-only open-loop waypoint prediction without requiring privileged actor-state information (e.g., exact simulator coordinates) at inference time. The authors aim to quantify the "privileged-to-camera gap"—the performance difference between models using exact road-user states and those relying solely on camera inputs.

Methodology

Framework Overview

The proposed approach treats camera-first waypoint prediction as a supervised, actor-centric representation learning problem. The system introduces Road-User Sidecar (RU-sidecar) supervision: auxiliary labels derived offline from simulator states that guide the training of a camera-based model. These labels are never used as inference inputs.

The framework consists of three main components:

Deployable Camera Path: A student model that takes multi-view RGB, ego-state, and route commands as input. It predicts actor-centric intermediate representations (actor tokens), ranks their relevance, predicts short-horizon futures for selected actors, and uses these to predict future ego waypoints.
Offline Label Generator: A module that processes simulator state to generate sidecar labels ( $S_t$ $S_{t}$ ) for training. These include:
- Actor Geometry & Attributes ( $B_t$ ): Class, position, heading, velocity, and validity masks.
- Image-Grounded Labels ( $G_t$ ): Projected actor locations in camera views.
- Target-Conditioned Relevance ( $C_t$ ): A relevance score ( $c_i$ ) ranking actors based on distance, relative velocity, time-to-conflict, route-lane relevance, vulnerable road user (VRU) status, and hindsight future ego-path overlap.
- Selected-Actor Futures ( $F_t$ ): Short-horizon motion targets for the top- $K$ relevant actors.
Non-Deployable Simulator Path: A diagnostic reference model using exact simulator actor states ( $A_t$ ) as input. This serves as an oracle to estimate the remaining performance gap and, optionally, as a teacher for distillation.

Model Architecture

The deployable model uses a shared visual encoder (ConvNeXt-Tiny) to process RGB history. It employs a student interface comprising:

Actor-Token Head: Predicts a fixed set of camera-derived actor slots and attributes.
Selector: Ranks actor slots by relevance to the ego vehicle.
Dynamics Head: Predicts short-horizon futures for the top- $K$ selected actors.
Waypoint Planner: Attends to the selected actor-future tokens to predict the ego trajectory.

During training, predicted slots are matched to sidecar actors via deterministic greedy nearest-center matching. Losses include:

Primary: Waypoint imitation loss ( $L_{wp}$ ).
Sidecar Losses: Objectness/geometry ( $L_{obj}$ ), attributes ( $L_{attr}$ ), relevance ranking ( $L_{sel}$ ), and selected-actor dynamics ( $L_{dyn}$ ).
Optional Distillation: Teacher alignment ( $L_{dist}$ ) using the simulator-state model.

Experimental Protocol

Experiments utilize a fixed simulator snapshot from LMDrive with route-disjoint splits (no route appears in multiple splits). The evaluation is strictly open-loop, measuring Final Displacement Error (FDE) and Average Displacement Error (ADE) on held-out routes. The study compares:

A plain waypoint-only RGB baseline.
A matched "no-teacher" non-sidecar control (same architecture, no sidecar losses).
RU-sidecar models (with and without teacher alignment).
Non-deployable diagnostics (Oracle sidecar-selected and Simulator-state dynamics).

Key Results

Quantitative Performance

On the test split, the RU-sidecar model without teacher alignment achieved a Final Displacement Error (FDE) of 1.223 ± 0.01 m.

This represents a 32.6% reduction compared to the plain waypoint-only RGB baseline (1.815 m).
It represents a 28.7% reduction compared to the matched no-teacher non-sidecar control (1.716 m).
The model improved over the baseline on 1445/1494 held-out routes and over the matched control on 1417/1494 routes.

Teacher Alignment: Adding a simulator-state teacher reduced FDE further to 1.186 m but introduced higher seed variability. The authors treat this as secondary to the robust no-teacher result.

Diagnostics:

The Oracle sidecar-selected actor diagnostic (using exact actor selection but camera features) achieved 1.146 m FDE.
The Simulator-state actor-dynamics diagnostic (using exact actor states) achieved 0.174 m FDE.
The large gap between the RU-sidecar model (1.223 m) and the simulator-state diagnostic (0.174 m) confirms a significant "privileged-to-camera gap" remains.

Slice Analysis

Improvements were consistent across all non-empty actor-conditioned slices:

Actor-Dense Scenes: 29.1% FDE reduction for samples with $\ge$ 4 valid sidecar actors.
Vulnerable Road Users (VRU): 30.0% FDE reduction when a pedestrian or cyclist is present.
Horizon-wise: Error reduction was observed across all 6 prediction horizons.

Ablation Findings

Removing actor-future supervision ( $L_{dyn}$ ) or selector supervision ( $L_{sel}$ ) caused performance to regress close to the baseline, indicating both components are critical for the gains.
Simple waypoint loss reweighting or shuffled-teacher controls yielded negligible improvements, suggesting the gains are specific to the actor-centric supervision structure.

Significance and Claims

The paper claims to demonstrate that simulator-derived, training-only sidecar supervision significantly improves open-loop waypoint prediction for camera-first models. By formulating the problem as actor-centric representation learning, the model learns to encode road-user structure (existence, relevance, and motion) from RGB inputs alone, which in turn aids trajectory planning.

Key contributions include:

Formulation: Treating camera-first waypoint prediction as a supervised actor-centric representation problem.
Method: Introducing RU-sidecar labels for actor grounding, relevance, and motion without compromising RGB-only inference.
Evaluation: A controlled "control ladder" that isolates the effects of actor supervision from architectural changes or loss reweighting, using route-disjoint splits.
Gap Quantification: Providing empirical evidence of the remaining gap between camera-derived representations and privileged actor states.

Limitations and Modesty:
The authors explicitly state that their claims are limited to open-loop simulation diagnostics. The study does not establish closed-loop safety, crash avoidance, real-world transfer, or deployment readiness. Furthermore, the actor-future head is characterized as a "representation-shaping and planner-conditioning mechanism" rather than a deployable forecasting module, as its absolute future-state prediction (Dyn-FDE) remains poor. The primary supported claim is the efficacy of the no-teacher RU-sidecar recipe, as teacher alignment offered only marginal gains with higher instability.