Technical Summary: BeyondSight: Object Permanence for End-to-End Autonomous Driving

1. Problem Statement

Autonomous driving systems operate in partially observable environments where actors (vehicles, pedestrians) frequently become fully occluded by infrastructure or other vehicles. Analysis of the nuScenes dataset reveals that approximately 30% of actors are fully unobservable at any given timestep, including those in close proximity to the ego vehicle.

Current end-to-end autonomous driving (E2E-AD) systems typically couple actor existence to instantaneous observations. When an actor becomes fully unobservable, its representation in the model often degrades or disappears entirely. This removal of "invisible" agents from the scene representation forces downstream prediction and planning modules to reason only over currently visible agents. Consequently, these systems exhibit reactive or brittle behavior in occlusion-heavy scenarios (e.g., intersections), failing to maintain safe trajectories for agents that are temporarily hidden but still present. Furthermore, standard perception benchmarks implicitly equate observability with existence, filtering out unobservable actors from both supervision and evaluation, thereby preventing models from learning persistent representations across observability gaps.

2. Methodology: BeyondSight

The authors propose BeyondSight, a permanence-aware E2E-AD framework that explicitly decouples actor existence from instantaneous observability. The system maintains persistent actor hypotheses over time, enabling joint perception, prediction, and planning to reason about actors even when they are temporarily unobservable.

Core Architecture

BeyondSight builds upon the SparseDrive architecture, which uses sparse queries to represent dynamic actors and map elements. The key innovation is a three-stage inference pipeline inspired by Bayesian filtering, designed to propagate and update actor hypotheses:

1. Temporal Prior Decoder: This module propagates actor queries from the previous frame ($q_{t-1}$) to the current timestep ($t$) to generate a motion-conditioned hypothesis set ($q_t^{\text{prior}}$). Crucially, this stage operates solely on query embeddings using self-attention and a lightweight MLP, without accessing current image features. This allows the model to maintain hypotheses for actors even when no visual evidence is available.
2. Observation Decoder: This stage combines the temporal prior with newly initialized object queries and current BEV features ($F_t$) to produce observation-conditioned hypotheses ($q_t^{\text{obs}}$). This component is supervised only using observable ground-truth actors.
3. Posterior Fusion Decoder: This module reconciles the propagated prior and the observation-conditioned hypotheses. It selects the proposal with higher classification confidence for each actor and refines it to produce the final scene representation ($q_t^{\text{post}}$). This fused representation contains both observable and unobservable actors, providing richer context for downstream planning.

Optimization and Supervision

The training follows a two-stage protocol (perception pre-training followed by end-to-end joint training). To enforce object permanence, the authors introduce Permanence-Aware Supervision:

• Extended Ground Truth: The detection and motion losses are computed over the full set of actors ($G_{\text{all}} = G_{\text{obs}} \cup G_{\text{unobs}}$), including those currently unobservable.
• Observability Classification: The posterior decoder predicts an observability state ($\hat{o}_t^i$) for each query, supervised via binary cross-entropy. This explicitly teaches the model to distinguish between an actor that is occluded versus one that is absent.
• Loss Function: The total objective includes standard detection, map, motion, and planning losses, augmented with a modified detection loss ($\mathcal{L}_{\text{det}}'$) that supervises unobservable actors and an observability classification loss ($\mathcal{L}_{\text{obs}}$).

3. Key Contributions

The paper introduces three primary contributions:

1. Formalization of Object Permanence: The authors define object permanence for E2E-AD as a representation-level constraint requiring that previously observed actors remain represented in the model's hypothesis set even when temporarily unobservable.
2. nuScenes-Permanence Benchmark: An extension of the nuScenes dataset that provides supervision and evaluation protocols for unobservable actors. This involves reconstructing missing trajectory states using physically grounded motion priors for intervals where sensor support is absent, expanding the annotation set by approximately 30%.
3. BeyondSight Framework: A permanence-aware extension to sparse-query E2E-AD architectures that maintains persistent hypotheses through temporal propagation and observability-aware fusion, enabling reasoning under prolonged occlusion.

4. Experimental Results

Experiments were conducted on the nuScenes validation set and the proposed nuScenes-Permanence benchmark.

Standard Performance

On standard nuScenes metrics, BeyondSight maintains competitive performance while improving downstream planning:

• Planning: The open-loop planning error ($L2_{\text{avg}}$) improved from 0.61 (SparseDrive) to 0.54, and the collision rate decreased from 0.08% to 0.07%.
• Perception: Detection performance (mAP) increased from 0.415 to 0.427, and tracking performance (AMOTA) improved from 0.372 to 0.401.
• False Positives: The model did not incur additional false positives under the official evaluation protocol, demonstrating that maintaining persistent hypotheses does not degrade standard benchmark performance.

Object Permanence Performance (nuScenes-Permanence)

The proposed observability-conditioned metrics reveal significant gains in reasoning about occluded actors:

• Unobservable Detection: The model achieved an mAP$_{\text{unobs}}$ of 0.249, a dramatic improvement over the baseline SparseDrive, which scored 0 (as it drops unobservable actors).
• Holistic Understanding: The aggregated metric mAP$_{\text{all}}$ improved from 0.389 to 0.413.
• Planning under Occlusion: In a high-occlusion subset where hidden actors' future trajectories approached the ego vehicle, BeyondSight reduced planning error from 0.64 to 0.56 and collision rates from 0.163 to 0.08.

Ablation Studies

Ablation experiments confirmed that:

• Temporal propagation alone (without permanence-aware supervision) does not enable detection of fully unobservable actors (mAP$_{\text{unobs}}$ remains 0).
• The combination of temporal propagation, posterior fusion, and supervision on unobservable actors is required to achieve the reported gains.
• The Posterior Fusion Decoder is critical for reconciling propagated hypotheses with current observations.

5. Significance and Limitations

The paper argues that object permanence is a critical modeling principle for robust end-to-end autonomous driving. By decoupling actor existence from instantaneous observability, BeyondSight enables systems to reason about the full dynamic environment, leading to safer planning in occlusion-heavy scenarios. The introduction of nuScenes-Permanence provides the first systematic protocol for training and evaluating this capability.

Limitations:

• Stale Hypotheses: Persistent hypotheses may become inaccurate ("stale") during very long occlusions if actors deviate significantly from the learned temporal prior, potentially leading to localization drift.
• Motion Assumptions: The nuScenes-Permanence annotations assume approximately continuous motion. The benchmark may not capture abrupt hidden maneuvers (e.g., sudden stops, sharp turns, or intent changes) that cannot be reliably inferred from surrounding observations.
• Future Directions: The authors suggest future work should explore uncertainty-aware memory, simulation-based annotations with privileged labels, and multi-observer datasets for direct occlusion evaluation.