The Impact of Class Uncertainty Propagation in Perception-Based Motion Planning

1. Problem Statement

Autonomous vehicles (AVs) operate in highly dynamic urban environments where uncertainty is inherent in sensor data, occlusions, and the multimodal nature of other agents' future behaviors. While modern AV stacks include perception, prediction, planning, and control modules, most systems fail to effectively propagate uncertainty from the perception and prediction stages into the motion planning stage.

Existing uncertainty-aware planners often rely on prediction models that are miscalibrated (i.e., their predicted uncertainty does not match empirical reality). The paper addresses a critical gap: How does the calibration of upstream prediction uncertainty (specifically class uncertainty) impact downstream closed-loop motion planning performance? Previous studies have largely relied on open-loop evaluations, which reset the ego state at every timestep, thereby masking the compounding effects of planning decisions based on poor uncertainty estimates.

2. Methodology

A. Novel Planner: Uncertainty-Aware Stochastic MPC (UA-SMPC)

The authors propose a novel Uncertainty-Aware Stochastic Model Predictive Control (UA-SMPC) framework.

• Formulation: It extends a convex SMPC formulation to handle multi-modal predictions from multiple agents.
• Collision Avoidance: Unlike previous works that use conservative over-approximations, this method employs an exact probabilistic collision avoidance constraint. It calculates the minimal "keep-out" region required to guarantee a collision probability $P(\text{no collision}) \geq p$ for a chosen coverage level.
  • It models agents as Gaussian Mixture Models (GMMs).
  • It transforms the vehicle geometry (rectangle) and the agent's uncertainty (ellipsoid) into Mahalanobis coordinates to compute the exact distance between the ego vehicle and the uncertainty-adjusted keep-out region.
• Policy Structure: The planner uses a parameterized policy that incorporates state feedback. It shares the first control input across all prediction modes (to ensure immediate reaction to observed states) but allows mode-specific controls for future timesteps to adapt to different predicted trajectories.

B. Prediction Models for Comparison

To isolate the effect of uncertainty propagation, the authors compare two prediction architectures paired with the same UA-SMPC planner:

1. Trajectron++ (T++): A state-of-the-art (SOTA) trajectory forecasting model that is uncertainty-agnostic. It predicts trajectories based on state vectors but does not propagate class uncertainty (the probability of an agent's class or intent).
2. HAICU: A follow-up to Trajectron++ that is uncertainty-aware. It incorporates both state vectors and class probabilities into its node and edge histories, explicitly propagating class uncertainty to the output distribution.

C. Experimental Setup

• Benchmark: The nuPlan large-scale planning benchmark, evaluated in a closed-loop setting with reactive agents.
• Datasets: Models were trained on the PUP (Perceptual Uncertainty in Prediction) dataset, which provides unfiltered class probabilities. They were then evaluated on nuPlan (where class uncertainty is not natively available, forcing HAICU to rely on its learned propagation capabilities or default assumptions).
• Baselines: A constant velocity model with fixed covariance was used as a benchmark.
• Metrics:
  • Prediction: minADE, minFDE, Negative Log-Likelihood (NLL), Entropy, Expected Calibration Error (ECE).
  • Planning: Ego Progress, Jerk (Comfort), Time to Collision (TTC), and the nuPlan Closed-Loop Score.

3. Key Contributions

1. Novel UA-SMPC Formulation: An extension of SMPC that directly incorporates multi-modal GMM predictions with an exact, non-approximated probabilistic collision avoidance constraint, avoiding the conservatism of previous methods.
2. Closed-Loop Analysis of Uncertainty Propagation: A comprehensive study comparing uncertainty-aware (HAICU) vs. uncertainty-agnostic (Trajectron++) prediction pipelines in a closed-loop environment, demonstrating that upstream calibration significantly impacts downstream safety and efficiency.
3. Sensitivity Analysis: A systematic analysis of how scaling predicted covariances affects planning performance, revealing that well-calibrated models respond predictably to uncertainty scaling, while miscalibrated models exhibit erratic behavior.

4. Key Results

Prediction Performance

• Calibration vs. Point Accuracy: On the nuPlan dataset, HAICU showed slightly worse point accuracy (minADE/minFDE) compared to Trajectron++ in some short-horizon scenarios. However, HAICU achieved significantly better distributional quality (lower NLL and lower entropy) and better calibration.
• Long-Horizon Degradation: Both learned models degraded in point accuracy over long horizons (8s), with the constant velocity baseline becoming competitive. However, HAICU maintained superior NLL, indicating its uncertainty estimates remained more reliable.

Closed-Loop Planning Performance

• Superiority of Uncertainty-Aware Planning: Despite having slightly worse point accuracy on nuPlan, HAICU-backed UA-SMPC achieved the highest Closed-Loop Score, particularly in long-horizon (8s) scenarios.
• Safety and Smoothness: HAICU produced smoother motion plans (lower Jerk) and safer trajectories (higher Time to Collision) compared to Trajectron++.
• Qualitative Differences: In visual rollouts, Trajectron++ failed to anticipate complex maneuvers (e.g., a trailing agent turning right), leading to aggressive, jerky swerves to avoid collisions it didn't predict. HAICU correctly identified these risks early, allowing for smoother, proactive adjustments.
• Correlation: The study found that distributional quality (NLL/Entropy) is a stronger predictor of planning performance than point accuracy (minADE/minFDE) for GMM-based planners.

Sensitivity Analysis

• HAICU: Showed a predictable trade-off. Reducing the covariance scaling factor (making predictions less conservative) improved progress but reduced safety (TTC). The best overall score was achieved at $\alpha = 1/3$, suggesting HAICU's default predictions were slightly over-conservative but well-calibrated.
• Trajectron++: Exhibited erratic behavior across scaling factors. Because its baseline uncertainty was miscalibrated, scaling the covariance did not yield consistent improvements, highlighting the fragility of planning on poorly calibrated inputs.

5. Significance and Conclusion

The paper establishes that uncertainty calibration is critical for robust autonomous driving.

• Calibration over Accuracy: For motion planning, a model that accurately represents the distribution of possible futures (even if the mean prediction is slightly off) is more valuable than a model with high point accuracy but poor uncertainty estimates.
• Compounding Effects: Miscalibrated predictions in open-loop settings may seem acceptable, but in closed-loop planning, they lead to compounding errors, resulting in unsafe or inefficient behaviors (e.g., unnecessary braking or jerky maneuvers).
• Design Implication: Future AV stacks should prioritize uncertainty-aware prediction models that propagate class and state uncertainties, as this directly enables planners to make safer, more efficient decisions in complex, dynamic environments.

The authors conclude that while point accuracy metrics (like ADE) are standard for evaluation, distributional metrics (NLL, Entropy, ECE) are essential for assessing the suitability of a predictor for uncertainty-aware motion planning.