Technical Summary: Instance-Level Post Hoc Uncertainty Quantification in Object Detection

1. Problem Formulation

Object detection is a safety-critical component of autonomous driving, where accurate quantitative uncertainty of bounding box predictions is essential for safety assurance. Current detection models typically lack the ability to quantify bounding box uncertainty during training. Once tuned for safety, these models cannot be altered, necessitating post hoc uncertainty quantification methods that do not require retraining or modifying the original network.

The paper addresses the challenge of estimating uncertainty for every individual bounding box (instance-level) while satisfying four key constraints:

• Non-intrusive/Post Hoc: The method must apply to pre-trained models without altering their behavior.
• Instance-Level: Uncertainty must be estimated at the level of individual bounding boxes, not just global model uncertainty.
• Inference Speed: The estimation must be computationally efficient enough for online autonomous driving systems.
• Epistemic Uncertainty: The method must capture uncertainty arising from the model's lack of knowledge (parameter uncertainty), rather than aleatoric noise.

Existing solutions face significant trade-offs:

• Sampling-based methods (e.g., MC Dropout, Deep Ensembles): While powerful, they are not strictly post hoc as they often require training multiple models or produce predictive distributions that deviate from the original Maximum A Posteriori (MAP) model.
• Linearized methods (e.g., Generalized Linear Model - GLM with Laplace Approximation): These are strictly post hoc and preserve the MAP prediction but are computationally prohibitive for instance-level uncertainty. Calculating the Jacobian $J(x)$ for every scalar output component requires a separate backpropagation pass per output. For a detector predicting $m$ boxes with 9 parameters each, this requires $9m$ backpropagations, which is too slow for real-time applications.

2. Methodology: Monte-Carlo Generalized Linearized Model (MC-GLM)

The authors propose MC-GLM, an approximate inference method that combines the post hoc guarantees of the Laplace approximation with the efficiency of Monte Carlo sampling.

Core Concept

The method approximates the predictive covariance $\Sigma_z(x)$ of the network output $z = f(x; \theta)$ using the Laplace approximation around the MAP estimate $\theta^*$:
$$\Sigma_z(x) \approx J(x) \Sigma_\theta J(x)^\top$$
where $J(x)$ is the Jacobian and $\Sigma_\theta$ is the posterior covariance of the parameters (computed offline via KFAC).

Instead of computing the full Jacobian $J(x)$ via $d$ backpropagations (where $d$ is the output dimension), MC-GLM estimates the covariance via Monte Carlo sampling. It recognizes that $J(x)\Sigma_\theta J(x)^\top$ is equivalent to the covariance of directional derivatives $J(x)\eta$, where $\eta \sim \mathcal{N}(0, \Sigma_\theta)$.

Algorithm Steps

1. Offline Phase: Compute the KFAC (Kronecker-factored Approximate Curvature) approximation of the Fisher information matrix to obtain the parameter covariance $\Sigma_\theta$. This step is performed once on the training data.
2. Online Phase (Per Input $x$):
   • Compute the base prediction $z_0 = f(x; \theta^*)$.
   • Sample $k$ weight perturbations $\eta_i \sim \mathcal{N}(0, \Sigma_\theta)$. The sampling exploits the block-diagonal and Kronecker structure of KFAC to be efficient.
   • Perform $k$ forward passes with perturbed weights: $z_i = f(x; \theta^* + \epsilon \eta_i)$.
   • Approximate the directional derivatives using finite differences: $u_i \approx (z_i - z_0) / \epsilon$.
   • Construct a matrix $A(x)$ of these $k$ directional derivatives.
   • Column-center $A(x)$ to obtain $\tilde{A}(x)$ and compute the sample covariance:
     $$\hat{\Sigma}_{z, MC-GLM}(x) = \frac{1}{k-1} \tilde{A}(x) \tilde{A}(x)^\top$$
3. Instance-Level Aggregation: For object detection, the output vector is reshaped to group parameters by bounding box. The method computes a per-box covariance tensor $\hat{\Sigma}_i(x) \in \mathbb{R}^{9 \times 9}$, capturing correlations within a box while ignoring cross-box correlations for efficiency.

Key Efficiency Feature

The computational cost of MC-GLM is $k+1$ forward passes, where $k$ is a small, constant hyperparameter (e.g., 10). Crucially, this cost is independent of the number of detected instances ($m$). In contrast, exact GLM requires $9m$ backpropagations. This allows MC-GLM to be parallelized and scaled to any number of detections without increasing inference time.

3. Key Contributions

• MC-GLM Framework: A novel method that provides instance-level, approximately post hoc uncertainty quantification for object detection.
• Computational Efficiency: By replacing the expensive Jacobian calculation with a fixed number of forward passes ($k$), the method achieves linear time complexity with respect to the number of samples rather than the number of output instances.
• Post Hoc Compatibility: The method guarantees that the predictive mean matches the original pre-trained model's output ($f(x; \theta^*)$), unlike sampling-based Bayesian methods that shift the mean.
• KFAC Integration: The method leverages KFAC for efficient offline computation of the parameter covariance and structured sampling, making it feasible for large-scale detectors.

4. Experimental Results

The method was evaluated on the nuScenes dataset using the CenterPoint 3D object detector.

• Metrics: Performance was measured using Probability of Accurate given Certain ($P(A|C)$), Probability of Uncertain given Inaccurate ($P(U|I)$), and Accuracy vs Uncertainty ($AvU$).
• Comparison Baselines:
  • MCI (Monte Carlo Integration): Uses MC Dropout.
  • GLM: Exact Laplace approximation with full Jacobian computation.
• Findings:
  • Uncertainty Quality: MC-GLM significantly outperforms the exact GLM in $P(U|I)$ (0.40 vs 0.14 for IoU), indicating better identification of inaccurate predictions. It achieves comparable $P(A|C)$ to GLM and MCI.
  • Speed: MC-GLM is substantially faster than GLM. On average, GLM required 10,636 ms per input (due to $9m$ backpropagations), while MC-GLM required only ~542 ms (11 forward passes). MCI was slightly faster (525 ms) but failed the post hoc requirement.
  • Ablation Studies:
    • Fisher Depth: Using Fisher information from deeper network layers improves $AvU$ but increases memory usage; the head-only approximation offers a good trade-off.
    • Data Subsampling: Estimating the Fisher matrix on 5-10% of the training data yields comparable uncertainty quality, reducing offline costs.
    • Hyperparameters: A small finite-difference step ($\epsilon = 10^{-7}$) and a modest number of samples ($k=10$) are sufficient for high-quality estimates.

5. Significance and Claims

The paper claims that MC-GLM fills a critical gap in safety-critical autonomous driving systems. It provides a practical, instance-level uncertainty quantification tool that adheres to strict post hoc constraints (no retraining, preserving original model behavior) while maintaining inference speeds suitable for real-time deployment.

The authors emphasize that unlike previous methods which are either too slow (exact GLM) or alter the model's predictive distribution (MC Dropout/Ensembles), MC-GLM offers a balanced solution. It enables the quantification of epistemic uncertainty for every detected object with a computational cost that is constant and independent of the number of detections, making it viable for online safety assurance in autonomous vehicles.