Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings

Technical Summary: Unveiling Transferability in Trajectory Prediction via Latent Scene Embeddings

Problem Definition
The paper addresses the challenge of generalizing trajectory prediction models across different datasets. While deep learning has advanced motion prediction, models trained on one dataset often fail to generalize to others due to significant disparities in scene layouts, agent behaviors, sensing modalities (e.g., camera vs. drone), and map structures. Although large foundation models suggest that pooling diverse data could yield transferable representations, practical constraints (latency, compute) and the risk of performance degradation from low-quality or irrelevant data necessitate a more selective approach. The core problem is determining which datasets are suitable for pretraining or adaptation for a specific target domain without exhaustive trial-and-error experimentation.

Methodology
The authors propose a framework that learns a shared latent space to characterize trajectory datasets and quantify their transferability.

1. Latent Embedding Model:

   • Architecture: A deep encoder-decoder model is trained jointly on all 24 datasets. The encoder maps input features (agent states and optional map data) to node-level latent variables using Graph-Gated Recurrent Units (Graph-GRUs).
   • Training Objectives: The model is optimized using two complementary tasks: reconstructing observed input features and forecasting future states. This ensures the latent space captures both structural data properties and relevant forecasting dynamics.
   • Regularization: Latent variables are constrained to the surface of a unit sphere to encourage comparability and prevent degenerate solutions.
   • Aggregation: Node-level embeddings are aggregated into scene-level statistics (mean and covariance). These are further aggregated to define a dataset-level distribution, modeled as a multivariate Gaussian characterized by a mean vector and a covariance matrix (regularized via low-rank approximation and jitter to ensure numerical stability).

2. Transferability Metric:

   • Transferability between a source dataset $D_i$ and a target dataset $D_j$ is quantified using the Kullback–Leibler (KL) divergence between their respective Gaussian distributions in the latent space: $D_{KL}(D_i \| D_j)$.
   • This metric is directional (asymmetric), capturing the fact that a source dataset might approximate a target well, while the reverse is not true.

3. Evaluation Protocol:

   • The framework was evaluated across 24 diverse trajectory datasets (covering pedestrians, vehicles, and mixed traffic; collected via cameras, drones, and instrumented vehicles).
   • Zero-Shot Transfer: A baseline predictor (QCNet) was trained on one dataset and evaluated on others without adaptation.
   • Fine-Tuning: Models pretrained on source datasets were fine-tuned on target datasets to assess adaptation quality and catastrophic forgetting.
   • Correlation Analysis: The KL divergence scores were correlated with actual performance metrics (minADE6) to validate the predictive power of the latent distances.

Key Contributions

1. Dataset-Level Representation Learning: The introduction of a latent embedding framework that models trajectory datasets as distributions in a shared space, enabling comparison based on distributional geometry rather than surface-level statistics.
2. Directional Probabilistic Measure: A proposal of a KL-divergence-based metric to capture asymmetric transfer success, demonstrating that latent distances predict both zero-shot and fine-tuned performance.
3. Large-Scale Systematic Study: A comprehensive evaluation involving 24 datasets and 552 transfer pairs. This study identifies underutilized datasets that improve performance on standard benchmarks and reveals structural relationships (e.g., between drone-acquired and instrumented-vehicle datasets) that are not apparent from metadata alone.

Results

• Correlation with Performance: The study found a strong positive correlation ($\rho = 0.811$) between the KL divergence and zero-shot transfer performance (minADE6). Higher divergence consistently predicted poorer transfer.
• Fine-Tuning and Forgetting: The latent distance also predicted the degree of catastrophic forgetting during fine-tuning. Smaller divergences correlated with better adaptation and less loss of source knowledge.
• Gaussian Approximation: The KL divergence consistently outperformed alternative similarity measures (L1, Wasserstein, MMD) in predicting transferability, empirically justifying the Gaussian modeling assumption.
• Latent Dimensionality: An ablation study showed that a latent dimension of $L=32$ provided the optimal balance for zero-shot prediction, while larger dimensions ($L=128$) showed diminishing returns or overfitting trends for certain tasks.
• Dataset Relationships: The latent space revealed that datasets like WOMD, Argoverse, and nuScenes form distinct clusters, with specific asymmetric relationships (e.g., WOMD transfers well to nuScenes, but not vice versa). It also highlighted that drone-acquired datasets (e.g., inD, INTERACTION) share significant latent similarity with instrumented-vehicle datasets, suggesting cross-domain transfer potential.

Significance
The paper claims to provide a practical, data-driven mechanism for dataset selection in motion prediction. By quantifying transferability through latent distributional geometry, the framework allows researchers to:

• Predict cross-dataset generalization performance without training models on every possible source-target pair.
• Select optimal pretraining sources for new target domains.
• Identify underutilized datasets that can enhance performance on popular benchmarks.
• Move toward more robust and generalizable predictive systems by understanding the structural relationships between data sources rather than relying solely on domain-specific cues.

The work concludes that transferability in trajectory prediction can be predicted via learned latent representations, offering a foundation for more efficient development of foundation models and transfer learning strategies in autonomous driving and robotics.