Technical Summary: Mode-as-Sequence

1. Problem Statement

Multimodal motion forecasting for autonomous driving faces a fundamental challenge: inherent under-supervision. In real-world training data, each scene provides only one realized future trajectory, yet multiple plausible futures (modes) exist (e.g., a vehicle may turn left, go straight, or yield). This scarcity of ground-truth labels for all valid modes leads to two primary issues in existing parallel decoding paradigms:

1. Mode Collapse: Models tend to generate redundant, near-duplicate hypotheses because they lack explicit mechanisms to differentiate between valid but distinct futures.
2. Unreliable Confidence Ranking: Without clear supervision for non-winning modes, confidence scores become ambiguous, making it difficult for downstream planners to select the most probable trajectory.

Current solutions often rely on generating a dense pool of candidates followed by heuristic post-processing (e.g., Non-Maximum Suppression) or winner-take-all (WTA) training strategies, which can lead to unstable mode distinctions and inefficient inference.

2. Methodology

The paper proposes Mode-as-Sequence, a unified decoding framework that reframes multimodal prediction from generating an unordered set of modes to generating an ordered sequence of modes. This approach explicitly models dependencies between modes, where each subsequent mode is conditioned on the previously generated ones.

Core Components

• Sequential Mode Modeling (ModeSeq):

  • Mechanism: Instead of predicting $K$ modes in parallel, the decoder generates them autoregressively ($k=1 \to K$). The $k$-th mode is conditioned on the scene embedding and the embeddings of all previously generated modes ($m_1, \dots, m_{k-1}$).
  • Benefit: This creates an inductive bias where later modes are encouraged to explore complementary behaviors rather than duplicating earlier ones, naturally improving diversity and reducing redundancy without dense candidates.
  • Architecture: Utilizes a recurrent structure with a "Memory Transformer" (to attend to previous modes) and a "Context Transformer" (to ground the mode in the scene).

• Parallel ModeSeq:

  • Motivation: To overcome the latency bottleneck of autoregressive generation, especially for large $K$ or joint multi-agent prediction.
  • Mechanism: Preserves the causal dependency of sequential modeling but decodes all $K$ modes in a single forward pass. This is achieved via masked mode-to-mode self-attention, where a lower-triangular mask ensures that mode $k$ can only attend to modes $j \leq k$.
  • Benefit: Enables efficient, GPU-friendly inference while maintaining the structural benefits of sequential dependency.

• Early-Match-Take-All (EMTA) Training:

  • Strategy: To address sparse supervision, EMTA identifies the earliest mode in the sequence that matches the ground truth (within a threshold $\delta$). This mode is supervised as the positive sample for regression and confidence.
  • Effect: The remaining modes are implicitly pushed to cover alternative, non-matching futures. This encourages the model to learn a diverse set of hypotheses where the first few modes are highly representative and confident.
  • Extension (MA-EMTA): For joint multi-agent prediction, this is extended to supervise the earliest joint mode that matches the ground-truth joint future, ensuring interaction-consistent rollouts.

• Auxiliary Components:

  • Mode Rearrangement: Between decoding layers, modes are reordered based on their predicted confidence scores. This aligns the sequence index with the ranking, ensuring that subsequent refinement layers focus on the most promising hypotheses first.
  • Ranking Regularizer: A margin-based loss is added to prevent confidence inversions (where a non-matching mode scores higher than the matching mode), crucial for metrics like mAP.

3. Key Contributions

1. Unified Framework: Introduction of Mode-as-Sequence, a paradigm that translates unordered mode sets into ordered sequences to explicitly model mode-to-mode dependencies, addressing sparse multimodality without dense candidates.
2. Dual Instantiations:
   • ModeSeq: A recurrent decoder for diverse, well-ranked hypotheses.
   • Parallel ModeSeq: A parallelized realization using causal attention, enabling efficient large-$K$ inference and scalable joint-scene prediction.
3. Training Strategy: Development of EMTA and MA-EMTA, which leverage the sequential structure to assign supervision to the earliest matching mode, thereby improving mode coverage and confidence calibration.
4. State-of-the-Art Performance: Validation on large-scale benchmarks, achieving top rankings in Waymo Open Dataset challenges.

4. Experimental Results

The framework was evaluated on the Waymo Open Motion Dataset (WOMD) and Argoverse 2.

• Waymo Motion Prediction (2024 Challenge):

  • ModeSeq achieved 1st place in the LiDAR-free motion prediction track.
  • It outperformed strong baselines (e.g., MTR v3, QCNet) in ranking metrics (Soft mAP/mAP) and Miss Rate (MR) without using LiDAR point clouds or model ensembling.
  • On the validation set, ModeSeq improved Soft mAP from 0.4508 (QCNet) to 0.4562.

• Waymo Interaction Prediction (2025 Challenge):

  • Parallel ModeSeq achieved 1st place in the Interaction Prediction Challenge.
  • It demonstrated superior performance in both ranking (mAP v2: 0.2949) and trajectory accuracy (Min FDE: 1.6897) compared to recent baselines like IMPACT and TrajFlow.
  • The method successfully handled joint multi-agent futures with interaction consistency.

• Ablation Studies:

  • Sequential vs. Parallel: Both variants outperformed standard parallel decoders (DETR-like) when using the same encoder.
  • EMTA: Replacing WTA with EMTA significantly improved Soft mAP and MR, confirming the benefit of early-match supervision.
  • Mode Extrapolation: The model trained on 6 modes could successfully decode up to 24 modes at inference time, showing diminishing returns but consistent improvement in coverage, validating the sequential modeling capability.
  • Efficiency: While recurrent ModeSeq has higher latency, Parallel ModeSeq achieves ~48ms per scene, making it suitable for real-time deployment.

5. Significance and Claims

The paper claims that Mode-as-Sequence provides a principled solution to the "sparse supervision" problem in multimodal prediction. By shifting from independent parallel generation to causal sequential modeling, the framework:

• Eliminates the need for dense candidate pools and heuristic post-processing (like NMS).
• Naturally induces a confidence ranking, as the sequence order correlates with likelihood, simplifying the learning of calibrated scores.
• Balances accuracy and diversity more effectively than previous methods, as evidenced by simultaneous improvements in ranking metrics (mAP) and trajectory errors (Min FDE).
• Scales efficiently through the Parallel ModeSeq variant, enabling practical deployment in autonomous driving systems where both high-quality hypotheses and low latency are required.

The authors emphasize that their approach does not rely on specific encoder architectures (using QCNet for results but noting compatibility with others) and offers a generalizable framework for both single-agent and complex joint multi-agent forecasting.