SHARP: Short-Window Streaming for Accurate and Robust Prediction in Motion Forecasting

1. Problem Statement

Autonomous vehicles (AVs) require motion forecasting models that can continuously predict the future trajectories of surrounding agents in real-time. However, existing state-of-the-art methods face significant limitations in dynamic, evolving traffic scenes:

• Heterogeneous Observation Lengths: In real-world driving, agents enter the field of view at different times. Some have long historical tracks, while others are newly detected with only a few frames of history. Most models are trained on fixed-size historical windows and degrade significantly when observation lengths vary.
• Streaming Limitations: Current streaming approaches (e.g., RealMotion, DeMo) often rely on large, overlapping observation windows (e.g., 3 seconds) or fixed numbers of inference passes. This introduces latency for newly detected agents and fails to generalize well when the number of streaming steps changes during deployment.
• Temporal Consistency: Many methods treat consecutive predictions as independent snapshots or rely on implicit context propagation, failing to explicitly model the correspondence of specific agents across time steps.

2. Methodology: SHARP

The authors propose SHARP (Short-window streaming for Accurate and Robust Prediction), a framework designed to handle continuously evolving scenes with variable observation lengths.

Core Architecture

SHARP utilizes an efficient Transformer-based architecture that avoids heavy iterative refinement during decoding to minimize latency. The pipeline consists of:

1. Agent and Lane Encoding:
   • Agents: Historical states are encoded using a compact self-attention encoder ($f_A$) after normalizing to local coordinates.
   • Lanes: Map information is encoded using a PointNet-based encoder ($f_L$).
2. Instance-Aware Context Streaming ($f_{IA}$):
   • This is the core innovation. Unlike prior methods that rely solely on positional proximity, SHARP explicitly models agent correspondences across time steps.
   • It uses a cross-attention mechanism with a learnable attention mask. This mask selectively modulates weights to focus on the same agent instance from the previous time step ($t-1$) to the current step ($t$), even if the agent's position has shifted significantly.
   • This allows the model to maintain consistent latent representations for agents as they move through the scene, enabling robust propagation of motion features over short windows.
3. Target-Centric Context Encoding ($f_{TC}$):
   • To enhance prediction, the model aggregates features around the endpoints of previous predictions ($F_{t-1}$).
   • These "target-centric" features provide auxiliary context about regions the agent is likely to move toward, improving accuracy especially for short observation windows.
4. Trajectory Decoding:
   • A DETR-like decoder uses learnable mode queries to predict multi-modal future trajectories.
   • It employs Trajectory Relay, where past predictions are transformed into the current coordinate frame and fused via cross-attention to refine current outputs without being rigidly constrained by them.

Dual Training Objective

To ensure robustness across varying context lengths, SHARP employs a Dual Training Strategy:

• $L_{stream}$: Trains the model to predict trajectories using the accumulated streaming context (incorporating $S_{t-1}^{enc}$ and $F_{t-1}$).
• $L_{chunk}$: Trains the model to predict trajectories using only the current observation chunk (snapshot-based), acting as a regularizer.
• Objective: $L_{dual} = L_{stream} + L_{chunk}$.
• Benefit: This forces the model to learn robust temporal representations that work whether it has a long history or a short, fresh observation, preventing overfitting to specific context lengths.

Short-Window Processing

Instead of using large 3-second windows, SHARP processes short, non-overlapping windows (e.g., 1 second). This aligns with the physical reality of data arriving incrementally, reduces latency for new agents, and allows for more frequent gradient updates during training.

3. Key Contributions

1. Novel Streaming Framework: Introduction of SHARP, which explicitly addresses the challenge of heterogeneous observation lengths in continuously evolving scenes.
2. Instance-Aware Context Streaming: A novel module that uses attention masks to explicitly track agent identities across time steps, improving temporal consistency compared to implicit positional methods.
3. Dual Training Scheme: A training objective that jointly optimizes for both streaming context and single-chunk inputs, ensuring the model generalizes to any observation horizon.
4. State-of-the-Art Performance: Achieving new benchmarks in streaming multi-agent prediction while maintaining minimal latency.

4. Experimental Results

The authors evaluated SHARP on Argoverse 2 (AV2), Argoverse 1 (AV1), and nuScenes.

• Evolving Scene Robustness (AV2):
  • SHARP was tested across varying context lengths ($T_{cl}$) and prediction time steps ($t_p$).
  • Unlike baselines (RealMotion, DeMo) whose performance degrades significantly when the number of streaming steps changes, SHARP maintains consistent accuracy.
  • Example: At $t_p=5s$ with a 3s context, SHARP achieved a minFDE6 of 1.44m, outperforming DeMo (1.78m) and RealMotion (1.65m).
• Multi-Agent Benchmark (AV2):
  • SHARP set a new State-of-the-Art (SOTA) for streaming multi-agent prediction.
  • It achieved an avgMinFDE6 of 1.15m and avgBrierMinFDE6 of 1.80, outperforming both snapshot-based (e.g., MTR++) and streaming-based (DeMo) methods.
• Single-Agent Benchmark (AV2 & AV1):
  • SHARP matched or exceeded SOTA performance on standard single-agent benchmarks, demonstrating that streaming capabilities do not compromise accuracy on fixed-length tasks.
• Latency:
  • SHARP is highly efficient. On an NVIDIA RTX 3090, it achieves 33ms latency for a batch of 16 agents, compared to 35ms (DeMo) and 44ms (RealMotion). It scales efficiently to larger batches (57ms for 64 agents).
• Ablation Studies:
  • Removing the Instance-Aware module caused performance drops, especially for longer contexts.
  • Removing Dual Training significantly hurt performance on short contexts (e.g., 1s input), confirming its necessity for robustness.

5. Significance

• Real-World Applicability: SHARP bridges the gap between theoretical benchmarks (fixed windows) and real-world deployment (continuous, variable-length streams). It solves the "cold-start" problem for newly detected agents without sacrificing accuracy for long-tracked agents.
• Efficiency: By using short windows and avoiding iterative refinement, SHARP offers a lightweight solution suitable for resource-constrained AV hardware.
• Community Impact: The paper highlights the need for evaluation protocols that test models under evolving scene conditions rather than just static snapshots, setting a new standard for future motion forecasting research.

In summary, SHARP represents a significant leap forward in motion forecasting by combining explicit instance tracking, short-window processing, and a dual-objective training regime to deliver robust, low-latency predictions in dynamic traffic environments.