BEVTraj: Map-Free End-to-End Trajectory Prediction in Bird's-Eye View with Deformable Attention and Sparse Goal Proposals

The Big Problem: Navigating Without a GPS Map

Imagine you are driving a car in a brand-new city where Google Maps doesn't exist. There are no digital road lines, no traffic light databases, and no "construction zone" alerts.

Most self-driving cars today rely heavily on these perfect, pre-made digital maps (called HD Maps) to know where they can drive. But these maps are expensive to make, hard to keep up to date, and useless if you drive somewhere they haven't covered yet (like a muddy construction site or a new neighborhood).

If the map is wrong or missing, the car gets confused.

The Solution: BEVTraj (The "Eagle-Eye" Driver)

The authors of this paper built a new system called BEVTraj. Instead of asking, "What does the map say?", it asks, "What do my eyes see right now?"

It uses the car's cameras and LiDAR (laser scanners) to build a live, 3D picture of the world directly in front of it. It predicts where other cars and pedestrians will go based only on what it sees in real-time, without needing a pre-built map.

Here is how it works, broken down into three simple concepts:

1. The "Eagle Eye" View (Bird's-Eye View)

Most self-driving systems look at the world like a human driver: straight ahead through the windshield. But predicting where a car will turn is hard if you only see it from the front.

BEVTraj transforms all the camera and laser data into a Bird's-Eye View (BEV).

The Analogy: Imagine you are a hawk flying 100 feet above the street. You can see the whole intersection, the lanes, the pedestrians, and the cars all at once from above. This "God's-eye view" makes it much easier to understand the geometry of the road and where everyone is going.

2. The "Smart Spotlight" (Deformable Attention)

The problem with looking at the whole world from a hawk's view is that there is too much information. The "image" is huge, dense, and full of noise (trees, buildings, sky). If the computer tries to analyze every single pixel, it gets overwhelmed and slow.

BEVTraj uses a trick called Deformable Attention.

The Analogy: Imagine you are in a crowded, noisy party. If you try to listen to everyone talking at once, you'll go crazy. Instead, you use a smart spotlight. You only focus your ears on the specific people you need to hear (the car turning left, the pedestrian stepping off the curb).
How it works: Instead of processing the whole "party" (the whole road), BEVTraj dynamically moves its "spotlight" to only the few spots on the road that actually matter for the car's future path. This makes it fast and efficient.

3. The "Crystal Ball" (Sparse Goal Proposals)

Older systems tried to guess where a car would go by drawing thousands of possible paths (like throwing darts at a board and hoping one hits). This is wasteful and often results in silly predictions (like a car driving through a wall).

BEVTraj uses a Sparse Goal Candidate Proposal (SGCP) module.

The Analogy: Instead of throwing 1,000 darts, imagine a master archer who only shoots three or four arrows. But these aren't random; the archer looks at the wind, the target's movement, and the terrain, then aims perfectly at the most likely spots.
How it works: The system predicts a small, high-quality set of "goals" (destinations) that the car is likely to aim for. It doesn't guess wildly; it intelligently narrows down the options to the most realistic ones, skipping the need for messy cleanup later.

Why Is This a Big Deal?

It's Flexible: Because it doesn't need a pre-made map, this car can drive anywhere in the world, even in places where no one has mapped the roads yet.
It's Robust: If there is a construction zone, a detour, or a road that changed overnight, a map-based car might crash because its map says "Go Straight." BEVTraj sees the cones and the new path and adapts instantly.
It's Accurate: The paper shows that even without the "cheat code" of a perfect map, BEVTraj predicts movements just as well as the expensive map-based systems.

The Bottom Line

BEVTraj is like teaching a self-driving car to be a super-observant human driver rather than a robot following a script. It looks at the world, spots the important details, ignores the noise, and makes smart guesses about what will happen next—all without needing a GPS map to tell it where the road is.

This makes self-driving cars safer, cheaper to build, and ready to drive anywhere on Earth.

1. Problem Statement

Trajectory prediction is critical for autonomous driving safety, yet current state-of-the-art methods heavily rely on High-Definition (HD) maps. These maps provide structured priors (lane topology, road geometry) but suffer from significant limitations:

Cost and Maintenance: HD maps are expensive to create and maintain.
Geographic Limitations: They are often unavailable in unmapped areas or regions with rapid changes (e.g., construction zones, accidents).
Reliability: Online map reconstruction from sensors is prone to errors (misclassification, missing elements).

Conversely, map-free approaches that use raw sensor data directly face their own challenges:

Data Density: Sensor-derived Bird's-Eye View (BEV) features are dense, image-like, and unstructured, unlike the sparse, vectorized nature of HD maps.
Computational Inefficiency: Processing the entire dense BEV grid with global attention is computationally prohibitive.
Context Relevance: Trajectory prediction is agent-centric; only a small, dynamic subset of the scene is relevant to a specific agent's future motion. Global attention mechanisms often dilute critical cues with irrelevant background information.

Core Challenge: How to efficiently and adaptively aggregate task-relevant spatial context from dense, unstructured BEV features without relying on pre-built HD maps.

2. Methodology: BEVTraj

The authors propose BEVTraj, an end-to-end framework that predicts multimodal trajectories directly from raw sensor data (LiDAR and cameras) in BEV space. The architecture consists of two main components:

A. Scene Context Encoder

This module generates a compact scene representation by fusing agent history and environmental context.

Sensor Encoder: Uses BEVFusion to fuse camera images and LiDAR point clouds into a dense BEV feature map ( $B$ ). This ensures consistency between the prediction space and feature representation.
Pre-Encoder: Processes the historical trajectories of surrounding agents. Unlike previous methods that compress time early, BEVTraj applies temporal self-attention followed by social self-attention to model motion dynamics and inter-agent interactions before compressing the temporal dimension.
BEV Deformable Aggregation (BDA): The core innovation for handling dense BEV features.
- Instead of attending to the entire grid, BDA uses deformable attention to selectively sample a sparse set of key spatial locations.
- It utilizes learnable reference positions that are iteratively refined to focus on geometrically relevant regions (e.g., road boundaries, obstacles) relative to the target agent.
- This compresses the dense BEV map into a fixed number of aggregated vectors, enabling efficient, geometry-aware context extraction.

B. Iterative Deformable Decoder

This module predicts the target agent's future trajectory through three stages:

Sparse Goal Candidate Proposal (SGCP):
- Problem: Existing methods (e.g., DenseTNT, MTR) use dense goal sampling or predefined anchors (k-means), which are inflexible and often generate physically implausible candidates.
- Solution: SGCP directly predicts a small set of sparse, sample-adaptive goal candidates.
- Mechanism: It uses learnable "mode queries" conditioned on the agent's dynamic state and the BEV feature map. It employs a two-stage process: first, a dense pool of scene features is aggregated via BDA (spatial exploration); second, a small set of mode queries selects the most relevant goals (goal selection). This avoids mode collapse without needing heuristic post-processing like Non-Maximum Suppression (NMS).
Initial Trajectory Prediction (ITP): Generates initial trajectories based on the proposed goals using cross-attention with the scene context and deformable attention on the BEV map.
Iterative Trajectory Refinement (ITR): Refines the initial trajectories iteratively. It distributes queries across future timestamps, allowing the model to treat each time step as an independent token for better motion modeling. It updates trajectory predictions by computing refinement offsets ( $\Delta x, \Delta y$ ).

Loss Functions

The model is trained end-to-end using a composite loss function:

Goal Loss: $\ell_2$ regression on the best-matching goal.
Displacement Loss: Smooth $\ell_1$ loss on final displacement error.
Dense Trajectory Loss: $\ell_1$ loss for dense future trajectory regression (supervising surrounding agents).
Multimodal Loss: A combination of Negative Log-Likelihood (NLL), KL Divergence, Entropy Regularization, and Auxiliary Displacement Loss to ensure diverse and accurate mode predictions.

3. Key Contributions

Map-Free End-to-End Framework: BEVTraj is the first method to achieve competitive trajectory prediction performance using only raw sensor data (no HD maps), offering superior adaptability to unmapped or dynamic environments.
BEV Deformable Aggregation (BDA): A novel module that adaptively aggregates dense BEV features into a compact representation using deformable attention. It solves the computational inefficiency of processing dense grids while preserving geometric constraints.
Sparse Goal Candidate Proposal (SGCP): A mechanism that predicts a small, realistic set of goals conditioned on the scene and agent state. It eliminates the need for dense sampling and heuristic post-processing, reducing redundancy and improving efficiency.
Agent-Centric Spatial Reasoning: By transforming BEV features into the target-agent-centric coordinate frame during aggregation, the model effectively handles the misalignment between ego-centric sensor data and agent-centric prediction tasks.

4. Experimental Results

The method was evaluated on nuScenes and Argoverse 2 Sensor datasets.

Performance vs. HD Map Methods: BEVTraj achieves performance comparable to state-of-the-art HD map-based methods (e.g., MTR, Wayformer, DeMo).
- On nuScenes, it achieved a Miss Rate of 0.2783, significantly outperforming MTR (0.4240) and Wayformer (0.3868).
- It demonstrated strong robustness in Miss Rate metrics, indicating fewer catastrophic failures.
Robustness in Adverse Conditions: In challenging scenarios (rain, night, construction, heavy traffic), BEVTraj maintained stable performance. Notably, it outperformed map-based baselines in night and construction scenarios, where map-based methods often fail due to reliance on outdated or inaccurate map priors.
Occupancy Prediction: When converted to an occupancy grid task, BEVTraj achieved an IoU of 52.7% (end-to-end) and 78.2% (with ground-truth history), demonstrating its ability to capture coherent scene-level dynamics.
Ablation Studies:
- Removing the Pre-Encoder degraded performance, highlighting the importance of temporal-social modeling.
- Removing SGCP and reverting to dense sampling increased the Miss Rate significantly.
- Using deformable attention was crucial for runtime efficiency; increasing the number of modes with vanilla attention caused a massive latency spike (from ~19ms to ~85ms for 64 modes).

5. Significance and Impact

Scalability and Flexibility: By removing the dependency on HD maps, BEVTraj enables autonomous vehicles to operate in unmapped regions, rural areas, or construction zones where map data is unavailable or unreliable.
Efficiency: The sparse goal proposal and deformable attention mechanisms significantly reduce computational overhead compared to dense sampling or global attention approaches, making real-time deployment more feasible.
Safety: The model's ability to maintain high accuracy in adverse weather and dynamic environments suggests a higher safety margin for autonomous driving systems operating in the real world.
Generalizability: The techniques (deformable attention on BEV, sparse goal proposal) are applicable beyond driving, potentially benefiting robotics navigation and surveillance systems where structured maps are absent.

In conclusion, BEVTraj demonstrates that raw sensor data, when processed with adaptive attention mechanisms, can provide sufficient geometric and semantic priors for high-quality trajectory prediction, challenging the necessity of expensive HD maps for safe autonomous navigation.