💻 computer science

TACO: Trajectory Aligning Cross-view Optimisation

TACO is a tightly-coupled IMU and fine-grained Cross-View Geo-localisation pipeline that enables robust, GNSS-independent absolute positioning by dynamically triggering satellite-image matching to correct inertial drift, achieving a nearly six-fold reduction in trajectory error on the KITTI dataset.

Original authors: Tavis Shore, Oscar Mendez, Simon Hadfield

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tavis Shore, Oscar Mendez, Simon Hadfield

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car in a city where the GPS signal is completely broken. Maybe you are in a deep "urban canyon" surrounded by tall buildings, or perhaps someone is jamming the satellite signals. Your car's internal computer (the IMU) knows how fast you are going and which way you are turning, but it has a fatal flaw: it gets confused over time.

Think of the IMU like a person walking in a dark room with their eyes closed. They can count their steps and guess their direction, but after a few minutes, they might think they are in the kitchen when they are actually in the garage. This is called drift. Without a real map or a GPS signal, they will wander further and further away from their true location.

TACO is a new system designed to fix this problem without needing a working GPS. It combines the car's internal sensors with a clever trick: looking at the sky from the ground.

Here is how TACO works, broken down into simple concepts:

1. The "Smart Pause" (The Trigger)

Most navigation systems that use satellite images try to look at the sky every single second. This is slow and uses a lot of battery. TACO is smarter. It listens to the "drunk" IMU.

The Analogy: Imagine the IMU is a person walking in the dark. TACO is a friend holding a flashlight. The friend doesn't shine the light every step; they wait until the walker starts to stumble or drift too far off course.
How it works: TACO calculates exactly how much the IMU might be wrong. As soon as the potential error gets too big (about 1 meter), it wakes up the camera to take a picture.

2. The "Five-Point Guess" (The Search)

When the camera wakes up, it doesn't just guess one spot. It knows the car has moved since the last time it looked.

The Analogy: Imagine you are looking for a friend in a crowded park. You know they were near the fountain 10 seconds ago, and you know they were walking north. Instead of looking at just the fountain, you look at the fountain, a spot 10 feet north, 10 feet south, 10 feet east, and 10 feet west.
How it works: TACO takes five different satellite images around where it thinks the car should be. It compares the street view (what the car sees) with these five satellite views to find the best match. This keeps the system fast and accurate.

3. The "Compass Check" (The Gatekeeper)

Sometimes, a street looks like another street (like a grid of identical houses). The computer might get confused and say, "You are here!" when you are actually 90 degrees away.

The Analogy: If your friend says, "I'm at the park," but your compass says you are facing North and the park is to the East, you know something is wrong.
How it works: TACO checks the car's compass. If the satellite match says the car is facing a different direction than the compass says, TACO rejects that guess. It only accepts fixes that make geometric sense.

4. The "Confidence Score" (The Filter)

Not all matches are perfect. Some are clear, some are blurry.

The Analogy: If a friend gives you directions, you trust them more if they are sure ("I'm 100% sure") than if they are guessing ("I think maybe...").
How it works: TACO assigns a confidence score to every match. If the match is weak, it only nudges the car's position slightly. If the match is strong, it corrects the position firmly. This prevents the system from getting confused by bad data.

The Results: Why It Matters

The researchers tested TACO on real driving data (the KITTI dataset) where the GPS was turned off.

Without TACO: The car's internal navigation drifted wildly, ending up 97 meters away from the true path after a while.
With TACO: The car stayed on track, ending up only 16 meters away. That is a 6 times improvement.
Efficiency: It does this while only using the camera for about 5% to 10% of the time. The rest of the time, the camera sleeps, saving power.

The Bottom Line

TACO is a system that lets a robot or car navigate perfectly in places where GPS fails (like tall cities or war zones). It uses a tiny bit of satellite imagery, only when absolutely necessary, to "reset" the car's internal compass and keep it from getting lost forever. It runs fast on standard computer chips, making it ready for real-world use today.

Technical Summary: TACO – Trajectory Aligning Cross-view Optimisation

Problem Statement

Autonomous mobile robotics relies heavily on Global Navigation Satellite Systems (GNSS) for absolute positioning. However, GNSS signals are frequently occluded, reflected, or jammed in critical environments such as urban canyons and contested airspace. While Visual-Inertial Odometry (VIO) and visual SLAM provide high-rate relative motion, they accumulate unbounded drift over time and require either pre-built maps or revisits to previously mapped regions to correct errors. Neither approach is suitable for first-traversal missions in unmapped terrain without external communication.

Recent Cross-View Geo-Localisation (CVGL) methods have advanced from coarse tile-level retrieval to fine-grained metric pose regression, achieving sub-tile accuracy. However, these methods have historically been evaluated only as "one-shot" localisers (matching a single ground image to a single satellite crop) and have never been integrated as the primary absolute fix in a live, continuously running pipeline. The challenge is to create a drift-bounded, map-free localiser that operates indefinitely using only onboard sensors (IMU and monocular camera) and a single initial GNSS reading, without requiring live external communication or expensive sensor suites.

Methodology

TACO (Trajectory Aligning Cross-view Optimisation) proposes a tightly-coupled pipeline that fuses IMU dead-reckoning with fine-grained CVGL. The system operates on a single consumer GPU and utilizes a monocular camera and IMU, consuming GNSS only at startup to set the initial UTM origin.

Core Components

IMU Preintegration and Error Modeling:
The system maintains a dead-reckoning estimate of position and heading. A closed-form error model tracks accumulated uncertainty since the last accepted fix. It calculates cross-track error (driven by gyro random walk) and along-track error (driven by accelerometer bias drift). This model generates a scalar trigger value, $\epsilon_{IMU}$ , which scales with distance traveled and time elapsed.
Triggering and Multi-Crop Search:
A CVGL inference is triggered when the estimated IMU error exceeds a threshold (1.0 m) or a time limit (2.0 s) is reached. To ensure the query remains within the CVGL matcher's capture radius despite IMU drift, TACO employs a forward-biased search:
- Forward Bias: The search center is shifted forward along the current heading by a fraction of the estimated error ( $\delta_{fwd}$ ) to account for vehicle movement between the last anchor and the trigger.
- Five-Point Cross: Instead of a single query, the system evaluates five candidate satellite crops around the biased center (forward, backward, left, right, and center). The candidate with the highest match weight is selected. This keeps the inference cost fixed at five forward passes per trigger.
CVGL Module (FG2):
TACO utilizes FG2, a fine-grained matcher that regresses metric pose within a satellite tile. It lifts ground features to a Bird's-Eye-View (BEV) grid and matches them against satellite cells using a shared DINOv2 backbone and Procrustes alignment. The output is a 2D translation, rotation, and a confidence weight.
Yaw-Residual Gate:
Before fusing a CVGL fix, the system checks for heading consistency. Since the satellite tile is rotated to align with the onboard compass, the rotation matrix returned by the network encodes the residual heading error. Fixes with a residual exceeding $\approx 20^\circ$ are rejected to prevent the filter from locking onto symmetric structures (e.g., parallel parking bays) that yield high match weights but geometrically incorrect poses.
UKF Fusion with Anisotropic Noise:
Accepted fixes update an Unscented Kalman Filter (UKF). The measurement noise model is anisotropic and scaled by the inverse of the match weight. This reflects the geometry of the BEV matcher, which typically resolves along-heading position more accurately than cross-heading position. The filter updates only the position state, preserving the compass-derived heading and IMU velocity estimates.
Offline Smoothing:
A factor graph (GTSAM) runs offline for post-processing. It incorporates the same IMU odometry and CVGL fixes, augmented by geometrically vetted loop closures. Loop closures are detected based on spatial proximity, temporal gap, and a path-to-chord ratio test to distinguish true revisits from parallel road segments.

Key Contributions

First Live Fine-Grained CVGL Localiser: TACO is the first system to use fine-grained CVGL as the sole absolute anchor in a live, indefinite-runtime localiser, replacing live GNSS with a single startup fix.
Error-Triggered Forward-Biased Search: An IMU-error-triggered mechanism combined with a forward-biased five-point search ensures the matcher operates within its capture radius at a fixed, low inference cost.
Geometric Consistency Gating: A yaw-residual gate and anisotropic per-fix noise model ensure that only geometrically consistent and confidence-weighted fixes are admitted into the state estimator.

Results

The system was evaluated on the KITTI odometry benchmark (eight sequences) using a single consumer GPU (RTX 3090).

Accuracy: On the representative group of sequences (urban drives with turns), TACO reduced the median Absolute Trajectory Error (ATE) from 97.0 m (IMU-only) to 16.3 m, a 5.9× reduction. It outperformed other minimal-sensor baselines (DROID-SLAM, DPVO) on all representative sequences.
Drift Bounding: While IMU-only error grew super-linearly with distance, TACO maintained a bounded error envelope (approximately 30 m) over multi-kilometer trajectories.
Efficiency: The system operates with a CVGL duty cycle of only 5–10% (camera active only during triggers). The per-frame fusion cost is < 0.1 ms, and the total GPU cost is 2.0–4.1 GPU-seconds per 100 m, an order of magnitude lower than comparable map-aided localisers that run CVGL on every frame.
Hardware: The system runs in real-time on a single RTX 3090 with ~1.5 GB VRAM usage, suggesting feasibility for deployment on embedded modules like Jetson Orin.

Significance and Claims

The paper claims that TACO demonstrates the viability of using fine-grained CVGL not just as a one-shot retrieval tool, but as the primary absolute correction signal in a sensor-fusion loop for moving ground vehicles. By tightly coupling IMU dead-reckoning with sparse, triggered CVGL updates, TACO achieves drift-bounded, map-free, and GNSS-free localisation using only sensors standard on most outdoor autonomous platforms.

The authors emphasize that this approach solves the "first-traversal" problem in unmapped terrain without requiring expensive LiDAR, pre-built maps, or continuous external communication. The system's ability to maintain bounded error over indefinite runtimes using a single startup GNSS fix represents a significant step toward robust autonomy in GNSS-denied environments. The paper modestly notes that further accuracy improvements are gated by the underlying CVGL backbone (FG2) rather than the fusion architecture itself.