⚡ electrical engineering

Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments

This paper presents a data-efficient, interpretable method for dynamic obstacle avoidance in unstructured environments that leverages pretrained UniDepth for monocular 3D reconstruction and SuperPoint/SuperGlue for tracking to compute time-to-collision, enabling effective evasive maneuvers on real-world data without requiring robot-specific training or simulation.

Original authors: Erik Jagnandan, Mulugeta Haile, Gregory Barber, Pratik Chaudhari

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Erik Jagnandan, Mulugeta Haile, Gregory Barber, Pratik Chaudhari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a dense, messy forest with a robot friend. Suddenly, a tree branch swings toward you, or a rock rolls into your path. How does the robot know to stop? Most modern robots try to learn this by playing millions of hours of video games in a perfect, fake world (a simulation) and then hoping they can do the same in the real, messy world. But the authors of this paper say, "No thanks." They argue that this "sim-to-real" approach is like trying to learn to surf by only watching cartoons; the water feels different, and the waves don't behave the same way.

Instead, this team built a robot brain that skips the video game entirely. They used a "pre-trained" vision model called UniDepth. Think of UniDepth as a super-smart artist who has already studied millions of real-world photos and learned how to guess how far away things are just by looking at a single picture (monocular depth). The robot doesn't need to learn how to see; it just borrows this artist's knowledge.

Here is how their system works, step-by-step:

The Eyes and the Brain: The robot takes a video. UniDepth looks at each frame and draws a "depth map," turning the flat 2D picture into a 3D landscape where the robot knows exactly how far away every pixel is.
The Dot Trackers: To figure out if something is moving toward the robot, they need to track specific points. They use two other tools, SuperPoint and SuperGlue, which act like a team of tiny, super-attentive detectives. These detectives find interesting dots (keypoints) on trees, rocks, and bushes and follow them across many frames of video. Unlike older methods that only looked at bright, high-contrast spots, these detectives find dots everywhere, even in the shadows.
The Math Magic: Once the detectives have tracked a dot for at least 5 consecutive frames, the system uses a fancy math engine called the XM solver to smooth out the dot's path in 3D space.
The "Time-to-Collision" (TTC) Clock: For every dot the robot is tracking, it calculates a "Time-to-Collision." This is a simple clock: If I keep moving at my current speed, how many seconds until I hit this dot?
- If a dot is moving away or staying still, the clock doesn't run.
- If a dot is coming closer, the clock ticks down.
- The robot sets a safety rule: If any dot's clock hits 1 second (which means the robot is about 1 meter away at its normal speed of 1 m/s), it's time to act.
The Evasive Move: The robot finds the dot with the lowest TTC (the most dangerous one). It then draws a 2D arrow on the ground pointing away from that dot and moves in the opposite direction.

What did they find?
The team tested this on real-world data from the M3ED dataset, which includes a Boston Dynamics Spot robot (a four-legged robot) walking through forests and cities. They didn't train the robot on any new data; they only spent 74 seconds of footage from one specific sequence ("spot-forest-easy-1") to tweak the settings (hyperparameters).

The results were a mix of "pretty good" and "needs work":

The Good News: The system successfully spotted at least one dangerous moment for 20 out of 22 unique physical obstacles (like trees and rocks) in the test videos. When it did spot a danger, it was right about 84% of the time regarding which direction to move to avoid it. It was also very good at not panicking; it only raised a false alarm in 0.47% of the frames where there was no danger.
The Bad News: The system missed a lot of the actual dangerous moments. It only correctly identified 38% of the frames where a collision was imminent (a "recall" of 0.38).

Why did it miss some?
The authors explain that the system is only as good as its eyes and its memory.

The Memory Gap: Sometimes the "detectives" (SuperPoint/SuperGlue) lost track of a dot before they could watch it for the required 5 frames. If the dot disappears, the robot can't calculate the time-to-collision.
The Eye Glitch: Sometimes the artist (UniDepth) made a mistake guessing the distance, causing the 3D path of a dot to jump around wildly. The math engine (XM solver) tried to fix this, but if the starting data was wrong, the final path was still wobbly, leading to wrong TTC calculations.

The Bottom Line
The paper proves that you don't need to train a robot from scratch with thousands of hours of data or simulate a fake world to get it to avoid obstacles. You can use pre-trained models that already know how the real world looks. While this specific robot isn't perfect yet—it misses about 62% of the immediate dangers—it shows a promising, data-efficient way to build robots that can understand 3D space and react to obstacles without needing a massive, expensive training camp. The authors suggest that future versions could swap out the "detectives" for even better trackers to catch more of those missed moments, but for now, this is a solid step toward robots that can navigate the wild without getting lost in a video game.

Technical Summary: Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models

Problem Statement

Dynamic obstacle avoidance in unstructured outdoor environments remains a significant challenge for autonomous mobile robots. Traditional approaches face two primary bottlenecks:

Data Scarcity: Training end-to-end learning-based models (e.g., Robot Transformer) requires prohibitively large, robot-specific datasets collected over months, creating a scalability barrier.
Sim-to-Real Gap: Reinforcement Learning (RL) policies trained in simulation often fail to transfer to the real world due to discrepancies in visual realism, contact dynamics, and environmental diversity. Furthermore, these policies often lack interpretability, making their behavior difficult to guarantee in safety-critical scenarios.

Additionally, while sensor modalities like LiDAR and stereo vision offer depth information, they have limitations in range and density compared to monocular vision. Stereo depth becomes unreliable beyond 5–10 meters, and LiDAR may miss small or distant obstacles due to sparsity. The authors propose a vision-only approach that leverages the range and density of RGB-based depth estimation to address these issues without requiring large-scale data collection or simulation training.

Methodology

The proposed system is a data-efficient, interpretable pipeline that operates entirely on real-world data using pretrained vision foundation models. It avoids training neural networks from scratch, relying instead on hyperparameter tuning. The pipeline consists of the following stages:

1. Depth Estimation and Keypoint Tracking

Monocular Depth: The system utilizes UniDepth, a transformer-based pretrained model, to generate dense, metric depth maps from RGB video. UniDepth is chosen for its ability to generalize across domains and camera configurations without requiring stereo pairs or LiDAR.
Feature Correspondence: To track motion, the system employs SuperPoint (for keypoint detection) and SuperGlue (for feature matching). Unlike handcrafted methods (e.g., ORB, SIFT) that bias toward high-contrast regions, SuperPoint provides even spatial distribution across the frame, ensuring coverage of obstacles in low-texture or dark areas.
3D Lifting: Tracked 2D pixel coordinates are projected into 3D camera-frame coordinates using the predicted metric depth from UniDepth and known camera intrinsics (pinhole camera model).

2. Trajectory Optimization and Filtering

Bundle Adjustment: To ensure trajectory accuracy, the system performs Scaled Bundle Adjustment (SBA) using the XM Solver over a sliding window of the last five frames.
Noise Mitigation: The pipeline filters out keypoints tracked for fewer than five frames, points beyond 20 meters (deemed too distant for immediate evasion), and points exhibiting frame-to-frame displacements greater than 2 meters (indicating depth estimation errors).
Reinitialization: If a keypoint's trajectory is corrupted by depth discontinuities, it is reinitialized to prevent historical errors from affecting future computations.

3. Time-to-Collision (TTC) Computation

Velocity Estimation: The 3D velocity of each keypoint is computed via linear regression of its position sequence.
TTC Calculation: For each keypoint, the Time-to-Collision is calculated as the ratio of its metric depth ( $d$ ) to its relative velocity along the optical axis ( $v$ ):
$TTC(i) = \frac{d(i)}{\max(-v(i), 0)}$
Points moving away from the camera or stationary are excluded. Keypoints with a Closest Point of Approach (CPA) greater than 3 meters are discarded as they pose no collision risk.
Motion Primitive Generation: The system identifies the keypoint with the minimum TTC. It then generates a 2D motion primitive in the ground plane that moves the robot in the direction opposite to the CPA of this critical keypoint.

Key Contributions

The paper outlines three primary contributions:

Data-Efficient Control: Demonstrating that pretrained vision foundation models enable interpretable and generalizable robotic control without the need for sim-to-real transfer or large-scale robot-specific data collection.
Novel Pipeline: Developing a TTC-based avoidance method that integrates monocular depth estimation (UniDepth), bundle adjustment (XM Solver), and keypoint correspondence (SuperPoint/SuperGlue).
Validation in Unstructured Environments: Validating the approach on real-world data from the M3ED dataset (forests and urban scenes), showing the ability to detect approaching obstacles and generate corrective actions.

Experimental Results

The system was evaluated on the M3ED dataset, specifically using sequences from the Boston Dynamics Spot quadruped in forest and outdoor urban environments. The "spot-forest-easy-1" sequence was used for hyperparameter tuning, while other sequences served for evaluation. A TTC threshold of 1 second (corresponding to a ~1-meter buffer at 1 m/s speed) was used to define collision risk.

Detection Performance:
- Precision: 0.49 (49% of detected frames were true positives).
- Recall: 0.38 (38% of frames with ground-truth TTC < 1s were detected).
- Obstacle Coverage: Despite the low frame-level recall, the system successfully identified at least one frame with TTC < 1s for 20 out of 22 unique physical obstacles present in the test sequences. This suggests that while the system may miss specific frames, it reliably triggers an evasive response for almost all approaching obstacles.
Action Accuracy:
- Among the 185 true positive detections, the system generated the correct evasive motion direction (positive dot product with ground truth) in 84% of cases.
Data Efficiency: The approach required zero model training. Only 74 seconds of data from a single sequence were used for hyperparameter tuning.

Limitations and Future Work

The authors acknowledge that the recall of 0.38 is a significant limitation, primarily caused by:

Tracking Failures: SuperPoint and SuperGlue sometimes fail to maintain long-term correspondences across consecutive frames.
Depth Errors: Inaccuracies in UniDepth's metric depth estimation lead to 3D trajectory discontinuities, forcing keypoint reinitialization.
Optimization Constraints: The XM Solver converges to a mathematically optimal solution based on flawed geometric constraints, leading to physically inaccurate trajectories in challenging scenes.

Future work aims to improve recall by integrating more robust long-term trackers (e.g., DROID-SLAM features or Kalman filters) and to validate the system through a complete end-to-end deployment on the Boston Dynamics Spot platform, measuring latency and real-time feasibility.

Significance

The paper argues that its primary significance lies in bypassing the sim-to-real gap and eliminating the need for massive data collection. By leveraging pretrained models trained on real-world data distributions, the approach offers a "zero-shot" generalization capability that is inherently interpretable. Unlike black-box RL policies, this method explicitly estimates 3D structure and motion, providing transparent reasoning for its avoidance decisions. The results demonstrate that high-level robotic control can be achieved with minimal data (74 seconds) while maintaining robustness across diverse, unstructured obstacle types.