💻 computer science

GHOST: Ground-projected Hypotheses from Observed Structure-from-Motion Trajectories

This paper presents GHOST, a scalable self-supervised method that leverages large-scale monocular structure-from-motion trajectories from dashcam videos to generate ground-projected spatial masks, enabling the training of a deep segmentation network that predicts feasible vehicle path hypotheses in complex urban environments without manual annotation.

Original authors: Tomasz Frelek, Rohan Patil, Akshar Tumu, Henrik I. Christensen

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomasz Frelek, Rohan Patil, Akshar Tumu, Henrik I. Christensen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to drive a car, but you don't have a team of experts to sit down and draw lines on thousands of maps saying, "Here is the road, here is the sidewalk, and here is where you can't go." That would take forever and cost a fortune.

Instead, the researchers behind this paper, GHOST, came up with a clever trick: Let the car teach itself by watching other cars drive.

Here is the story of how they did it, broken down into simple concepts.

1. The Problem: The "Blank Map" Dilemma

Usually, to teach an AI to drive, you need "Ground Truth"—perfect, hand-drawn maps showing exactly where a car should go. But making these maps is slow, expensive, and doesn't work well for weird vehicles like electric scooters or off-road trucks.

The researchers asked: "What if we just watched millions of hours of dashcam videos from regular people driving? Can we learn from their mistakes and successes without anyone telling us what to do?"

2. The Solution: The "Ghost in the Machine"

They created a system called GHOST (Ground-projected Hypotheses from Observed Structure-from-Motion Trajectories). Think of it as a digital detective.

Here is how the detective works in three steps:

Step A: The "3D Movie Maker" (Structure-from-Motion)

The system takes a boring, flat video of a car driving down a street. Using a mathematical trick called Structure-from-Motion (SfM), it acts like a magic eye. It looks at how the buildings and trees move in the background and figures out exactly how the camera (the car) moved through 3D space.

Analogy: Imagine watching a movie of a boat sailing on a lake. Even though the screen is flat, you can tell the boat turned left because the trees on the shore moved to the right. The system does this math to figure out the car's path.

Step B: The "Shadow Puppet" (Projecting to the Ground)

Now the system knows the car's path in 3D space, but it's floating in the air like a ghost. To make it useful, the system calculates how high the camera is off the ground (like measuring the height of a person's eyes). Then, it projects the car's path straight down onto the ground, like a shadow puppet.

The Result: This creates a "shadow" or a mask on the ground showing exactly where the car drove. This is the Teacher's Answer Key. The system didn't need a human to draw this; it just calculated it from the video physics.

Step C: The "Student" (The AI Learner)

Now they have a massive library of videos and their own "Answer Keys" (the shadows). They train a deep learning AI (a student) to look at a single photo of a street and guess where the "shadow" should be.

The Twist: The AI isn't just copying the exact path the car took. Because the training data is so huge and varied, the AI learns the rules of the road. It learns that "roads usually curve here," "you can't drive on the sidewalk," and "at an intersection, you can go left OR right."
The Superpower: When the AI sees a new street, it doesn't just predict one line. It predicts multiple possible paths (hypotheses). It's like a chess player who sees three or four good moves ahead, not just one.

3. Why This is a Big Deal

No Maps Needed: You don't need expensive HD maps. The AI learns the layout of the world just by looking at pictures.
Works for Anything: They tested this on a regular car, and then they fine-tuned it for an electric scooter. Because the AI learned the concept of "drivable space" rather than just "car lanes," it figured out how to ride a scooter on bike paths and sidewalks instantly.
It's Self-Driving: The system is "self-supervised." It creates its own homework and its own answer key. It doesn't need humans to label anything.

4. The "Magic" of the Loss Function

The researchers used a special scoring rule (called an asymmetric loss) to teach the AI.

The Rule: It's okay if the AI guesses a little too much (predicting a path that might be drivable), but it's a big mistake if it misses a path that is drivable.
The Result: The AI becomes bold. Instead of drawing a tiny, safe line right down the middle, it draws a wide, confident area covering all the places a vehicle could go. This helps the car make better decisions in complex situations.

Summary

Think of GHOST as a student who watches thousands of hours of driving videos. Instead of being told "Turn left at the red light," the student watches the video, figures out where the car went, and learns the pattern of driving.

Eventually, when you show this student a single photo of a street they've never seen before, they can point to every spot where a vehicle could safely drive, whether it's a car, a truck, or a scooter. They didn't memorize the map; they learned the logic of movement.

1. Problem Statement

Autonomous driving requires understanding feasible vehicle trajectories within complex urban environments. Traditional approaches rely on:

Explicit semantic supervision: Costly human annotations of drivable areas or lanes.
Calibrated sensor suites: Expensive LiDAR or multi-camera setups.
HD Maps: Static infrastructure data that often lacks dynamic context or connectivity in unmapped areas.

The authors argue that large-scale dashcam video repositories (e.g., YouTube) contain vast amounts of implicit supervision. Every video encodes a real vehicle's interaction with the environment, demonstrating feasible motion conditioned on geometry, traffic rules, and social constraints. The challenge is to distill this implicit ego-motion data into a scalable, self-supervised learning pipeline that predicts multiple feasible trajectory hypotheses from a single RGB image without manual labeling.

2. Methodology

The GHOST framework operates in three main stages: SfM Pipeline, Post-Processing & Mask Generation, and Model Training.

A. Data Acquisition & Structure-from-Motion (SfM)

Source: The authors utilize a 30-hour subset of crowdsourced dashcam videos from YouTube, covering diverse weather, lighting, and hardware configurations (various FoVs, distortions, and mounting positions).
SfM Pipeline: They employ COLMAP to process raw, uncalibrated video sequences.
- Uses SIFT feature extraction and incremental mapping to generate sparse 3D point clouds and unscaled 6-DoF camera poses.
- To manage computational complexity ( $O(N^4)$ for global bundle adjustment), videos are partitioned into independent 20-minute intervals.
Output: Sparse 3D reconstructions and relative camera trajectories.

B. Post-Processing: Ground Projection & Mask Generation

Since SfM outputs are scale-ambiguous, the pipeline must ground the trajectory to the physical world to create training masks.

Camera Height Estimation:
- Assumptions: The vehicle follows a kinematic bicycle model, and the camera's optical axis is roughly parallel to the road surface.
- Process: The system isolates 3D keypoints heuristically corresponding to the road surface (projecting to the lower-central image region). It applies a Theil-Sen estimator (a robust, median-based regression) to the depth-height profile of these points to estimate the camera's height above the ground.
- Global Consistency: The median of height estimates across all frames is used to determine a global camera height ( $H_{cam}$ ).
Mask Generation:
- The 6-DoF camera poses are translated along the gravity vector by $H_{cam}$ to project the trajectory onto the ground plane.
- A lateral width proportional to the camera height ( $1.5 \times H_{cam}$ ) is assigned to create a spatial mask representing the vehicle's footprint.
- Quality Control: "Null frames" (where SfM detects motion but fails to project a valid trajectory) are filtered out, resulting in a high-fidelity dataset of 650,000 annotated frames.

C. Model Architecture & Training

Formulation: Trajectory estimation is treated as a binary image segmentation problem. The network predicts a probabilistic mask of feasible future paths from a single RGB image.
Architecture: A standard U-Net with a ResNet34 encoder (pretrained on ImageNet) and a decoder that upsamples features to full resolution. (A SegFormer backbone was also tested).
Loss Function (Key Innovation):
- The authors use a weighted asymmetric negative log-likelihood loss.
- Rationale: The goal is to predict a distribution of feasible paths, not just copy the single demonstrated path.
- Mechanism: The loss heavily penalizes false negatives (missing a feasible region) while reducing the penalty for false positives (predicting extra feasible regions).
- Effect: This encourages the model to learn a structured prior over all possible valid motions (multi-modal hypotheses) rather than collapsing to a single narrow path. The output is a probabilistic sigmoid mask, preserving uncertainty near boundaries.

3. Key Contributions

Scalable Self-Supervised Pipeline: A method to extract ego-motion supervision directly from unconstrained monocular videos using SfM, eliminating the need for manual annotation or LiDAR.
Trajectory as Segmentation: Reformulating future trajectory hypothesis estimation as an image segmentation task, allowing the model to implicitly capture scene layout, lane topology, and intersection structures.
Structured Multi-Modal Priors: Empirical evidence that large-scale self-supervision induces models that generalize beyond the single demonstrated trajectory, predicting multiple feasible routes simultaneously.
Cross-Platform Transfer: Successful transfer of the model from cars to electric scooters via light fine-tuning, demonstrating applicability to micro-mobility where structured datasets are sparse.

4. Results & Evaluation

Dataset: Evaluated on NuScenes (specifically the OpenLaneV2 subset), which was out-of-distribution regarding geography (Boston/Singapore vs. training data cities) and driving side (Right-hand vs. Left-hand drive).
Metrics: Used Soft Intersection over Union (Soft IoU) to evaluate probabilistic masks without binary thresholding.
Performance:
- ResNet34 achieved a Soft IoU of 0.5579 (Overall) and 0.5718 (Multi-Lane).
- SegFormer achieved 0.5645 (Overall) and 0.5699 (Multi-Lane).
- The models performed comparably or better in multi-lane scenarios, confirming they do not collapse to single hypotheses.
Qualitative Findings:
- The model successfully predicts straight paths, lane changes, and splits.
- It avoids dynamic obstacles (e.g., incoming trucks) even though the ground truth (HD map) might label those paths as feasible.
- Scooter Transfer: Fine-tuning on 40 minutes of scooter data allowed the model to adapt to bike lanes and sidewalks, retaining the ability to predict multiple hypotheses. In contrast, training from scratch on scooter data resulted in overfitting to a single hypothesis.

5. Significance & Limitations

Significance:
- Map-Free: The approach learns spatial priors directly from vision, reducing dependency on expensive HD maps and calibrated sensors.
- Generalizability: The model generalizes across different camera intrinsics, extrinsics, and vehicle types (cars to scooters).
- Scalability: Leverages the massive scale of internet video data, making it feasible to train on diverse, unconstrained environments.
Limitations:
- SfM Constraints: Monocular SfM struggles with degenerate motion profiles (e.g., straight-line driving with low parallax) and dynamic objects, leading to some data loss (filtered out).
- Ground Truth Issues: Evaluation relies on HD maps which may lack topological connectivity or fail to account for dynamic obstacles, potentially penalizing the model for correct physical reasoning (e.g., avoiding a blocked lane).
- Planarity Assumption: The height estimation assumes local road planarity, which can be violated on steep gradients or complex interchanges.

In conclusion, GHOST demonstrates that ego-motion distillation from raw video is a powerful paradigm for learning structured, generalizable path proposals, offering a viable alternative to traditional, annotation-heavy approaches in autonomous driving and micro-mobility.