Multiple Object Detection and Tracking in Panoramic Videos for Cycling Safety Analysis

This paper proposes a novel three-step framework that addresses the unique challenges of panoramic video—such as distortion and boundary continuity—to enhance object detection and tracking for cyclists, demonstrating significant improvements in accuracy and practical effectiveness for real-world cycling safety analysis, particularly in vehicle overtaking detection.

The Gist

The Big Picture: The "Blind Spot" Problem

Imagine you are riding a bicycle through a busy city. You have eyes in the front, but you can't see what's happening directly behind you or far to your sides without constantly twisting your neck. This is a major safety risk. Cars often sneak up on cyclists from behind, leading to dangerous "near-miss" accidents.

Traditionally, researchers study these accidents by looking at police reports. But police reports are like trying to understand a storm by looking at a single raindrop; they are too sparse and don't tell you exactly where or when the danger happened.

To fix this, researchers strapped a 360-degree camera (like a GoPro Max) to a cyclist's helmet. This camera sees everything: front, back, left, and right, all at once. It's like giving the cyclist "super-vision."

The Problem: The "Stretched Pizza" Effect

Here is the catch: 360-degree cameras don't take normal photos. They take a "panoramic" photo that looks like a flat map of the world. If you try to flatten a globe onto a piece of paper, the edges get stretched and distorted.

• The Distortion: Objects near the top and bottom of the image look like they are being squashed or stretched like taffy.
• The Split: If a car is driving right across the "seam" where the left and right sides of the photo meet, the computer sees it as two separate, broken pieces of a car.
• The Confusion: Standard computer vision (AI) models are trained on normal, rectangular photos. If you feed them this "stretched pizza" image, they get confused. They might miss small objects or think a car is two different cars.

The Solution: The "Three-Step Magic Trick"

The researchers developed a clever three-step framework to teach the AI how to see through the camera's distortion.

Step 1: The "Unwrapping" (Object Detection)

Instead of trying to read the whole distorted map at once, the researchers cut the panoramic image into four smaller, normal-looking slices (like cutting a pizza into four wedges).

• The Analogy: Imagine you have a giant, crumpled map. Instead of trying to read the whole thing, you smooth out four small sections at a time.
• The Fix: They run a standard AI detector on these four slices. Because the slices are less distorted, the AI can easily spot cars, people, and bikes.
• The Re-assembly: Once the AI finds the objects in the slices, the system "glues" the pieces back together. If a car was split in half across the edge, the system recognizes it's one car and merges the two halves back into a single box.

Step 2: The "Name Tag" (Object Tracking)

Once the AI spots the objects, it needs to follow them as they move. This is called "Multiple Object Tracking."

• The Problem: In a 360-degree view, a car might disappear off the left edge of the screen and reappear on the right edge. A normal tracker would think, "Oh, that car disappeared! And look, a new car just appeared on the right!" It would give the car a new ID number, losing track of who it was.
• The Fix: The researchers taught the AI two new rules:
  1. The "Wrap-Around" Rule: If a car leaves the left side, the AI knows to look for it on the right side immediately. It treats the video like a loop, not a straight line.
  2. The "Category" Rule: The AI is told, "Don't confuse a bicycle with a truck." Even if a bike and a truck look similar for a split second, the AI checks their "name tags" (categories) to make sure it doesn't swap their identities.

Step 3: The "Overtaking Alarm" (The Application)

Finally, they used this improved system to automatically detect overtaking.

• How it works: The system watches a vehicle. If it sees a car start behind the cyclist, move forward past them, and end up in front of them, it flags it as an "overtake."
• The Result: They tested this on real videos from London. The system was very good at spotting when a car passed a cyclist, even in tricky conditions like night or rain. It achieved an 82% success rate (a high score in this field).

Why Does This Matter?

Think of this technology as a digital safety net.

• For Cyclists: It could eventually lead to smart helmets or bike computers that beep a warning: "Car approaching from the left!" before the cyclist even turns their head.
• For City Planners: It provides hard data. Instead of guessing where dangerous spots are, cities can see exactly where cars are passing too close to cyclists. This helps them design safer bike lanes and enforce speed limits.

The Limitations (The "But...")

The system isn't perfect yet.

• Nighttime: It struggles to see black cars in the dark (the "black car in the night" problem).
• Big Objects: Very large vehicles (like buses) can sometimes get "cut up" by the slicing method, making them hard to track perfectly.
• Head Turns: If a cyclist turns their head quickly, the camera moves, and the AI might get confused about which way the car is actually moving.

Summary

In short, this paper teaches a computer to see the world through a fisheye lens without getting dizzy. By cutting the image into manageable pieces, reassembling the broken parts, and teaching the AI to remember that the world is a circle, they created a tool that can automatically spot dangerous driving around cyclists. This is a huge step toward making cities safer for everyone on two wheels.

Technical Summary

1. Problem Statement

Cyclists face a disproportionate risk of injury, but conventional crash data (e.g., STATS19) is too sparse to analyze risk factors at fine spatial and temporal scales. While naturalistic studies using video data offer a solution, existing computer vision (CV) models struggle with panoramic (360°) equirectangular videos due to three specific challenges:

1. Geometric Distortion: Equirectangular projection causes severe deformation of objects near the poles (top/bottom) and makes distant objects appear very small.
2. Boundary Continuity: Objects crossing the left/right image boundaries (0°/360°) are split, causing standard models to detect them as two separate entities or miss them entirely.
3. Data Scarcity: There is a lack of annotated panoramic datasets, making it difficult to train models from scratch.

Consequently, applying standard pre-trained models (designed for limited field-of-view images) directly to panoramic cycling videos results in poor detection accuracy and frequent ID switches in tracking, hindering the automated analysis of safety-critical events like overtaking.

2. Methodology

The authors propose a novel three-step framework to adapt deep learning models for panoramic cycling videos:

Step 1: Enhanced Object Detection via Projection and Merging

To mitigate distortion and improve detection of small objects, the authors developed a pipeline to process equirectangular images:

• Projection: The original 360° image is segmented and projected into four perspective sub-images (centered at 0°, 90°, 180°, and -90° longitude) with a 120° Field of View (FOV) and 30° overlap. This reduces distortion compared to processing the full equirectangular image.
• Detection: A pre-trained YOLO12n model (trained on the COCO dataset) is applied to each sub-image.
• Reprojection & Merging: Detected bounding boxes are reprojected back to the original equirectangular coordinates. A specific merging algorithm combines boxes of the same category that are split across sub-image boundaries or overlap, ensuring long objects (e.g., buses) are treated as single entities.

Step 2: Improved Multiple Object Tracking (MOT)

The detection results are fed into StrongSORT, a state-of-the-art tracking-by-detection model. The authors modified StrongSORT to handle panoramic specificities:

• Category-Aware Association: The tracker is modified to respect object categories. It penalizes matches between tracks and detections of different (non-adjacent) categories, reducing false associations between, for example, a car and a bicycle.
• Boundary Continuity Handling: To track objects crossing the 180° meridian (the split between left and right views), the system virtually extends the image boundaries. It calculates distances (Mahalanobis and IoU) between a track and a detection on both the original image and two virtual extended copies, selecting the shortest distance to maintain track continuity and prevent ID switches.

Step 3: Automated Overtaking Detection

A rule-based application utilizes the tracking data to detect overtaking maneuvers:

• Direction Recognition: Objects are classified as moving "forward" or "backward" relative to the cyclist based on their movement across the image center over a 5-frame window.
• Overtaking Logic: An overtake is detected when a vehicle moving forward crosses specific longitude thresholds (e.g., -90° to 90°) relative to the cyclist. The system outputs the start/end frames and track IDs for confirmed overtakes.

3. Key Contributions

• Novel Detection Framework: A method to project equirectangular images into perspective sub-images and merge results, significantly improving detection accuracy for small and medium objects without requiring retraining from scratch.
• Panoramic-Aware Tracking: Modifications to StrongSORT that incorporate category filtering and boundary continuity logic, specifically addressing the unique challenges of 360° video tracking.
• Annotated Dataset: Creation of a specialized dataset with 24,454 annotated objects and 224 tracks from London cyclists, including ground truth for overtaking events.
• End-to-End Safety Pipeline: An integrated system that moves from raw video to automated detection of specific safety-critical behaviors (overtaking), validated on real-world data.

4. Experimental Results

The methodology was evaluated on panoramic videos collected in London under diverse conditions (day, night, rain, glare).

• Object Detection:

  • The proposed method improved the Average Precision (AP) for small objects significantly compared to the baseline YOLO12n.
  • At an input resolution of 1920, the proposed model achieved an AP50:95 of 19.4% (with merging) compared to 13.3% for the original model.
  • The merging strategy effectively recovered precision for medium-sized objects that were previously split by projection.

• Multiple Object Tracking (MOT):

  • Integrating category support and boundary continuity reduced False Positives (FP) by 33.5% (from 818 to 544).
  • ID Switches (IDs) were reduced by 10.9% (from 101 to 90).
  • The IDF1 score improved from 45.4% (Improved YOLO only) to 50.0% (with all enhancements), matching the performance of the original model on standard videos.

• Overtaking Detection:

  • The full pipeline achieved an F1-score of 0.82 (Precision: 83.3%, Recall: 80.0%) for detecting overtaking events.
  • False Positives were primarily caused by cyclists checking their shoulders or turning, which altered the relative motion of static vehicles.
  • False Negatives occurred mainly with black cars at night (detection failure) or buses with unstable bounding boxes.

5. Significance and Impact

• Safety Analysis: This research provides a scalable method to generate "near-miss" data from routine cycling videos, overcoming the limitations of sparse crash records. This allows for fine-grained analysis of risk factors (e.g., time of day, vehicle type).
• Policy and Infrastructure: The automated detection of overtaking distances and locations can inform urban planning (e.g., lane widths, buffer zones) and policy enforcement (e.g., minimum passing distance laws).
• Real-Time Potential: The framework is designed for lightweight deployment, suggesting potential for real-time rider alerts (e.g., early warnings of approaching vehicles) via edge devices.
• Generalizability: While tested on cycling, the framework is applicable to any scenario requiring 360° monitoring, such as autonomous vehicles or pedestrian safety analysis.

Limitations & Future Work:
The authors note that detection of very large objects can still degrade due to projection splitting, and night-time detection of dark vehicles remains challenging. Future work includes increasing sub-image overlap, parallel processing for speed, and retraining ReID networks specifically for vehicles to further reduce ID switches.