ORMOT: A Dataset and Framework for Omnidirectional Referring Multi-Object Tracking

This paper introduces ORMOT, a novel task extending referring multi-object tracking to omnidirectional imagery to overcome field-of-view limitations, supported by the newly constructed ORSet dataset and the ORTrack large vision-language model framework.

The Gist

Imagine you are trying to find a specific friend in a crowded city square using a video camera.

The Old Problem (The "Tunnel Vision" Camera):
Most existing video cameras are like looking through a long cardboard tube. You can only see what's directly in front of you. If your friend walks behind a building or moves to the side, they disappear from your view. If you tell a computer, "Find the person who opens the door and then walks up the stairs," a standard camera might lose them the moment they step behind a wall. The computer gets confused, thinking, "Wait, did they vanish? Or is that a different person?" This is the problem with current Referring Multi-Object Tracking (RMOT). It's great at finding things, but it struggles when the camera's view is limited or when the instructions involve a long sequence of events happening over time.

The New Solution (The "360-Degree Fishbowl"):
The authors of this paper, ORMOT, decided to fix this by switching to Omnidirectional Cameras. Think of these cameras not as a tube, but as a fishbowl or a security dome that sees everything around it—360 degrees, all the time.

Here is the breakdown of their three big contributions, explained simply:

1. The New Task: "The All-Seeing Eye"

Instead of just tracking objects, they want the computer to track objects based on language descriptions (like "the guy in the red hat who kicks the ball") but in a 360-degree world.

• The Analogy: Imagine playing a game of "Where's Waldo?" but the book is a giant, rotating sphere. If Waldo walks around the back of the sphere, he doesn't disappear; he just moves to the other side of your view. The new system, ORMOT, is designed to keep track of Waldo even as he circles the globe, understanding that "walking around the corner" is the same person, not a new one.

2. The New Dataset: "ORSet" (The Training School)

To teach computers this new skill, they couldn't use old videos. They needed a special training ground.

• The Analogy: They built a massive library called ORSet. It's like a giant movie studio with 27 different "rooms" (scenes) filmed with 360-degree cameras.
• The Twist: They didn't just label the people; they wrote scripts for them. Instead of just saying "Person A," they wrote complex sentences like, "The person who pushes the door open, walks through the hallway, and then disappears off the left edge of the screen."
• Why it's special: In a normal video, "disappearing off the left edge" means they are gone. In this 360-degree world, "disappearing off the left" just means they are about to "reappear on the right." The dataset teaches the computer to understand these unique 360-degree rules.

3. The New Framework: "ORTrack" (The Smart Detective)

They built a new AI brain called ORTrack to solve these puzzles.

• The Analogy: Imagine a detective who has a super-powerful library (a Large Vision-Language Model) in their head.
  • Step 1 (The Search): When you give the detective a description ("Find the person with the backpack"), the AI doesn't just look for "backpacks." It understands the sentence. It knows what a backpack looks like, even if the person is far away or the image is stretched out by the 360-degree lens.
  • Step 2 (The Zoom): The AI uses a clever trick called "Two-Stage Cropping." It takes a wide, blurry look at the whole room to get the context, then zooms in tight on the specific person to see their face or clothes clearly. It's like looking at a map of the whole city, then zooming in on a specific street corner.
  • Step 3 (The Connection): As the video plays, the AI links the person from one frame to the next. Even if the 360-degree camera distorts the image (making a straight road look curved), the AI uses its "brain" to know, "Ah, that's still the same person walking straight."

Why Does This Matter?

• No More "Lost" Targets: In a 360-degree world, people rarely disappear completely. They just move around the circle. This system ensures the computer never loses track of them.
• Understanding Complex Stories: It can follow long, complicated instructions like, "Find the person who enters the room, talks to the waiter, and then leaves through the back door." Old systems would get lost after the first step; this one follows the whole story.
• Real-World Use: This is huge for things like autonomous driving (cars need to see 360 degrees), smart home security (tracking a pet or person moving around a house), and robotics (robots need to understand their full surroundings).

In a Nutshell:
The authors realized that our current "tunnel vision" cameras are too limited for complex, real-world tracking. They built a new 360-degree world (the dataset), a new set of rules for describing movement in that world, and a super-smart AI detective (ORTrack) that can understand language and keep track of people even when they are moving in a full circle around the camera. It's like upgrading from a pair of binoculars to a crystal ball that never loses sight of the action.

Technical Summary

1. Problem Definition

Multi-Object Tracking (MOT) is a core computer vision task, but existing methods struggle when extended to Referring Multi-Object Tracking (RMOT), where objects must be tracked based on natural language descriptions.

• Limitation of Current RMOT: Existing RMOT methods rely on datasets captured by conventional cameras with a limited Field of View (FoV).
• The Core Issue: In conventional views, targets frequently move out of the frame. This causes:
  1. Fragmented Tracking: Loss of identity when objects re-enter the frame.
  2. Context Loss: Inability to understand "long-horizon" language descriptions (e.g., "People who push open the door and then go up the stairs") because critical actions (like pushing the door) may occur outside the camera's view.
• Proposed Solution: The authors introduce Omnidirectional Referring Multi-Object Tracking (ORMOT), a new task that utilizes 360° imagery to provide continuous spatial coverage and richer contextual cues for language understanding.

2. Key Contributions

The paper makes three primary contributions:

1. ORMOT Task Definition: A novel formulation of RMOT adapted for omnidirectional imagery, addressing FoV limitations and enhancing the model's ability to comprehend complex, long-horizon spatial and temporal language descriptions.
2. ORSet Dataset: A comprehensive dataset constructed based on the JackRabbot Dataset (JRDB).
   • Scale: 27 diverse omnidirectional scenes, 848 language descriptions, and 3,401 annotated objects.
   • Unique Features: Includes Omnidirectional-Specific Descriptors to handle 360° unique challenges, such as:
     • Boundary-crossing motion: Objects disappearing at one edge and reappearing at the opposite.
     • Circumferential orientation: Using clock-face directions (e.g., "3 o'clock") instead of relative "left/right."
     • Projection-aware disambiguation: Correcting for visual distortions (e.g., straight roads appearing curved).
     • FoV transition marking: Distinguishing physical exits from view limitations.
3. ORTrack Framework: A Large Vision-Language Model (LVLM)-driven framework designed specifically for ORMOT. It achieves State-of-the-Art (SOTA) performance in a zero-shot setting without task-specific retraining.

3. Methodology: The ORTrack Framework

ORTrack leverages the open-vocabulary reasoning of LVLMs to detect and track objects based on arbitrary language queries. The pipeline consists of three stages:

A. Language-Guided Detection via LVLM

Unlike traditional detectors restricted to fixed categories, ORTrack uses an LVLM (e.g., Qwen2.5-VL) as an open-vocabulary detector.

• Process: The model takes an omnidirectional frame ($I_t$) and a language description ($L$) as input.
• Mechanism: It performs visual encoding, language encoding, and multi-modal cross-attention to align the text with visual regions, outputting bounding boxes for the referred targets.

B. Two-Stage Cropping-Based Feature Extraction

To handle the distortion and wide-angle nature of omnidirectional images, ORTrack extracts features in two stages to balance global context and local detail:

1. Global Contextual Cropping: The detected bounding box is expanded by a margin ratio ($\alpha$) to capture surrounding context, mitigating the instability of local features caused by wide-angle compression.
2. Fine-grained Target Cropping: The exact target region is cropped.
3. Feature Fusion: Both regions are encoded using a frozen CLIP visual encoder. The final feature is a weighted sum of the global and local features ($f_{final} = f_{local} + \lambda f_{global}$).

C. Cross-Frame Association

To maintain identity consistency across frames:

• Similarity Matching: Cosine similarity is computed between features of detected objects in consecutive frames.
• Hungarian Algorithm: Used to find the optimal one-to-one assignment, minimizing the association cost.
• Lifecycle Management: Unmatched detections start new tracks; tracks without matches for a threshold duration are terminated.

4. Experimental Results

The framework was evaluated on the ORSet test set under zero-shot conditions (no fine-tuning on the dataset).

• Performance Metrics: ORTrack significantly outperformed existing open-source RMOT methods (TransRMOT, TempRMOT).
  • HOTA (Higher Order Tracking Accuracy): ORTrack achieved 9.97, compared to 2.41 and 2.00 for baselines.
  • Association Accuracy (AssA): 16.15 vs. 4.24/9.01.
  • Detection Accuracy (DetA): 6.37 vs. 1.40/0.45.
• Ablation Studies:
  • LVLM Choice: Qwen2.5-VL (7B) proved superior to other LVLMs (DeepSeek-VL, LLaVA-NEXT, InternVL3.5), highlighting the importance of model size and reasoning capability.
  • Feature Encoder: Using CLIP for feature extraction offered a better trade-off between accuracy and computational efficiency (higher FPS) compared to using the LVLM directly for feature encoding.
  • Association Strategy: ORTrack's specific association strategy significantly outperformed standard OC-SORT when paired with the same LVLM detector.
• Qualitative Results: The model successfully tracked objects in complex scenarios, such as correcting for projection distortion (tracking a person on a "straight road" that looks curved) and inferring emotions or specific roles (e.g., a waiter) based on subtle visual cues.

5. Significance and Future Work

• Significance: This work bridges the gap between vision-language tracking and omnidirectional perception. It demonstrates that 360° imagery is not just a visual upgrade but a semantic necessity for understanding long-horizon actions and spatial relationships. The ORSet dataset provides a critical benchmark for future research in zero-shot generalization and temporal grounding.
• Future Directions: The authors identify two main challenges for future work:
  1. Improving detection robustness against severe omnidirectional distortions.
  2. Enhancing stable identity association in scenarios with extreme geometric distortions and rapid scale changes.

In conclusion, ORMOT and the ORTrack framework represent a significant step forward in making multi-object tracking more robust, context-aware, and capable of handling complex, real-world 360° environments through natural language interaction.