💻 computer science

SPOT!: Map-Guided LLM Agent for Unsupervised Multi-CCTV Dynamic Object Tracking

This paper introduces SPOT, a map-guided LLM agent that utilizes spatial road data and vehicle dynamics to predict vehicle trajectories across multi-CCTV blind spots without prior training, thereby maintaining continuous tracking and reducing ID switching more effectively than existing methods.

Original authors: Yujin Roh, Inho Jake Park, Chigon Hwang

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yujin Roh, Inho Jake Park, Chigon Hwang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to follow a specific car through a busy city using a network of security cameras. The problem is that the cameras are spaced far apart. There are "blind spots" between them where the car disappears from view. In the real world, this causes the tracking system to lose the car or accidentally switch its ID to a different vehicle, breaking the story of where the car went.

The paper introduces a new system called SPOT (Spatial Prediction Over Trajectories). Think of SPOT not as a traditional computer program that just crunches numbers, but as a super-smart detective who has memorized the entire city map and knows how drivers behave.

Here is how SPOT works, broken down into simple steps:

1. The Detective's Toolkit: The "Map Book"

Most tracking systems struggle because they only see what's in front of the camera lens. SPOT, however, has a special "Map Book."

The Analogy: Imagine the city's road network and the locations of every camera are written down in a giant library of text documents.
How it works: The system breaks the map into small chunks (like chapters in a book) describing roads, intersections, and exactly where each camera is looking. This allows the system to "read" the map just like a human reads a story.

2. Translating the Scene: From Pixels to Reality

When a car is seen on a camera screen, it's just a moving dot of pixels.

The Analogy: It's like seeing a shadow on a wall and trying to guess the size and shape of the object casting it.
How it works: SPOT uses a math trick (called ray-casting) to instantly translate that 2D shadow on the screen into a real 3D location on the actual street map. It knows exactly where the car is in the real world, not just on the screen.

3. The "Blind Spot" Prediction: Guessing the Next Move

This is the magic part. When the car drives out of the camera's view and into a blind spot, the system doesn't panic.

The Analogy: Imagine you are watching a runner disappear behind a building. A normal camera just stops watching. SPOT, however, acts like a coach who knows the runner's habits. If the runner was sprinting and heading slightly left, the coach predicts they will come out on the other side of the building, likely near a specific exit.
How it works:
- Reading the Driver: SPOT analyzes how the car was moving (speed, acceleration, sharp turns) to figure out if the driver is "aggressive" or "cautious."
- Simulating Paths: It runs a mental simulation (like a game of chess) to imagine all the possible roads the car could take.
- The "Brain" (LLM): This is where the Large Language Model (LLM) comes in. Instead of just doing math, the LLM acts as a Strategic Navigation Supervisor. It reads the "Map Book," looks at the driver's habits, and uses common sense to say, "Given this driver's speed and the road layout, it is highly unlikely they would make a U-turn here; they are probably heading to the next intersection."

4. Picking the Next Camera

Once SPOT predicts where the car will reappear, it doesn't just guess randomly.

The Analogy: It's like a relay race. The first runner (Camera A) passes the baton to the second runner (Camera B). SPOT calculates exactly which runner is in the best position to catch the baton.
How it works: It checks which camera's view will overlap with the predicted path of the car. It selects the best "Next Camera" to ensure the car is caught again the moment it leaves the blind spot, keeping the tracking continuous.

The Results: Does it work?

The authors tested this system in a virtual city (a video game simulation called CARLA) designed to look exactly like a real city with traffic lights and intersections.

They compared SPOT against older methods and different types of AI "brains."
The Winner: The system using a specific type of AI (DeepSeek) was the best at guessing where the car would go next. It made fewer mistakes about the car's location and was much better at picking the correct next camera compared to systems that didn't use this "Map Book" and "Detective" approach.

In summary: SPOT solves the problem of losing cars between cameras by giving the computer a "brain" that understands maps and driver behavior, allowing it to predict where a car will pop up next, even when it's completely out of sight.

Technical Summary: SPOT! Map-Guided LLM Agent for Unsupervised Multi-CCTV Dynamic Object Tracking

1. Problem Statement

CCTV-based vehicle tracking systems face significant structural limitations when attempting to maintain continuous trajectories across multiple camera environments. Specifically, the intervals between cameras and restricted Fields of View (FOV) create "blind spots" where visual overlap is absent. In these disconnected segments, traditional methods relying on appearance matching or spatio-temporal constraints often fail, leading to object ID switching, trajectory fragmentation, and a loss of reliability in real-time path prediction. Furthermore, existing research is largely confined to image coordinate systems, with limited exploration of transforming trajectories into world coordinates or reasoning about paths based on road structures and high-dimensional behavioral patterns (e.g., driver habits).

2. Methodology

The paper proposes SPOT! (Spatial Prediction Over Trajectories), a framework designed to track vehicles across multi-CCTV blind spots without prior training. The system operates as a map-guided Large Language Model (LLM) agent that integrates observed object trajectories with structural map data to infer movement paths.

Core Architecture

The framework utilizes a Semantic Orchestration process led by an LLM orchestrator, which coordinates two specialized agents:

Agent 1 (Perception Agent): Transforms 2D pixel coordinates from CCTV video into 3D world coordinates. It utilizes intrinsic and extrinsic camera parameters to perform ray-plane intersection analysis, projecting vehicle positions onto the road plane ( $z=0$ ). It further calculates kinematic states (velocity, acceleration, heading) using the Finite Difference Method to profile driver behavior (e.g., aggressiveness).
Agent 2 (Planning Agent): Performs high-level spatial reasoning and path prediction. It receives the kinematic states from Agent 1 and queries a map-based knowledge base to predict the vehicle's path through blind spots and identify the next likely CCTV location.

Map Documentation and RAG

To enable the LLM to reason about spatial data, the system converts road structures and CCTV configurations into structured textual documents:

Road Representation: Waypoints extracted from road centerlines are defined as document units containing ID, location, and yaw.
Network Topology: Intersections are classified as CCTV candidate regions.
CCTV Modeling: Metadata (position, direction, FOV) constructs FOV polygons.
Road-CCTV Gates: Pre-analyzed intersections between FOV polygons and waypoints define handover points.

These documents are organized using Retrieval-Augmented Generation (RAG) techniques. The system employs a hybrid retrieval strategy combining:

Spatial Filtering: Using an R-tree index to retrieve documents within a specific radius.
Semantic Filtering: Using MinHash to estimate Jaccard similarity for text-based relevance.
VLA Tokenization: Transforming continuous physical states (position, velocity) into discrete semantic tokens (e.g., [SCENE] pos=...) to align numerical physics with the LLM's linguistic space.

Path Prediction and Handoff

For blind-spot traversal, the system employs a Beam Search algorithm on the road graph:

Driver Profiling: An "Aggressiveness Index" is calculated based on the 90th percentile of acceleration history to adjust curvature penalties.
Neuro-Symbolic Fusion: At critical decision points (e.g., intersections), the LLM acts as a "Local Re-ranker." It evaluates candidate paths based on driver intent and road context, outputting a probability that is converted into a Log-Odds term to adjust the beam search scores.
Handoff Selection: The system simulates the intersection between the predicted trajectory and CCTV FOVs to estimate dwell time and entry angles, selecting the Top-K most probable next cameras.

3. Key Contributions

Zero-Shot Spatial Reasoning: The paper introduces a method to structure map information as textual documents, enabling LLMs to perform spatial reasoning and path inference without task-specific training data.
Pixel-to-World Transformation: A robust pipeline for converting CCTV pixel coordinates into world coordinates, enabling trajectory analysis in the actual road environment rather than just image space.
Neuro-Symbolic Integration: A novel fusion of symbolic beam search (for physical feasibility and kinematic consistency) and neural LLM reasoning (for semantic intent and driver profiling) to resolve ambiguities in blind spots.
Explainable RAG: The system provides natural language-based rationales for inferred trajectories and route recommendations, offering explainability for surveillance decisions.

4. Experimental Results

The system was evaluated using the CARLA simulator (Town02) with a virtual urban environment containing non-overlapping CCTV FOVs to simulate realistic blind spots.

Setup: The experiment compared SPOT! against a Heuristic Baseline and four LLM configurations (DeepSeek-chat V3.2, Llama-3-8B, GPT-4o-mini, and Gemini-1.5-Flash).
Metrics: Performance was measured using Average Displacement Error (ADE), Final Displacement Error (FDE), and the accuracy of predicting the next CCTV (Top-1 and Top-K).
Findings:
- All LLM-based variants significantly outperformed the Heuristic Baseline (which had an ADE of 15.78m).
- DeepSeek-chat V3.2 achieved the best overall performance with the lowest ADE (3.64m) and the highest Top-1 CCTV accuracy (70%).
- Llama-3-8B demonstrated strong robustness with a 68% Top-1 accuracy, suggesting viability for edge-side deployment.
- The results indicate that integrating structured map documents with LLM-based reasoning substantially enhances trajectory prediction and next-camera estimation in blind-spot scenarios.

5. Significance and Claims

The paper claims that SPOT! effectively addresses the challenge of maintaining continuous vehicle tracking in multi-CCTV environments with non-overlapping fields of view. By leveraging map-guided LLM agents, the system can infer movement paths in blind spots without relying on external training data or appearance-based re-identification.

The authors posit that this approach holds significant potential to complement or even replace existing learning-based multi-camera tracking techniques, particularly in scenarios where visual overlap is impossible. The study emphasizes the value of combining map documentation with LLM spatial reasoning to provide explainable, real-time path predictions. Future work is identified as extending the logic to ensure continuous ID association at the predicted subsequent CCTV locations and validating the system in real-world urban environments with additional contextual metadata (e.g., traffic signals).