ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search

1. Problem Definition

The paper addresses the challenge of interactive multi-camera person search, a task where an agent must identify a specific target individual across a network of synchronized cameras based on a vague witness statement.

Unlike traditional Person Re-Identification (Re-ID) which relies on clear visual queries or static text descriptions, ARGOS frames the problem as an interactive reasoning task under information asymmetry.

• The Agent: Must plan a dialogue, decide when to ask about visual attributes, when to query spatial locations, and when to invoke temporal reasoning tools.
• The Witness: Provides natural language responses that are often ambiguous, uncertain, or incomplete. Crucially, the witness can only reliably answer questions about three specific visual attributes (gender, upper clothing color, lower clothing color); for other attributes, they express uncertainty.
• The Environment: A multi-camera network with a known Spatio-Temporal Topology Graph (STTG) encoding camera connectivity and empirically measured transition times between zones.
• The Goal: Identify the unique target person within a limited turn budget by iteratively eliminating candidates based on appearance, spatial feasibility, and temporal constraints.

2. Methodology

A. The ARGOS Benchmark

The authors constructed a comprehensive benchmark comprising 2,691 tasks across 14 real-world scenarios (7 factory, 7 university campus) using the MTMMC dataset. The benchmark is divided into three progressive tracks:

1. Track 1 (Who): Tests semantic perception. The agent receives a completed dialogue and must extract attributes to filter the gallery.
2. Track 2 (Where): Tests spatial reasoning. The witness reports a sighting in a multi-camera zone; the agent must resolve the specific sub-area using spatial questions and attribute queries.
3. Track 3 (When): Tests temporal reasoning. The witness reports two sightings at different times/locations; the agent must eliminate candidates whose movements violate the physical constraints of the STTG.

Key Dataset Features:

• Gallery: 1,273 unique persons, each annotated with 24 fine-grained visual attributes extracted via Vision-Language Models (VLMs).
• Witness Simulator: A deterministic simulator that returns ground-truth values for observable attributes (in varied natural language templates) and "I'm not sure" for non-observable ones. This ensures reproducibility and isolates agent failures from witness noise.
• STTG (Spatio-Temporal Topology Graph): A directed weighted graph where nodes are cameras/zones and edges represent transitions (OVERLAP, SOFT ADJ, TRAVEL) with statistical transition times (min, median, max).

B. The ARGOS Agent Architecture

The agent is a tool-augmented LLM operating in an observe-think-act loop with four distinct modules:

1. Analyst: Queries the gallery, computes attribute elimination power over the current candidate set, and retrieves zone structures.
2. Planner: Decides the next action (ask attribute, query spatial location, or check temporal feasibility) based on the current state and dialogue history.
3. Interviewer: Executes the chosen action by invoking specific tools (e.g., asking the witness, checking the STTG).
4. Interpreter: Parses the witness's natural language response into structured data and applies filters to update the candidate pool.

Tools: The agent has access to 8 tools, including gallery queries, zone retrieval, witness interaction, and a critical Temporal Feasibility Checker (T5) that validates candidate trajectories against the STTG.

C. Evaluation Metrics

• Top-1 Accuracy: Used for Track 1.
• Turn-Weighted Success (TWS): The primary metric for Tracks 2 and 3. It rewards both correctness and efficiency (fewer turns).
  $$TWS = \frac{1}{N} \sum_{i=1}^{N} s_i \cdot \frac{\tau^*_i}{\max(\tau_i, \tau^*_i)}$$
  Where $s_i$ is correctness, $\tau_i$ is the agent's turn count, and $\tau^*_i$ is the oracle-optimal turn count.

3. Key Contributions

1. First Interactive Multi-Camera Benchmark: ARGOS is the first framework to reformulate person search as an active, multi-turn reasoning problem integrating appearance, space, and time.
2. Spatio-Temporal Topology Graph (STTG): A novel graph structure encoding camera connectivity and empirical transition times, serving as both the structural backbone for task generation and the agent's grounding tool.
3. Progressive Tracks: The three-track design (Who, Where, When) systematically isolates and evaluates semantic perception, spatial disambiguation, and temporal feasibility reasoning.
4. Tool-Augmented Agent Framework: Demonstrates that LLMs equipped with domain-specific tools (especially the STTG checker) significantly outperform direct inference, establishing a new paradigm for agentic surveillance tasks.

4. Experimental Results

Experiments were conducted using four LLM backbones (GPT-5.2, GPT-4o, GPT-5-mini, Claude Sonnet 4).

• Performance Gap: The benchmark remains far from solved.
  • Best TWS on Track 2 (Spatial): 0.383 (Claude Sonnet 4).
  • Best TWS on Track 3 (Temporal): 0.590 (GPT-5.2).
  • These scores are significantly lower than the Oracle (1.0), indicating substantial room for improvement.
• Importance of Tools: Ablation studies show that removing domain-specific tools is catastrophic.
  • Removing the Temporal Tool on Track 3 drops Top-1 accuracy by 49.6 percentage points (from 80.6% to 31.0%).
  • Removing the Spatial Tool on Track 2 drops accuracy by 33.8 percentage points.
  • Without any tools, performance collapses to near-random guessing (11.4%).
• Strategy vs. Accuracy: Strategic reasoning (Planner module) improves efficiency more than raw accuracy. An agent without strategic planning achieved similar Top-1 accuracy to the full agent on Track 3 but required twice as many turns, resulting in a much lower TWS.
• Failure Modes:
  • Track 2: The primary bottleneck is Natural Language (NL) parsing. Agents often fail to correctly extract attributes from varied witness phrasing, leading to incorrect filtering.
  • Track 3: The bottleneck is temporal reasoning. Agents struggle to effectively utilize the STTG to eliminate candidates based on time constraints.
  • Premature Prediction: Some models (e.g., GPT-4o) tend to guess the answer before fully disambiguating candidates, leading to high error rates despite reasonable parsing.

5. Significance and Future Work

• Paradigm Shift: The paper moves the field from passive visual matching to active, interactive reasoning. It highlights that real-world surveillance requires synthesizing fragmented, uncertain human reports with physical constraints.
• Limitations: The current witness simulator is deterministic and does not model human memory errors or inconsistencies. The benchmark is limited to two environments (factory/campus).
• Future Directions: The authors suggest exploring adversarial witness settings (detecting lies), multi-hop STTG reasoning (indirect paths), and applying the protocol to more diverse camera layouts.

Conclusion: ARGOS establishes a rigorous testbed for multimodal spatial intelligence. It proves that while current LLMs possess strong reasoning capabilities, they fundamentally lack the ability to effectively ground their reasoning in physical constraints (STTG) and handle the uncertainty of human interaction without specialized tool augmentation.