Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals

Through collaboration with six police agencies, this paper presents a multi-robot video sensemaking testbed and a prototype tool (MRVS) that integrates LLM-based explanations to reduce operator workload and increase confidence, while highlighting key design requirements and concerns regarding false alarms and privacy.

The Gist

Imagine you are the captain of a ship, but instead of one lookout, you have a fleet of 10 robotic scouts patrolling the ocean around you. Each robot is streaming live video back to your bridge. Your job is to spot a shark, a storm, or a pirate ship.

The problem? If you try to watch 10 video feeds at once, your brain will explode. You'll miss the shark because you were staring at a seagull. This is exactly the problem public safety professionals (like police officers) are facing today. They are understaffed, overworked, and drowning in video feeds from drones, body cameras, and now, ground robots.

This paper is about building a "Super-Pilot's Dashboard" to help them manage this flood of information without losing their minds.

Here is the story of how they built it, broken down into simple steps:

1. The Problem: The "Juggling Act"

Police officers currently have to manually scan hours of video to find a few seconds of something important. It's like trying to find a specific needle in a haystack by looking at every single piece of hay one by one.

• The Risk: If they miss something, people could get hurt.
• The Burden: If they watch too much, they get exhausted and make mistakes.
• The Gap: Existing robots (like the one in New York City) were retired because they were too hard to use and didn't fit into how police actually work.

2. Study 1: Asking the Experts (The "Recipe" Phase)

Before building anything, the researchers went to the "kitchen" and asked 5 experienced police officers what they actually needed. They didn't just guess; they asked: "What should the robot look for?" and "How do you solve a case?"

They discovered three big things:

• Context is King: A person sitting on a bench isn't suspicious at a park, but it is suspicious in a bank lobby at 3 AM. The system needs to understand the "recipe" of the situation, not just the visual.
• The "Needle" Problem: Officers don't need a list of every single thing that happened; they need a summary that says, "Hey, check this out. A guy in a red hoodie dropped a bag here."
• Teamwork: Police work in shifts. If Officer A finds a clue, Officer B needs to know about it instantly without reading a 50-page email chain.

The Result: They created a list of 38 "Events of Interest" (like "suspicious loitering," "assault," or "abandoned bag") and 6 Rules for how the software should behave.

3. The Testbed: The "Training Gym"

To test their ideas, they needed data. But you can't wait for a real crime to happen to test your software.

• The Solution: They turned a university campus into a giant movie set.
• The Cast: 22 student actors.
• The Script: They acted out 38 different scenarios (like a fight, a car crash, or someone stealing a bike) while a ground robot patrolled around them.
• The Dataset: They recorded 20 hours of video (day and night) with these actors. This became the "gym" where they trained their AI.

4. The Solution: MRVS (The "Magic Dashboard")

They built a system called MRVS (Multi-Robot Video Sensemaking System). Think of it as a smart, interactive map that does the heavy lifting for the officer.

How it works (The Magic Features):

• The "Highlight Reel" (Video Debrief): Instead of watching 30 minutes of a robot walking down a street, the AI watches it for you. It cuts the video into "chapters" and gives you a summary card: "At 2:00 PM, a person dropped a bag. Confidence: High." You can click the card and jump straight to that second.
• The "Big Picture" (Situational Overview): It shows all 10 robots on one map. If Robot A sees a fight and Robot B sees a car crash, they both pop up on the map with color-coded urgency. You can see the whole story at a glance.
• The "Detective's Search" (Descriptor Search): This is the coolest part. You don't need a face or a license plate. You can type: "Find me a person wearing a blue jacket and red shoes." The system scans all the videos and shows you the matches. It's like using Google Images, but for video evidence.
• The "Shared Notebook" (Collaboration): If an officer finds a clue, they can tag it and share it with the whole team. The next shift can pick up right where the last one left off, like a shared Google Doc for police work.

5. The Results: Does it Work?

They tested this with 9 real police professionals.

• The Good News: The officers loved it. They said it saved them hours of work. They felt more confident because the AI explained why it flagged something (e.g., "I flagged this because the person ran away from the camera").
• The Bad News: The AI isn't perfect. Sometimes it gets it wrong (false alarms), and sometimes it misses small details (like something happening in the corner of the screen).
• The Verdict: The officers said, "Don't let the AI make the final call. Let it be our assistant that points us in the right direction, but we will make the final decision."

6. The Big Picture: Why This Matters

This paper isn't just about cool robots; it's about trust and safety.

• For the Police: It stops them from burning out. It lets one officer supervise 10 robots effectively, which is crucial because police departments are short-staffed.
• For the Public: It means faster responses to real emergencies and less time wasted on false alarms.
• The Catch: The researchers warn that we have to be careful. If we use this to watch everyone all the time, it invades privacy. The system needs to be transparent, so people know why a robot is looking at them, and the data must be protected.

Summary Analogy

Imagine you are a teacher with 30 students.

• Old Way: You have to watch a live video feed of every single student for 8 hours a day to see who is misbehaving. You will get a headache and miss the kid who is actually in trouble.
• New Way (MRVS): You have a smart assistant. The assistant watches the feeds, highlights the kid who is throwing a paper airplane, and puts a sticky note on your desk saying, "Check this out, it looks like a fight in the back row." You still make the decision to intervene, but you don't have to stare at the screen all day.

This paper shows us how to build that smart assistant for police, making the world safer for everyone without overwhelming the people trying to keep us safe.

Technical Summary

1. Problem Statement

Public safety agencies face chronic understaffing and high injury rates, necessitating scalable situational awareness tools. While ground robots offer a solution for monitoring large areas, their deployment has been hindered by a lack of practical integration into existing workflows. Current challenges include:

• Cognitive Overload: Professionals struggle to manually review fragmented, multi-stream video feeds from moving robots.
• Misalignment: Existing computer vision (CV) anomaly detection models are often trained on fixed-camera, static datasets and do not align with the specific "Events of Interest" (EoIs) defined by police operational needs.
• Lack of Context: Current tools fail to provide the spatial-temporal context, explainability, and collaborative features required for high-stakes decision-making and legal admissibility.
• Data Gap: There is no standardized dataset or testbed for multi-robot ground video sensemaking grounded in real-world public safety scenarios.

2. Methodology

The research followed a two-phase, human-centered design approach involving collaboration with six police agencies and nine public safety professionals.

Phase 1: Formative Study (Testbed Creation)

• Participants: Five active-duty professionals (Lieutenants, Captains, Detectives, Sergeants) with 7–22 years of experience.
• Event of Interest (EoI) Taxonomy:
  • Analyzed 13,234 crime records and 10 public anomaly video datasets.
  • Conducted surveys and interviews to refine a taxonomy of 38 visually observable EoIs (e.g., Arson, Loitering, Trespassing).
  • Categorized EoIs into four operational priority levels: Emergency, Urgent, Moderate, and Advisory.
• Design Requirements (DRs): Derived six core requirements for future systems:
  1. Context-aware EoI explanations.
  2. Automated video detection to reduce manual review.
  3. Unified situational overview across streams.
  4. Unified spatio-temporal awareness (map + timeline).
  5. Descriptor-based search (non-facial attributes).
  6. Team collaboration tools.
• Dataset Construction: Created a unique 20-video multi-robot dataset (10 day/night pairs, ~30 mins each) recorded by a Frodo Zero ground robot. It features 38 scripted EoIs enacted by 22 actors in a campus environment, with ground-truth labels, GPS, and timestamps. All faces and license plates were anonymized.

Phase 2: Summative Study (System Development & Evaluation)

• System Artifact (MRVS): Developed Multi-Robot Video Sensemaking (MRVS), an interactive system integrating a Multimodal Large Language Model (MLLM) backend with a human-centered frontend.
• Backend Architecture:
  • Utilized Gemini 2.0 Flash enhanced with a "Surveillance AI" persona prompt.
  • Integrated BoxMOT (YOLO + DeepOCSort) for multi-object tracking.
  • Processed video in 10-second overlapping segments to generate structured event cards (Type, Description, Rationale, Confidence, Keyframe).
  • Implemented a descriptor-based search module using LLaVA 1.6 for attribute extraction (clothing, vehicle color) and contrastive models for re-identification.
• Frontend Interface: Designed with five divisions to support four workflows:
  • Video Debrief & Situational Overview: Prioritized event cards with AI rationales.
  • Spatio-Temporal Reasoning: Synchronized video wall, map, and timeline.
  • Descriptor Search: Filtering by appearance attributes.
  • Collaborative Workspace: Shared annotations and case management.
• Evaluation:
  • Algorithmic Benchmark: Compared MRVS against HolmesVAD (state-of-the-art anomaly detection) and Gemini 2.0 (baseline LLM) on the custom dataset.
  • Expert Review: Nine professionals performed three investigative tasks (Urgent Response, Routine Review, Descriptor Search) followed by semi-structured interviews.

3. Key Contributions

1. Testbed Environment: The first multi-robot video sensemaking testbed featuring a professional-validated taxonomy of 38 EoIs, a ground-robot video dataset, and six design requirements.
2. System Artifact (MRVS): The first interactive system operationalizing multi-robot video sensemaking, tightly aligning MLLM backends with public safety procedures.
3. Technical Innovation: A novel pipeline combining object tracking with prompt-engineered MLLMs to generate explainable, confidence-scored event summaries and stable appearance descriptors.
4. Empirical Insights: Comprehensive evaluation showing that AI-augmented sensemaking can enable one operator to supervise multiple robots effectively, provided the system addresses false alarms and maintains human control.

4. Results

Algorithmic Performance

MRVS significantly outperformed baselines in detecting anomalies in a multi-robot, mobile context:

• Overall F1 Score: MRVS achieved 0.519, compared to Gemini 2.0 (0.453) and HolmesVAD (0.018).
• Nighttime Performance: MRVS showed the most dramatic improvement, achieving an F1 of 0.540 (vs. 0.438 for Gemini and 0.021 for HolmesVAD), demonstrating robustness in low-light conditions where traditional models fail.
• Trade-off: MRVS had slightly lower precision than Gemini 2.0 (0.473 vs. 0.696) but significantly higher recall (0.663 vs. 0.359). The authors intentionally prioritized recall to minimize missed threats in safety-critical scenarios, accepting a manageable increase in false positives for human verification.

Expert Review Findings

• Workload Reduction: Professionals reported reduced manual scanning time and increased confidence in triage due to AI-generated summaries and confidence scores.
• Trust & Control: Users valued the "explainability" (rationales) and the ability to override AI. They emphasized that AI must remain a decision-support tool, not a replacement for human judgment, especially for legal admissibility.
• Collaboration: The shared workspace was highly praised for enabling seamless handovers between shifts and reducing information loss.
• Concerns: Participants expressed concerns about false alarms, privacy (surveillance overreach), and the need for strict access controls and audit trails.
• Usability: The interface was found intuitive, with the synchronized map-timeline view being a critical feature for spatial reasoning.

5. Significance and Implications

• Bridging the Gap: This work demonstrates how to bridge the gap between emerging robotic/AI capabilities and the operational realities of public safety. It moves beyond "anomaly detection" to "sensemaking" by incorporating context, explanation, and collaboration.
• Scalability: The system proves that a single operator can effectively supervise a fleet of robots, addressing the critical issue of police understaffing.
• Design Guidelines: The paper provides a blueprint for future systems, emphasizing the need for:
  • Configurable EoIs: Systems must allow agencies to tailor event definitions to local laws and contexts.
  • Human-in-the-Loop: AI should accelerate routine tasks but leave critical judgments and accountability to humans.
  • Ethical Deployment: Deployment must consider privacy, community trust, and the risk of reinforcing biases in "hot spot" policing.
• Future Research: The testbed and dataset provide a reproducible foundation for researchers in HCI, Computer Vision, and Human-Robot Interaction to develop and evaluate next-generation public safety tools.

In conclusion, MRVS represents a significant step toward practical, scalable, and ethically grounded multi-robot video sensemaking, offering a viable path to enhance public safety operations without overwhelming frontline professionals.