🤖 machine learning

Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teaming

Co-GLANCE is a real-time, onboard system for heterogeneous robot teams that distills vision-language model reasoning into an efficient end-to-end model and employs conformal prediction to quantify perceptual uncertainty, thereby enabling statistically valid, active perception that significantly outperforms cloud-based baselines in accuracy and latency.

Original authors: Michal P. Podolinsky, Neel P. Bhatt, Pranay Samineni, Rohan Siva, Christian Ellis, Ufuk Topcu

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michal P. Podolinsky, Neel P. Bhatt, Pranay Samineni, Rohan Siva, Christian Ellis, Ufuk Topcu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are leading a rescue mission in a dense, messy forest. You have two types of helpers: a drone flying high above and a four-legged robot walking on the ground.

The problem is that neither of them can see everything on their own.

The drone sees the big picture but gets blocked by tree branches and can't see what's hiding underneath a bush.
The ground robot can walk under the bushes but can't see over tall walls or through thick trees.

If they just guess what's hidden, they might miss a person in trouble or waste time checking empty spots. This paper introduces a new system called Co-GLANCE to solve this "blind spot" problem.

Here is how it works, broken down into simple concepts:

1. The "Smart Brain" vs. The "Lightweight Worker"

Usually, to figure out what is hidden, you might need a super-powerful computer brain (called a Vision-Language Model) that can "think" about the scene. But this brain is so heavy and slow that it can't fit on the robots; it would have to send data to the cloud and wait for an answer, which takes too long.

Co-GLANCE's solution: They taught a small, fast, lightweight robot brain (a distilled model) to do the thinking.

The Analogy: Imagine a master chef (the big cloud brain) teaching a sous-chef (the onboard robot) how to cook a complex dish. The sous-chef learns the recipe so well that they can cook it instantly in the kitchen without calling the master chef for every step.
The Result: The robot can now instantly spot "occlusions" (hidden areas) and decide which robot should check them, all without needing an internet connection.

2. The "Self-Correction" Trick

When teaching the small robot, the team didn't just copy the master chef's notes. They added a self-review step.

The Analogy: It's like the master chef draws a sketch of a hidden area, but the sketch is a bit messy. Before the sous-chef learns from it, the master chef looks at the sketch again and says, "Wait, that line is wrong; fix it," or "You missed a spot here."
The Result: This "second look" makes the training data much cleaner, so the small robot learns to be much more accurate than if it just copied the messy notes.

3. The "Confidence Meter" (Uncertainty Quantification)

This is the most important part. The system doesn't just guess; it knows how sure it is.

The Analogy: Imagine a security guard checking IDs.
- High Confidence: If the ID looks perfect and the person looks familiar, the guard says, "Go ahead," and lets them pass immediately.
- Low Confidence: If the ID is blurry or the person looks suspicious, the guard doesn't guess. Instead, they say, "I'm not sure. I need a second opinion."
How Co-GLANCE uses this:
- If the robot is very sure about a hidden area, it sends the right robot to check it.
- If the robot is unsure, it triggers "Active Perception." This means it automatically dispatches the ground robot (or both robots) to get a closer look to clear up the confusion. It never guesses blindly in dangerous situations.

4. The "Team Huddle" (Robot Allocation)

Once the system spots a hidden area, it has to decide who goes to check it.

The Analogy: Think of a sports coach deciding who plays which position.
- If the hidden spot is under a low roof, only the ground robot can fit.
- If it's behind a simple wall, either the drone (flying over) or the ground robot (walking around) can do it.
- If it's a super complex mess (like a dense thicket behind a wall), the system might decide to send both robots to be safe.
The Result: The system assigns the job to the most capable robot, saving time and energy.

What Did They Prove?

The team tested this in real outdoor environments with trees, buildings, and obstacles.

Speed: Because the "brain" is now small and onboard, the system is 350 times faster than sending data to the cloud.
Accuracy: It was 25% better at spotting hidden areas and 36% better at assigning the right robot compared to the slow, cloud-based methods.
Reliability: They released a new dataset (a collection of real-world photos from both robots) so other researchers can test similar ideas.

In short: Co-GLANCE is a system that lets a team of different robots work together in the wild. It uses a fast, onboard "brain" to spot where they can't see, knows exactly when it's unsure, and instantly sends the right robot to get a better look, all without needing a Wi-Fi signal.

Technical Summary: Co-GLANCE

Problem Statement

Heterogeneous robot teams (specifically air-ground combinations) operating in unstructured outdoor environments face significant perceptual uncertainty due to occlusions. No single viewpoint provides reliable scene understanding; for instance, vegetation may block an aerial robot's view while remaining transparent to a ground robot, whereas obstacles insignificant from above may fully occlude a ground robot. Resolving these uncertainties requires both scene-based contextual reasoning (to understand what is hidden) and capability-aware robot allocation (deciding which robot can best resolve the occlusion).

While Vision-Language Models (VLMs) offer strong semantic priors for this reasoning, they present three critical limitations for onboard deployment:

Computational Prohibitions: They are too heavy for real-time onboard inference.
Cloud Dependency: They often require cloud-based inference, introducing latency and connectivity issues.
Uncalibrated Uncertainty: They lack statistically valid uncertainty quantification, making them unreliable for safety-critical active perception decisions.

Existing active perception methods often rely on heuristic or uncalibrated confidence, failing to provide guarantees on when additional sensing is truly needed.

Methodology: Co-GLANCE

The authors introduce Co-GLANCE, a real-time, onboard perception and decision-making framework designed to resolve perceptual uncertainty in heterogeneous robot teams. The system operates through three core components:

1. Knowledge Distillation with Contextual Self-Review

To eliminate cloud dependency, Co-GLANCE distills the semantic reasoning capabilities of a large VLM into a lightweight, end-to-end YOLO-seg-nano model for onboard inference.

Pipeline: The process begins with a VLM generating occlusion keywords from aerial frames, which are passed to an open-vocabulary segmentation model to create candidate masks.
Contextual Self-Review: Recognizing that VLM-generated masks often misalign with intended regions, the system employs a multi-turn conversational mechanism. The candidate masks are fed back to the VLM, which performs a "self-review" to remove incorrect masks, refine misaligned regions, propose new keywords, and assign platform allocation labels (ground, both, or either).
Training: The lightweight model is trained on these refined pseudo-labels, enabling it to perform occlusion segmentation and robot allocation in a single forward pass.

2. Two-Stage Uncertainty Quantification

To ensure reliability without cloud inference, Co-GLANCE combines selective abstention and conformal prediction to provide statistically valid coverage guarantees.

Stage 1 (Risk-Controlled): Uses selective abstention to identify high-confidence predictions. If a prediction's confidence exceeds a calibrated threshold ( $\hat{\lambda}$ ), it is accepted as a "singleton" decision with a guaranteed error rate (e.g., $1-\alpha$ ).
Stage 2 (Coverage-Controlled): For predictions below the threshold (abstentions), the system applies conformal prediction to generate a prediction set. This ensures that the true label is contained within the set with a specified probability ( $1-\epsilon$ ), providing a statistically valid fallback rather than a random guess.

3. Uncertainty-Driven Active Perception

The calibrated uncertainty estimates directly drive robot dispatch:

High Confidence: If Stage 1 confirms a singleton prediction (e.g., "occlusion exists, ground robot needed"), the system proceeds with the assigned plan.
Low Confidence/Abstention: If the system abstains (uncertainty is too high), it triggers active perception.
- For robot allocation, abstentions trigger a conservative dispatch of both robots to the region to ensure coverage.
- For object detection, abstentions dispatch the ground robot to acquire a closer viewpoint.
- The system routes robots using a Heterogeneous Vehicle Routing Problem (HVRP) solver, minimizing travel costs while respecting platform constraints (e.g., ground robots are weighted higher for traversal cost).

Key Contributions

Onboard Uncertainty-Aware Perception: Co-GLANCE is the first system to distill VLM reasoning for occlusion segmentation and robot allocation into a lightweight onboard model, achieving 25% higher occlusion detection accuracy and 36% higher robot allocation accuracy compared to cloud-based VLM baselines.
Calibrated Uncertainty for Active Perception: By integrating selective abstention with conformal prediction, the system provides statistically valid guarantees for segmentation, allocation, and detection outputs, directly triggering active perception only when necessary.
Contextual Self-Review Mechanism: A novel multi-turn refinement process that improves the consistency of VLM-generated supervision, significantly boosting recall and allocation accuracy over standard VLM baselines.
Real-World Deployment and Dataset: The authors validate the system on a real air-ground team (DJI Matrice 600 and Boston Dynamics Spot) and release a multimodal air-ground dataset containing over 4,000 synchronized frames, addressing a gap in existing literature which often relies on simulation or homogeneous teams.

Experimental Results

Evaluated in semi-structured outdoor environments with natural vegetation and infrastructure:

Accuracy: Co-GLANCE outperformed the cloud-based VLM baseline (ChatGPT-5.4 + Grounding DINO + SAM) by 25% in occlusion detection and 36% in robot allocation accuracy.
Efficiency: The distilled model reduced per-frame inference latency by 350 $\times$ (from >13 seconds to 37 ms) and eliminated API token usage entirely.
Uncertainty Calibration: The two-stage scheme successfully maintained error rates within target bounds (e.g., $\le$ 15% for allocation) while managing abstention rates (approx. 10% for allocation, 50% for person detection).
Dataset: The released dataset includes synchronized RGB, RTK GPS, and IMU data from both platforms, supporting future research in heterogeneous perception.

Significance

The paper argues that Co-GLANCE demonstrates the feasibility of practical, real-time deployment of complex semantic reasoning in heterogeneous robot teams without relying on cloud connectivity. By distilling VLM capabilities and rigorously quantifying uncertainty, the system bridges the gap between high-level semantic reasoning and the reliability requirements of safety-critical active perception. The authors position this work as a step toward autonomous operation in unstructured environments where single-viewpoint reliability is insufficient.

Limitations

The authors acknowledge several limitations:

Evaluation was conducted in semi-structured environments with a limited number of robots (one air, one ground).
The active perception policy is heuristic and not jointly optimized with downstream planning objectives.
The distilled model may inherit biases from the VLM-generated pseudo-labels used during training.