🤖 AI

Confidence-Aware Tool Orchestration for Robust Video Understanding

The paper introduces Robust-TO, an agentic video understanding framework that mitigates the "Blind Trust Problem" by integrating per-frame reliability scores into a unified tool orchestration system, thereby significantly improving accuracy and robustness against visual corruptions compared to state-of-the-art models.

Original authors: Yangfan He, Yujin Choi, Jaehong Yoon

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yangfan He, Yujin Choi, Jaehong Yoon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a crime based on a security camera video. But here's the catch: the video is messy. Some parts are blurry because the camera shook, some are washed out by bright sunlight, and some are blocked by a truck passing in front of the lens.

The Problem: "Blind Trust"
Most current AI video detectives suffer from what the authors call the "Blind Trust Problem." They look at every single frame of the video and assume, "This picture is perfect, and I can trust it completely." They don't realize that a blurry frame is actually a bad clue. Because they trust the bad clues just as much as the good ones, they often get the wrong answer, yet they remain 100% confident in their mistake. It's like a detective trying to read a license plate through a dirty windshield and insisting, "I see it clearly!"

The Solution: Robust-TO
The paper introduces Robust-TO, a new way for AI to watch videos. Instead of blindly trusting every frame, Robust-TO acts like a smart, skeptical editor who knows how to handle a messy video. It works in three main steps:

1. The "Quality Inspector" (Frame Selection)

Before trying to solve the mystery, Robust-TO scans the video and rates every single frame.

The Analogy: Imagine a film editor looking at a reel of footage. They mark the frames that are blurry, too dark, or blocked by an object as "Trash." They only keep the "Gold" frames—the ones that are clear and actually show what the question is asking about.
The Result: If the question is "What car ran the red light?", the AI ignores the frames where a truck is blocking the view or the windshield wipers are blurring the scene. It only uses the clear moments.

2. The "Specialist Team" (Confidence-Guided Tool Routing)

Once the AI has the good frames, it doesn't just guess. It breaks the big question into small, specific tasks and sends them to different "tools" (specialized AI programs).

The Analogy: Think of a medical team. If a patient has a broken leg, you send them to an orthopedist, not an eye doctor.
How it works: If the video is blurry, Robust-TO knows that a "detection tool" (which looks for sharp edges) might fail. So, it routes the task to a "captioning tool" (which is better at guessing what's happening in a blur) instead.
The Confidence Score: Every tool gives an answer and a confidence score. But here's the trick: the score is adjusted based on how dirty the video was. If a tool gives an answer based on a blurry frame, its confidence score is automatically lowered, like a warning label saying, "Take this with a grain of salt."

3. The "Chief Detective" (Tiered Evidence Synthesis)

Finally, the main AI gathers all the answers from the tools. It sorts them into three piles:

High Tier: Clear evidence from clear frames. This is the "truth."
Medium Tier: Okay evidence. It's only used if it matches the High Tier.
Low Tier: Garbage evidence from bad frames. This is thrown away unless there is absolutely nothing else to go on.
The Outcome: The AI builds its final answer using only the "High Tier" facts. If the evidence is shaky, it admits uncertainty rather than guessing wildly.

Why It Matters (The Results)

The authors tested this system on real-world videos that had been intentionally messed up with blur, glare, and occlusion (like a car driving in the rain).

The Old Way: When the video got messy, standard AI models crashed. Their accuracy dropped by 15–30%, but they didn't know they were failing.
The Robust-TO Way: Even with the messy videos, Robust-TO kept its accuracy high. It actually performed better than some of the most expensive, famous AI models (like Gemini-2.5-Pro) on these tough tests.
Efficiency: Because it ignores the bad frames, it doesn't have to process as much data. It solved the problems faster and used less computer power, yet got the right answer more often.

In a Nutshell
Robust-TO teaches AI to stop being a "blind optimist" and start being a "critical thinker." It learns to say, "I can't trust this part of the video, so I'll ignore it and focus on the clear parts," ensuring that when it gives an answer, it's actually based on solid evidence.

Technical Summary: Robust-TO for Confidence-Aware Tool Orchestration

1. Problem Definition: The Blind Trust Problem

The paper identifies a critical flaw in contemporary Video Large Language Models (Video-LLMs): the Blind Trust Problem. Current models implicitly assume that every input frame is equally reliable and informative. In real-world scenarios involving perturbations such as motion blur, glare, low-light noise, or occlusion, this assumption leads to significant performance degradation. Frontier models can suffer accuracy drops of 15–30 percentage points on embodied benchmarks under such conditions, yet their self-reported confidence remains largely unchanged. This "silent failure" is particularly dangerous in safety-critical domains like forensic analysis, surveillance, and autonomous driving, where the model proceeds with high confidence despite degraded visual evidence.

Existing approaches typically rely on uniform or query-similarity-based frame sampling without evaluating the intrinsic quality of the visual signal. Consequently, corrupted frames are treated as valid evidence, distorting the final prediction.

2. Methodology: Robust-TO Framework

To address this, the authors propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. The system operates through a three-stage pipeline guided by a host Video Language Model (VLM) trained with Group Relative Policy Optimization (GRPO).

A. Frame Selection via Quality Profiling

Before reasoning, the system assesses the quality of the video stream $\tilde{V}$ .

Quality Profiling: A parameter-free tool (assess_quality) analyzes each frame to estimate a disturbance profile $d(f_i)$ based on three dominant corruption types: blur, brightness deviation, and occlusion. These are computed via signal-level statistics (e.g., inverse Laplacian variance for blur, Sobel-magnitude for occlusion) without requiring training data.
Reliability-Relevance Scoring: Frames are scored using a composite metric $s(f_i) = (1 - d(f_i)) \cdot \text{sim}(\phi(f_i), \psi(q))$ , balancing visual reliability against query relevance.
Selection: The top- $K$ trustworthy frames are selected for downstream processing, filtering out corrupted frames that might otherwise be selected due to high query similarity.

B. Confidence-Guided Tool Routing

The host VLM decomposes the original query into atomic sub-queries and routes them to specialized perception tools.

Adaptive Routing: The routing decision considers the dominant corruption type in the selected frames. For instance, if blur is predominant, the system favors tools robust to spatial degradation (e.g., captioning) over those requiring sharp boundaries (e.g., object detection).
Calibrated Confidence Output: Each tool returns a tuple $(r_j, c_j)$ $(r_{j}, c_{j})$ , where $r_j$ $r_{j}$ is the result and $c_j$ $c_{j}$ is a calibrated confidence score.
- $c_j = c_j^{\text{intrinsic}} \times \rho(F_j)$
- $c_j^{\text{intrinsic}}$ is the tool's self-assessment (e.g., detection score).
- $\rho(F_j)$ is the input reliability, calculated as the mean reliability of the worst-performing subset of frames used by that tool call. This prevents a few clean frames from masking the presence of severely corrupted inputs.

C. Reliability-Aware Evidence Synthesis

The final answer is synthesized by grouping evidence into three tiers: High, Medium, and Low.

High-tier: Drives the primary conclusion.
Medium-tier: Retained only if consistent with the high-tier conclusion; otherwise discarded.
Low-tier: Used only as a fallback when no stronger evidence exists, with explicit uncertainty reporting.
This tiered approach ensures that the final prediction is grounded in the most reliable visual evidence available.

D. Training Objective: Confidence-Cost GRPO

The host VLM is trained end-to-end using GRPO with a composite reward function $R_{\text{total}}$ that balances four signals:

Correctness ( $R_{\text{acc}}$ ): Accuracy of the final answer.
Confidence-Cost ( $R_{\text{cc}}$ ): Encourages high-confidence outputs while penalizing expensive tool calls on degraded frames ( $c_j - \lambda \cdot \text{cost}(T_j)$ ).
Sub-Query Efficiency ( $R_{\text{subq}}$ ): Penalizes both over-decomposition (wasteful calls) and under-decomposition (information gaps), anchored to a target count estimated by a frozen VLM.
Format Reward ( $R_{\text{fmt}}$ ): Ensures adherence to structured output formats.

3. Key Contributions

Formalization of the Blind Trust Problem: The paper explicitly defines and quantifies the disconnect between model confidence and visual evidence reliability under real-world corruptions.
Robust-TO Framework: A novel agentic pipeline that couples frame-level quality profiling with adaptive tool routing and tiered evidence synthesis.
Unified Confidence Interface: A mechanism that outputs calibrated confidence scores by multiplying tool certainty with input frame reliability, allowing the reasoning model to weigh evidence dynamically.
Confidence-Cost Optimization: A GRPO training strategy that jointly optimizes for accuracy, evidence reliability, and inference efficiency, preventing the model from over-relying on expensive tools for corrupted data.

4. Experimental Results

The framework was evaluated on UrbanVideo-Bench and VSI-Bench, covering eight tasks (e.g., landmark position, route planning, action generation) under both clean and corrupted conditions (generated via RoVA).

Clean Data Performance: Robust-TO with a Qwen3-VL-7B backbone achieved 56.4% average accuracy, surpassing the strongest open-source baseline (Video-R1) by 10.6% and outperforming the proprietary Gemini-2.5-Pro (46.2%).
Corrupted Data Performance: Under five realistic corruption types (Motion Blur, Gaussian Noise, Glare, Occlusion, Low-Light), Robust-TO maintained 54.3% average accuracy. This is a 5.8% improvement over the strongest open-source baseline (Video-R1) and a 16.2% improvement over Gemini-2.5-Pro.
Robustness: Robust-TO exhibited the smallest accuracy drop from clean to corrupted conditions among all compared methods.
Efficiency: The adaptive key-frame selector reduced the average number of processed frames by 35% (from 32 to 20.7) and cut per-sample inference time by over 35%, while simultaneously improving accuracy.

5. Significance and Claims

The paper claims that Robust-TO fundamentally shifts video reasoning from a "blind trust" paradigm to a "confidence-aware" paradigm. By explicitly modeling the reliability of visual evidence, the system achieves graceful degradation rather than silent failure when faced with real-world video corruptions.

The authors emphasize that their approach does not merely scale parameters or data but introduces a structural change in how evidence is acquired and synthesized. The results demonstrate that adaptive visual tool use, guided by per-frame reliability estimates, benefits both corrupted and clean videos, offering a pathway toward more robust and trustworthy multimodal reasoning systems for safety-critical applications. The work is released with code and checkpoints to support future extensions in open-world video reasoning.