💬 NLP

EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

The paper introduces EgoArgus, a human-annotated benchmark dataset designed to evaluate the ability of Vision-Language Models to act as reliable egocentric assistants by arbitrating between visual evidence and user dialogue in daily scenarios, revealing current limitations in modality trust assessment and the restricted effectiveness of existing bias mitigation methods.

Original authors: Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen

Published 2026-08-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a pair of glasses that can see what you see, hear what you say, and offer helpful advice in real time. This is the promise of a new generation of artificial intelligence known as vision-language models. These systems are designed to act as daily assistants, watching the world through a camera mounted on your head or chest while listening to your conversation. They are meant to understand your situation, predict what you might do next, and step in with a warning or a suggestion when necessary. But for these assistants to be truly reliable, they must solve a difficult puzzle: when the video they see and the words you speak tell different stories, which one should they trust?

Researchers at National Yang Ming Chiao Tung University in Taiwan have built a new test to see if current artificial intelligence can handle this conflict. They created a dataset called EgoArgus, which simulates five different types of daily interactions between a user and an assistant. In some scenarios, the video and the conversation agree, working together to paint a clear picture. In others, the conversation might be completely unrelated to what is happening on screen, or it might be politely misleading. The most challenging cases occur when the user says one thing while the camera clearly shows something else, such as a person claiming to have turned off a faucet while the water is still running. The researchers wanted to see if the AI could ignore the misleading words and rely on the visual evidence, or if it would get confused and follow the wrong path.

To test this, the team gathered thousands of examples from real-life videos of people doing daily tasks, like cooking, shopping, or driving. They paired these videos with dialogue that fit into their five scenarios. They also created synthetic episodes where an AI assistant had to decide whether to speak up or stay silent. The results were revealing. When the user's words contradicted the video, the artificial intelligence models failed dramatically. Instead of trusting the camera, which showed the truth, the models blindly followed the user's incorrect statements. In these contradictory situations, the models performed worse than if they had simply guessed at random. They would confirm the user's mistake rather than correcting it, or they would suggest actions that made no sense given what was actually happening on screen.

The study also examined whether the AI could decide when to intervene. A good assistant knows when to speak and when to stay quiet. The researchers found that even the strongest models struggled with this balance. Some models were too eager, offering warnings when nothing was wrong, while others missed genuine safety hazards entirely. When they did intervene, their timing was often off, either acting too early before there was enough evidence or too late after the problem had already occurred. The team tried several methods to fix these errors, such as forcing the models to pay more attention to the video or training them to prefer visual evidence over text. These attempts provided only limited help. The models remained stubbornly biased toward the text, suggesting that simply adjusting their attention is not enough to make them reliable.

Further investigation into how these models think revealed that the problem runs deep. The researchers looked inside the models to see how they processed the video and the words. They found that the models kept the two types of information mixed together for a long time. It was only in the final stages of their processing that the models began to separate the visual evidence from the spoken words. By that point, however, the decision had often already been swayed by the misleading dialogue. This suggests that the models do not have a robust mechanism to weigh conflicting evidence in real time. They can distinguish between a picture and a sentence, but they cannot always decide which one is the truth when the two disagree.

The findings indicate that while these artificial intelligence systems are becoming better at understanding the world, they are not yet ready to be trusted as independent daily assistants. They lack the critical judgment needed to navigate a noisy environment where human speech might be mistaken, irrelevant, or even deceptive. For these assistants to become truly useful, they need to learn not just to see and hear, but to know when to trust their eyes over their ears. Until they can do that, they remain prone to following the wrong lead, even when the answer is right in front of them.

Technical Summary: EgoArgus

Problem Statement

Recent advancements in Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) have shifted the paradigm from passive recognition to interactive assistants capable of perceiving environments and following natural language instructions. In egocentric (first-person) settings, these assistants must integrate real-time video streams with user dialogue to understand situations, predict next actions, and decide when to intervene.

However, a critical gap remains: existing benchmarks primarily evaluate visual understanding in isolation. They fail to assess whether models can effectively arbitrate between visual evidence and user-provided language when these modalities are helpful, irrelevant, or directly conflicting. In real-world deployment, an assistant receives two simultaneous streams—video and dialogue—that may complement, contradict, or ignore one another. Current models struggle to determine which modality is trustworthy, often leading to failures where they follow misleading user dialogue even when the visual scene provides the correct answer.

Methodology

The authors introduce EgoArgus, a human-annotated benchmark designed to evaluate egocentric assistants on understanding and decision-making tasks across five distinct dialogue-video scenarios. The dataset is constructed from two main components:

Understanding Part (MCQA): Comprising 6,978 examples derived from real egocentric daily-task videos (sourced from EgoPlan-Bench2, QaEgo4D, and MIntRec). These examples are paired with user dialogue generated to instantiate specific scenarios.
Decision Part (Intervention): Comprising 789 synthetic assistant episodes generated using VISTA. These episodes target intervention decisions, covering safety, non-safety, and no-assistance situations, which are difficult to collect densely in the real world.

Five Assistance Scenarios

EgoArgus categorizes interactions into five scenarios to test modality arbitration:

Multimodal Grounded: Dialogue provides context (e.g., intent), while video grounds the object or state. Both are required for a correct answer.
Contradictory: Video provides the correct judgment, but dialogue explicitly contradicts it (e.g., user claims a faucet is off, but video shows it running).
Video-Grounded On-Topic: Video determines the judgment; dialogue is topically related but uninformative (e.g., user discusses a child while video shows the child is left behind).
Video-Grounded Off-Topic: Video determines the judgment; dialogue is completely unrelated (e.g., user chats about being tired while video shows an open car window).
Text-Grounded: Dialogue determines the judgment; video is irrelevant (e.g., checking library hours).
No Assistance: A control case where no intervention is warranted.

Evaluation Protocol

The benchmark evaluates seven recent VLMs (including Molmo2, Qwen3.5, InternVL3.5, Gemini-3.1, etc.) on three progressive capabilities:

Context Understanding: Inferring the current situation.
Next Action Prediction: Anticipating the subsequent step in a task.
Intervention Decision: Deciding if, when, and how to intervene.

The study employs oracle diagnostics (removing distractor modalities), layer-wise linear probing to analyze modality separation in model representations, and experiments with modality bias mitigation techniques (uniform attention reweighting and NaPO preference alignment).

Key Contributions

EgoArgus Benchmark: A human-annotated resource evaluating egocentric assistants across MCQA and assist-step formats, covering five specific dialogue-video relations and no-assistance cases.
Scenario Definition: Formalization of five dialogue-video relations (multimodal grounded, contradictory, video-grounded on/off-topic, text-grounded) with specific labels for safety and intervention timing.
Empirical Analysis of Reliability Gaps: Demonstration that contradictory dialogue drives real-video accuracy below random guessing for most models, and that accurately identifying intervention timing remains a significant challenge.
Diagnostic Insights: Evidence that existing modality bias mitigation methods (such as attention reweighting and preference alignment) provide limited enhancement, suggesting that current VLMs lack adaptive mechanisms to infer which modality is answer-bearing versus distracting.

Results

Understanding Part

Systematic Failure on Contradiction: Mean accuracy drops from 63.0% on multimodal-grounded examples to 18.5% on contradictory examples. Most models perform below random guessing in the contradictory scenario, indicating that the "dialogue lure" actively reverses decisions rather than just distracting.
Asymmetric Text Dominance: Models achieve 71.0% accuracy on text-grounded tasks (recovering answers from dialogue even with irrelevant video) but struggle to recover answers from video when dialogue conflicts.
Distractor Sensitivity: Off-topic dialogue reduces accuracy but less severely than contradictory dialogue, suggesting models can ignore unrelated text better than conflicting text.

Decision Part

Model Variance: Stronger models make more reliable intervention decisions, but even top models struggle with "no-assistance" cases, often over-intervening in benign scenes.
Timing Errors: Models tend to intervene too early or too late relative to oracle timing, with errors large enough to affect short safety events.
Error Distribution: Failures are distributed across grounding errors (misunderstanding evidence), step errors (vague/inadequate advice), timing errors, missed interventions, and false alarms.

Mechanism Analysis

Modality Separation: Layer-wise probing reveals that visual and dialogue token representations become separated only in middle-to-deep decoder layers. However, separation does not guarantee correct arbitration; models may distinguish sources yet still follow misleading dialogue.
Mitigation Limitations:
- Uniform Attention Reweighting: Increasing visual attention via a scalar $\epsilon$ is brittle; it fails to solve contradictory dialogue issues and degrades performance on text-grounded and multimodal scenarios.
- Preference Alignment (NaPO): Training with noise-aware preference optimization slightly improves text-grounded accuracy but reduces performance on contradictory and multimodal scenarios, lowering overall accuracy.

Significance and Claims

The paper claims that EgoArgus exposes a fundamental reliability gap in current VLMs when deployed as daily assistants. The primary finding is that contradictory dialogue is a critical failure point, causing models to ignore visible reality in favor of user text.

The authors argue that reliable egocentric assistants require adaptive mechanisms capable of:

Inferring which modality is answer-bearing.
Identifying which modality is a distractor.
Determining when non-answer context (like on-topic but uninformative dialogue) should be preserved for situational awareness.

The study concludes that current mitigation strategies are insufficient, highlighting the need for new architectural or training approaches to handle modality arbitration in dynamic, first-person environments. The work serves as a diagnostic tool for practitioners deploying VLMs, cautioning against blind trust in user-provided text when it conflicts with visual evidence.