Can Vision-Language Models Answer Face to Face Questions in the Real-World?

This paper introduces the Qualcomm Interactive Video Dataset (IVD) to benchmark the ability of vision-language models to answer real-time questions about live video and audio, revealing that while current models significantly lag behind humans, targeted fine-tuning can substantially improve their performance on these interactive tasks.

The Gist

Imagine you have a robot friend named "VisionBot." You've trained VisionBot to look at a photo of a living room and tell you, "That's a red couch," or "There's a cat sleeping on the rug." VisionBot is great at this. It's like a very smart librarian who can instantly find facts in a static book.

But now, imagine you want VisionBot to be a real-time conversation partner. You are holding your phone, walking around your house, and you ask VisionBot, "Hey, am I holding this cup the right way?" or "How many times did I just clap?" while you are doing it.

This is where VisionBot starts to stumble. It's like asking a librarian to join you in a live dance party and keep up with the rhythm, but the librarian is used to reading books in a quiet library. They know the facts, but they don't know when to speak or how to react to things happening right now.

The Problem: The "Time Travel" Gap

The paper argues that current AI models are like time travelers who only know the future.

• Old AI: You give it the entire video of a person clapping 10 times, and then you ask, "How many times did they clap?" The AI watches the whole thing, counts, and answers. It's like watching a movie and then taking a quiz.
• Real Life: In the real world, you ask the question while the person is clapping. The AI has to listen to the question, watch the video unfold, realize, "Oh, they are still clapping, I need to wait," and then answer at the exact right moment.

Current AI models are terrible at this. They either answer too early (before the action is finished) or they get confused because they can't combine what they see (video) with what they hear (audio) in real-time.

The Solution: The "QIVD" Playground

To fix this, the researchers at Qualcomm created a new playground called QIVD (Qualcomm Interactive Video Dataset).

Think of QIVD as a gym for AI robots.

• The Workout: They recorded 2,900 short videos of real people doing random things (clapping, pointing, holding objects) and asking questions like, "Is this my nose or my eye?" or "Did I just throw the ball?"
• The Twist: The dataset includes a special "stopwatch" for every question. It tells the AI exactly when enough information has appeared to answer correctly.
  • Example: If someone asks, "How many times did I clap?" the stopwatch says, "Don't answer yet! Wait until the clapping stops."
• The Goal: To train AI to learn not just what to say, but when to say it.

What They Discovered

The researchers put the smartest AI models (like GPT-4o and others) through this gym, and the results were surprising:

1. The "Blind Spot": Even the smartest AIs failed miserably at "face-to-face" questions. They often answered before the action was done or missed the audio cues. It's like a student who knows the math formula but forgets to wait for the teacher to finish reading the question before shouting out the answer.
2. The "Audio Blindness": Many models ignored the sound. If you asked, "Am I speaking loudly?" and the model only looked at the video, it couldn't answer. They are like people trying to have a conversation in a noisy room while wearing noise-canceling headphones.
3. The "Magic of Fine-Tuning": Here is the good news. When they took these clumsy robots and gave them extra practice specifically on this "gym" data (fine-tuning), they got much better. They learned to wait, to listen, and to combine sight and sound. It's like giving a new driver a few hours of practice in a parking lot; suddenly, they aren't crashing into everything.

The Big Picture

This paper is a wake-up call. We are building AI that is amazing at describing static pictures, but we are far from having AI that can be a helpful, real-time companion in our daily lives.

The Analogy:

• Current AI is like a photographer who takes a perfect picture and writes a caption later.
• What we need is a cameraman who can walk alongside you, listen to your questions, watch what you're doing, and give you helpful advice while you are doing it.

The QIVD dataset is the first step toward teaching our AI cameramen how to stop, look, listen, and speak at the right time. It's the bridge between "smart computers" and "helpful robots."

Technical Summary

1. Problem Statement

Current Large Multimodal Models (LMMs) have achieved significant success in offline visual reasoning (e.g., describing static images or analyzing pre-recorded videos). However, they struggle with situated, real-time interaction. The paper identifies a critical gap: existing models are not designed to converse with users "face-to-face" where:

• Input is Streaming: The model receives audio and video simultaneously as events unfold, rather than processing the entire clip at once.
• Context is Dynamic: Questions often rely on future visual or auditory information (e.g., "How many times did I clap?" requires waiting for the action to complete).
• Timing is Crucial: Models must determine when to answer. Answering too early (before the action finishes) or too late breaks the conversational flow.
• Ambiguity is High: Real-world interactions involve deictic references (e.g., "this," "that," "here") and require integrating audio cues with visual context to resolve ambiguity.

Existing benchmarks are biased toward offline, static reasoning and lack the "when-to-answer" dimension required for real-world AI assistants and humanoid robots.

2. Methodology

A. The Qualcomm Interactive Video Dataset (QIVD)

The authors introduce QIVD, a novel dataset and benchmark designed to evaluate online situated audio-visual reasoning.

• Data Collection: 2,900 short video clips (avg. 5.1s) recorded by crowd workers using mobile/laptop cameras. Users perform actions and ask open-ended questions related to the scene in real-time.
• Annotations: Each video includes:
  1. Transcribed Question: Human-generated transcript of the spoken question.
  2. Ground Truth Answer: Human-generated answer.
  3. Optimal "When-to-Answer" Timestamp: A critical annotation marking the exact moment in the video when sufficient context exists to answer the question correctly. This often occurs after the question is spoken.
• Taxonomy: The dataset covers 13 semantic categories, including Action Counting, Object Referencing, Audio-Visual integration, and Subjective questions.
• Key Statistics: The optimal answer timestamp averages 81.47% of the video duration, indicating that models often need to observe the majority of the clip before responding.

B. Baseline Streaming Approach

To address the lack of streaming-capable models, the authors propose a pipeline:

1. Streaming ASR: Uses Streaming-Whisper to transcribe questions in real-time chunks (0.25s) and detect the end of the question.
2. Trigger Mechanism: The system identifies the "when-to-answer" moment. Since ASR only detects the end of speech, the system must infer if further visual context is needed.
3. LMM Inference: The video/audio up to the trigger point and the transcribed question are fed into an LMM backbone to generate an answer.

C. Experimental Setup

The authors evaluated a wide range of models:

• Zero-Shot Evaluation: Tested open-source models (VideoLLaMA, LLaVA-NeXT, Qwen-VL, etc.) and closed-source models (GPT-4o, Gemini-2.5-Flash) using both ASR-derived and Ground-Truth (GT) questions/timestamps.
• Fine-Tuning: Fine-tuned VideoLLaMA2.1-7B-AV on QIVD to test if data-driven adaptation can bridge the performance gap.
• Streaming Adaptation: Modified Qwen2.5-Omni to operate in a streaming mode, training it to emit a special token to signal "listening" vs. "answering."
• Evaluation Metrics: Used an LLM judge (Qwen3-8B) to determine answer correctness, alongside standard metrics (BLEU, ROUGE, METEOR).

3. Key Contributions

1. QIVD Dataset: The first benchmark specifically for online, situated, face-to-face video question answering with explicit "when-to-answer" annotations.
2. Benchmarking LMMs: Comprehensive evaluation revealing that even state-of-the-art models (GPT-4o, Gemini-2.5) perform significantly worse than humans on real-time situated tasks.
3. Fine-Tuning Efficacy: Demonstration that fine-tuning on streaming audio-visual data significantly improves performance, particularly in temporal reasoning and audio-visual integration.
4. Streaming Baseline: A practical pipeline combining streaming ASR with LMMs to handle real-time inputs, highlighting the gap between current offline models and the requirements for live interaction.

4. Key Results

Performance Gap

• Human vs. AI: Humans achieve ~87% correctness. The best zero-shot model (GPT-4o) achieves only ~58.76% (using GT timestamps).
• Static vs. Temporal: There is a massive performance drop for models on temporal tasks (e.g., Action Counting) compared to static tasks (e.g., Object Detection).
  • Action Counting: Humans (85.7%) vs. Best Model (33.5%).
  • Audio-Visual: Humans (100%) vs. Best Model (34.8%).
• Failure Modes: Common errors include misinterpreting deictic references (e.g., "this" vs. "that"), incorrect action counting, and temporal sequencing confusion.

Impact of Fine-Tuning

• Fine-tuning VideoLLaMA2.1-7B-AV on QIVD yielded significant gains:
  • Action Counting: +16.96% improvement.
  • Audio-Visual: +17.39% improvement.
  • Subjective: +23.26% improvement.
• However, even after fine-tuning, performance on Action Counting remained low (~30%), suggesting that current architectures lack the necessary inductive biases for complex temporal reasoning.

Impact of Audio and Timing

• Audio Integration: Contrary to expectations, adding audio to a pre-trained vision-only model degraded performance initially. However, end-to-end fine-tuning with audio and video modalities resulted in substantial gains across all categories, proving that multimodal integration creates emergent capabilities.
• When-to-Answer: Providing the model with the correct "when-to-answer" timestamp significantly improved performance. Models that answered prematurely (based solely on ASR end-of-speech) performed poorly, highlighting the need for models to reason about the future of the video stream.

5. Significance and Conclusion

This paper establishes that while LMMs are powerful for static analysis, they are currently ill-equipped for real-time, situated human-AI interaction. The primary bottlenecks are:

1. Temporal Reasoning: Inability to wait for sufficient context before answering.
2. Multimodal Integration: Failure to seamlessly fuse audio and visual streams to resolve ambiguity.
3. Deictic Resolution: Difficulty understanding context-dependent references in a live setting.

QIVD serves as a crucial stepping stone for developing the next generation of AI assistants and humanoid robots. The authors conclude that while fine-tuning on streaming data helps, future research must focus on architectural innovations that natively support online temporal reasoning and active listening capabilities.