Technical Summary: GuideMe

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities in offline video understanding, their ability to function as real-time procedural coaches remains unexplored. A true procedural coach must operate in a closed-loop interaction: continuously monitoring a user's execution, detecting deviations or errors in real-time, and providing corrective guidance to steer the user back on track. Existing datasets and benchmarks primarily focus on offline annotations, step-level instructions, or post-hoc error labels. None support the evaluation of a complete, active coaching cycle under realistic streaming constraints where the assistant must proactively decide when to intervene, what to advise, and whether the user has erred, all without access to future information.

Methodology

1. The GuideMe Benchmark

The authors introduce GuideMe, the first multi-domain streaming benchmark designed to evaluate closed-loop corrective capabilities in procedural tasks.

• Scale and Scope: The dataset comprises 2,458 videos (223.7 hours) spanning four diverse domains: cooking, object manipulation, daily-life guidance, and fitness.
• Interaction Samples: It contains 47,775 interaction samples capturing four distinct assistant behaviors:
  1. Issuing next-step instructions.
  2. Providing completion feedback for correct execution.
  3. Detecting execution errors.
  4. Offering corrective guidance to fix mistakes.
• Annotation Pipeline: A scalable, three-stage automated pipeline was developed:
  • Instructional Activity Extraction: Raw step-level annotations are extracted, merged, and categorized into correct, wrong, or correction actions based on source task graphs.
  • Knowledge Generation: LLMs generate task descriptions and canonical step sequences from the extracted activities, using parallel sampling and consensus refinement to ensure logical consistency.
  • Conversation Generation: Natural dialogues are generated aligned with action boundaries, creating pre-action instructions and post-action feedback (either confirmation or correction).

2. Inference Protocol

The evaluation simulates a strict causal setting where the model observes a continuous video stream via a sliding window (default 60 seconds) and full dialogue history. The model must proactively decide at each timestamp whether to remain silent or generate a response.

• Sampling Strategy: The authors employ Anchor-Based sampling, querying the model at ground-truth intervention timestamps and an equal number of silent intervals to avoid the class imbalance inherent in dense sampling.

3. Evaluation Framework

A three-component framework is proposed to assess performance beyond simple text generation:

1. Temporal-Semantic Bipartite Matching: Measures sequence-level alignment by solving a minimum weight bipartite matching problem between generated and reference responses. It calculates Soft-F1 (sPrecision, sRecall) using semantic embeddings and temporal distance penalties, allowing for semantic equivalence rather than exact string matching.
2. Behavioral Classification: Evaluates instance-level intervention timing. Outcomes are categorized as:
   • Correct Silent (CS): Correctly remaining silent.
   • False Alarm (FA): Speaking when silence is required.
   • No Response (NR): Failing to speak when intervention is required.
   • Partly Correct (PC): Speaking at the right time.
     These are aggregated into a Score metric that rewards correct silence and high-quality interventions while penalizing false alarms and missed responses.
3. LLM-as-a-Judge: Assesses the content quality of matched prediction-reference pairs on a scale of 0–5 (rescaled to 0–100) based on relevance, factual plausibility, and causal reasoning.

Key Results

Extensive experiments were conducted on proprietary models (e.g., Gemini 3.1 Pro, GPT-5.2, Doubao), open-source MLLMs (e.g., Qwen3-VL series), and specialized streaming models (e.g., VideoLLM-online, Dispider).

• Performance Asymmetry: A critical finding is the asymmetry in capabilities. Models generally perform reasonably on basic instruction delivery (providing next steps) but fail significantly at error detection and corrective guidance.
• Failure Modes: Models tend to polarize into two failure modes:
  • Aggressive: Intervene frequently but generate many false alarms (e.g., VideoLLM-online).
  • Conservative: Remain silent too often, missing necessary interventions (e.g., Dispider, MMDuet2).
• Quantitative Gaps: Even the best-performing models struggle with the closed loop. For instance, Gemini 3 Pro achieves a high instruction sF1 (42.0) but drops significantly on error detection (33.8) and correction (34.3). The highest overall Score (45.7) was achieved by Qwen3.5-397B-A17B, largely due to a conservative strategy that minimizes false alarms, yet it still misses a significant portion of required interventions.
• Fine-Tuning Impact: Fine-tuning on the GuideMe training set improved temporal alignment (sF1 increased from 35.7 to 39.7 for Qwen3-VL-8B) but shifted behavior toward being more conservative, increasing missed interventions (NR) while reducing false alarms.
• Context Sensitivity: Removing ground-truth dialogue history and relying on the model's own predictions leads to a sharp degradation in performance, indicating that error accumulation in the dialogue history severely impacts the model's ability to coach effectively.

Significance and Contributions

The paper claims three primary contributions:

1. GuideMe Benchmark: The first multi-domain streaming benchmark that unifies diverse procedural tasks into a closed-loop evaluation framework, covering the full instruct-observe-correct cycle.
2. Annotation and Evaluation Pipeline: A scalable three-stage annotation pipeline for transforming heterogeneous datasets into streaming interaction samples, paired with a rigorous three-component evaluation framework (Temporal-Semantic Matching, Behavioral Classification, and LLM-as-a-Judge) tailored for proactive guidance.
3. Identification of a Capability Gap: The study reveals a fundamental limitation in current MLLMs: while they can describe what should happen (procedural knowledge), they are currently unable to effectively coach based on what is actually happening (real-time error detection and correction).

The authors conclude that GuideMe serves as a practical testbed for developing AI assistants that move beyond passive narration toward genuinely interactive procedural coaching, highlighting that the ability to decide when to speak and how to correct is the defining challenge for future research.