Technical Summary: Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection

Problem Definition

The paper addresses the challenge of egocentric mistake detection, a task distinct from standard action recognition. The goal is to determine whether a user performs an action incorrectly within instructional videos. This task is characterized by two primary difficulties:

1. Long-Tailed Distribution: Mistake samples are rare and often ambiguous compared to correct actions, leading to conventional models overfitting the dominant correct class.
2. Multi-Semantic Error Types: Mistakes occur at different levels. Some are local execution errors (the specific action is performed wrongly), while others are procedural errors (the action is executed correctly in isolation but is inappropriate for the current stage of the overall workflow).

Existing single-model approaches often struggle to efficiently capture both coarse-grained workflow context and fine-grained execution details simultaneously.

Methodology: UE-MCM

The authors propose the Understanding-Enhanced Model Collaboration Method (UE-MCM), a dual-branch architecture designed to balance speed, accuracy, and contextual reasoning.

1. Dual-Branch Architecture

The system consists of two complementary branches that process inputs differently and are fused adaptively:

• Large Model Branch (Fine-Grained Reasoning):

  • Input: Only the fine-grained action segment ($V^f_{t:t+\tau}$).
  • Backbone: Uses the Qwen3-VL Embedding model to extract high-capacity, semantically rich representations.
  • Function: Focuses on determining if the specific manipulation itself contains an execution mistake. The embedding model is frozen during training to preserve its semantic strength while optimizing the classifier.

• Small Model Branch (Workflow-Level Reasoning):

  • Input: Both the coarse-grained full video ($V^c_{0:T}$) and the fine-grained segment ($V^f_{t:t+\tau}$).
  • Backbone: Built upon a CLIP4CLIP video encoder, which is initialized from a CLIP model enhanced by Diffusion Contrastive Reconstruction (DCR).
  • Function: Performs efficient coarse-grained video understanding. By jointly observing the workflow and the specific segment, it identifies actions that are locally correct but inconsistent with the overall procedure.
  • Enhancement: The DCR technique improves the visual encoder by introducing reconstruction-guided contrastive signals, ensuring the representation preserves both class-discriminative semantics and detail-aware perceptual information.

2. Adaptive Collaboration Gate

The predictions from both branches are not simply averaged but are fused via a lightweight collaboration gate.

• The gate takes projected features from both branches ($\bar{h}_s, \bar{h}_l$) and their element-wise product.
• It outputs normalized weights ($\alpha_s, \alpha_l$) via a softmax function.
• The final logits are computed as a weighted sum: $z = \alpha_s z_s + \alpha_l z_l$.
• This design allows the system to dynamically balance workflow-level consistency reasoning against action-level correctness judgment for each input.

3. Long-Tail Optimization

To address the severe class imbalance between correct actions and rare mistakes, the classifiers are optimized using a composite objective function combining three complementary losses:

• Reweighted Cross-Entropy ($L_{WCE}$): Increases the penalty for underrepresented mistake samples based on class frequency.
• AUC-Oriented Loss ($L_{AUC}$): Improves ranking quality by encouraging mistake samples to receive higher scores than correct samples.
• Label-Aware Adjustment ($L_{LA}$): Calibrates the decision boundary using empirical class priors.

The final objective is $L = L_{WCE} + \lambda_{AUC}L_{AUC} + \lambda_{LA}L_{LA}$.

Key Contributions

1. Dual-Branch Collaboration: The introduction of UE-MCM, which separates fine-grained execution reasoning (Large Branch) from workflow-level consistency reasoning (Small Branch), addressing the multi-level nature of mistakes.
2. Representation Enhancement: The application of Diffusion Contrastive Reconstruction (DCR) to enhance CLIP-based encoders for better preservation of visual details and procedural context.
3. Long-Tail Strategy: The integration of reweighted cross-entropy, AUC-oriented learning, and label-aware adjustment to specifically target the recall and ranking of rare mistake instances.

Experimental Results

The system was evaluated on the HoloAssist 2026 Mistake Detection Challenge test set.

• Performance: The proposed method achieved an F-score of 0.60 using only the RGB modality.
• Comparisons:
  • It significantly outperformed the TimeSformer baseline (F-score 0.35) and the 2024 top-performing solution (F-score 0.51).
  • It improved upon the 2025 top-performing solution (F-score 0.57), particularly in Correct Recall (0.97 vs 0.97, with improved precision) and Mistake Recall (0.62 vs 0.63, with improved precision).
• Efficiency: The method attained competitive performance using only RGB inputs, matching or surpassing models that relied on multimodal inputs (e.g., RGB + Hands + Eyes).

Significance and Claims

The paper claims that UE-MCM effectively balances speed and accuracy, making it suitable for detecting subtle, rare, and ambiguous mistakes in egocentric instructional videos. By explicitly modeling the distinction between local execution errors and procedural inconsistencies, and by optimizing for long-tailed distributions, the system provides a robust solution for real-world mistake detection where data imbalance is a critical constraint. The authors emphasize that their approach successfully integrates action-level reasoning, workflow-level consistency, representation enhancement, and long-tail optimization into a unified framework.