Technical Summary: Revisiting Uncertainty on Evidential Learning for Partially Relevant Video Retrieval

Problem Statement

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos using text queries that describe only a specific segment of the video content. Unlike traditional Text-to-Video Retrieval (T2VR) which assumes fully descriptive queries for pre-trimmed videos, PRVR faces a fundamental challenge rooted in data asymmetry: brief, vague text queries are matched against rich, untrimmed video content.

This asymmetry introduces two compounded challenges:

1. Inter-video Semantic Ambiguity: Queries often exhibit fine-grained heterogeneity. They can be under-determined (insufficient semantics leading to low confidence) or polysemous (ambiguous semantics matching multiple candidate videos). Existing methods often ignore this heterogeneity or handle it implicitly, lacking mechanisms to explicitly quantify specific uncertainty types.
2. Intra-video Sparse Supervision: The prevalent Multiple Instance Learning (MIL) paradigm typically supervises only the single best-matching clip. This leads to sparse training signals, preventing the model from learning holistic representations and making it vulnerable to spurious local noise (e.g., false positives from irrelevant clips).

Methodology: The Holmes Framework

The authors propose Holmes, a hierarchical evidential learning framework designed to explicitly quantify and model uncertainty at both inter-video and intra-video levels.

1. Inter-video Evidential Learning

This component addresses query ambiguity by treating cross-modal similarity scores as evidence parameterized by a Dirichlet distribution.

• Three-Fold Principle for Query Identification: The framework categorizes queries based on three metrics derived from the Dirichlet distribution:
  • Epistemic Uncertainty ($u$): Reflects the strength of evidence. High $u$ indicates under-determined queries with limited evidence.
  • Label Consistency ($c$): Measures alignment with the ground truth. High $c$ does not guarantee precision, as polysemous queries may also align well with the ground truth.
  • Aleatoric Uncertainty ($\xi$): Measured by the expected entropy of the distribution. Low entropy indicates a precise query (probability mass concentrated on one video), while high entropy indicates a polysemous query (uniform distribution).
• Uncertainty Guided Identification (UGI): Queries are partitioned into three subsets: Under-determined ($S_u$), Precise ($S_p$), and Polysemous ($S_n$). This partitioning utilizes a Uncertainty-Dominant Partition Fusion strategy to reconcile discrepancies between frame-scale and clip-scale branches, prioritizing hypotheses with higher uncertainty dominance.
• Query Label Calibration: Based on the identified category, labels are recalibrated. Precise and under-determined queries retain original one-hot labels. Polysemous queries receive soft labels derived from model-estimated similarities to capture latent relevance and avoid over-penalizing semantically related samples.

2. Intra-video Evidential Learning

To resolve sparse supervision, Holmes accumulates denser evidence by establishing soft alignments between the query and multiple relevant clips.

• Flexible Optimal Transport (FOT): The method formulates a soft query-clip alignment using Optimal Transport (OT).
• Adaptive Dustbin: A key innovation is the introduction of a "dustbin" bucket (a constant column vector set to the bottom 30% percentile of similarity scores). This bucket absorbs irrelevant or noisy clips, preventing them from forcing a full mapping. This allows the model to learn from multiple relevant clips while suppressing spurious local responses, effectively providing denser temporal supervision.

3. Optimization

The model is trained using a two-stage scheme:

1. Warm-up: Optimizes standard similarity retrieval and inter-video evidential learning ($L_{inter}$) without label recalibration.
2. Joint Optimization: Activates intra-video evidential learning ($L_{intra}$) and label recalibration.
   The total loss combines the base retrieval loss, inter-video evidential loss, and intra-video evidential loss.

Key Contributions

1. Conceptual: The paper reframes PRVR through the lens of uncertainty, explicitly linking inter-video semantic ambiguity with intra-video sparse supervision.
2. Methodological:
   • Proposes Holmes, a dual-level evidential learning framework.
   • Introduces a three-fold principle for fine-grained query identification and adaptive label calibration.
   • Develops Flexible Optimal Transport (FOT) with an adaptive dustbin to enable dense and robust temporal alignment.
3. Empirical: Demonstrates significant improvements over state-of-the-art methods on three benchmarks, with ablation studies validating the efficacy of uncertainty quantification and noise suppression.

Experimental Results

Experiments were conducted on ActivityNet Captions, Charades-STA, and TVR.

• Performance: Holmes achieves state-of-the-art performance across all datasets. On ActivityNet Captions, it outperforms the strongest competitor (DreamPRVR) by 0.7 in SumR. On TVR, it surpasses MGAKD by 8.1 in SumR.
• Efficiency: Despite the added complexity of evidential learning, Holmes maintains comparable inference and training efficiency to similar-scale models (e.g., GMMFormerV2).
• Ablation Studies:
  • Removing multi-scale aggregation or single-branch learning degrades performance, confirming the necessity of hierarchical evidence.
  • Removing label calibration ($L_{inter}$) leads to the worst performance, highlighting the importance of handling query ambiguity.
  • Removing the intra-video evidential loss ($L_{intra}$) or replacing FOT with standard OT/softmax results in lower accuracy, demonstrating the effectiveness of the dustbin mechanism in filtering noise.
• Qualitative Analysis: Visualizations show that Holmes correctly ranks ground-truth videos above irrelevant ones and suppresses spurious local peaks, whereas models without $L_{intra}$ fail to distinguish relevant clips from noise.

Significance and Claims

The paper claims that Holmes offers a fresh perspective by moving beyond deterministic similarity scores to explicitly model prediction uncertainty.

• Interpretability: By distinguishing between confident matches and low-evidence guesses, the framework shifts retrieval from "black-box" predictions to interpretable, uncertainty-aware systems.
• Robustness: The intra-video evidential learning and adaptive dustbin mechanism actively mitigate "false positives" by filtering out spurious correlations and noise, which is critical for untrimmed video retrieval.
• Dataset Analysis: The ability to categorize polysemous and under-determined queries provides a pathway to diagnose dataset biases rather than blindly overfitting to them.

The authors note limitations, including reliance on pre-trained feature extractors (which may limit performance compared to end-to-end CLIP-based approaches) and the requirement for full video access, making it less suitable for online streaming scenarios. Future work is planned to incorporate advanced encoders and end-to-end training.