FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment

1. Problem Statement

Vision-Language Pretraining (VLP) aims to learn cross-modal representations from large-scale image-text pairs. A critical challenge in this domain is the presence of false negatives in contrastive learning objectives (Image-Text Contrastive - ITC, and Image-Text Matching - ITM).

• The Trade-off: Effective VLP requires hard negatives (semantically similar to the anchor but not the true match) to refine the embedding space. However, as the selection criteria for negatives become stricter (higher semantic similarity), the risk of misidentifying a true positive as a negative (a false negative) increases significantly.
• The Limitation of Existing Methods:
  • Fixed Heuristics: Methods that select negatives based on a fixed similarity threshold (e.g., always picking the top 10% most similar) fail because the "optimal" similarity range is anchor-specific and evolves during training. Complex anchors learn slowly and require easier negatives early on, while simple anchors can handle harder negatives later.
  • Pretrained Model Reliance: Recent approaches (e.g., MAFA) use strong pretrained models (like BLIP) to filter out false negatives by relabeling them as positives. However, these methods rely on static thresholds and the fixed capabilities of the pretrained model, which often struggle with complex or domain-specific scenarios, leading to either overly conservative filtering or missed false negatives.

2. Methodology: FALCON

The authors propose FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives without relying on external pretrained models or fixed heuristics.

Core Components

1. Negative Mining Scheduler ($\pi_\phi$):

   • Instead of a fixed rule, FALCON employs a learnable scheduler that predicts a hardness quantile value ($q \in [0, 1]$) for each anchor instance during mini-batch construction.
   • High $q$ (near 1.0): Selects very similar (hard) negatives.
   • Low $q$ (near 0.0): Selects dissimilar (easy) negatives.
   • The scheduler dynamically adjusts $q$ based on the current state of the embedding space and the specific anchor's difficulty.

2. Input Representation:

   • The scheduler does not take raw embeddings directly. Instead, it receives a normalized similarity distribution matrix ($\hat{S}$).
   • This matrix is constructed from the pairwise similarities (Image-to-Text and Text-to-Image) between the anchor and candidates within a local search space.
   • To ensure efficiency and permutation equivariance, the scheduler uses a lightweight 4-layer residual MLP that processes sorted rows of the similarity distribution.

3. Training Signal (The Proxy):

   • The scheduler is trained to maximize the reduction in Masked Language Modeling (MLM) loss ($\Delta L_{MLM}$) after a VLP model update using the constructed mini-batch.
   • Rationale: A significant drop in MLM loss indicates that the model successfully integrated visual context into language understanding, implying the mini-batch contained informative negatives that improved cross-modal alignment without introducing conflicting signals from false negatives.
   • The scheduler parameters ($\phi$) are updated via gradient ascent using the log-derivative trick (REINFORCE), treating the mini-batch selection as a policy.

4. Mini-Batch Construction Process:

   • Starting with a random anchor, the scheduler predicts a quantile $q$.
   • The next sample is selected from the remaining pool based on the $q$-th quantile of the similarity distribution relative to the current sample.
   • This recursive process builds a mini-batch where every anchor has a tailored negative hardness level.

3. Key Contributions

1. First Learning-Based Scheduling: FALCON is the first approach to explicitly learn and schedule the trade-off between hard and false negatives in contrastive learning, moving beyond static heuristics or external model filtering.
2. Novel Architecture: It introduces a lightweight, efficient negative mining scheduler that operates on similarity distributions, allowing for instance-specific adaptation with minimal computational overhead.
3. Comprehensive Validation: The method is validated across three distinct VLP frameworks (ALBEF, BLIP-2, SigLIP-2) and a wide range of downstream tasks (Retrieval, VQA, NLVR), demonstrating robustness and generalizability.

4. Experimental Results

The paper evaluates FALCON against strong baselines (ALBEF, GRIT-VLP, MAFA, DiHT, SRCL) on MSCOCO and large-scale web-crawled datasets (Conceptual Captions, DataComp).

• Performance Gains:
  • ALBEF: FALCON achieves state-of-the-art results on Image-Text Retrieval (R@1: 62.28 vs. 60.96 for MAFA), VQA, and NLVR2.
  • BLIP-2 & SigLIP-2: Significant improvements are observed when FALCON is integrated into these modern architectures, proving its compatibility with generative objectives (ITG loss).
  • DataComp: On the noisy DataComp benchmark, FALCON outperforms baselines by a large margin (e.g., +11.9% over MAFA on zero-shot retrieval), highlighting its ability to handle domain shifts where static pretrained filters fail.
• Ablation Studies:
  • Objective: Using $L_{MLM}$ as the proxy signal is crucial; using contrastive losses ($L_{ITC}$) as the proxy leads to performance degradation (likely due to exploiting trivial negatives).
  • Instance-Level vs. Batch-Level: Instance-level scheduling (different $q$ per anchor) significantly outperforms batch-level scheduling (single $q$ for all), confirming that optimal hardness is anchor-dependent.
  • Search Space Size: FALCON remains robust even with large search spaces, whereas baselines degrade due to increased false negative rates.
• Efficiency: While FALCON requires slightly more wall-clock time per epoch than GRIT-VLP due to the scheduler updates, it converges faster in terms of total time-to-convergence and achieves higher final performance.

5. Significance and Conclusion

FALCON addresses a fundamental bottleneck in Vision-Language Pretraining: the inability of static strategies to manage the dynamic tension between learning from hard examples and avoiding false negatives.

• Adaptability: By learning the scheduling policy online, FALCON adapts to the evolving structure of the embedding space and the specific semantic complexity of each anchor.
• Independence: Unlike methods relying on external "oracle" models (like BLIP-129M) to filter data, FALCON is self-contained, making it more generalizable to diverse datasets and domains where pretrained models may not be reliable.
• Future Impact: The paper suggests that such learning-based scheduling strategies are essential for the next generation of Large Vision-Language Models (LVLMs), as they provide a more robust foundation for cross-modal representation learning.

In summary, FALCON demonstrates that dynamically balancing negative hardness through a learned scheduler is a superior strategy to fixed heuristics or static filtering, leading to significantly improved cross-modal alignment and downstream task performance.