Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

Technical Summary: Frequency Is What You Need: Considering Word Frequency When Text Masking Benefits Vision-Language Model Pre-training

Problem Statement
Vision-Language Models (VLMs), such as CLIP, typically require massive computational resources (thousands of GPU-days) for pre-training on large-scale image-text datasets. To accelerate this process, recent works have employed masking strategies for image and text tokens. While image masking is well-established, text masking strategies have yielded mixed results. Previous state-of-the-art work (CLIPA) identified "syntax masking"—which prioritizes retaining nouns and adjectives based on Part-of-Speech (POS) tagging—as the superior text masking approach. However, this paper argues that the evaluation of these strategies was limited by training for a fixed, relatively small number of epochs. The authors observe that as training epochs increase, syntax masking performance degrades relative to other strategies, suggesting that the current understanding of optimal text masking is incomplete.

Methodology
The authors propose a systematic analysis of how different text masking strategies impact the distribution of word frequencies in the training data. They define text masking as the selection of a subset of tokens from a text $T$ to form a new input $T'$. The study compares four existing strategies:

1. Truncation: Retains the first $k$ tokens.
2. Random: Randomly selects $k$ tokens.
3. Block: Selects a contiguous block of $k$ tokens starting at a random index.
4. Syntax: Uses POS tagging (via tools like NLTK) to prioritize retaining nouns, followed by adjectives, verbs, and other words.

Through this analysis, the authors discover that strategies like truncation and syntax masking disproportionately alter the relative frequencies of words in the training set. For instance, syntax masking heavily retains nouns while suppressing function words ("Other" category), and truncation over-represents sentence-initial words. In contrast, random and block masking preserve the original word frequency distribution more faithfully.

Motivated by the finding that preserving word frequency correlates with better model performance, the authors introduce CLIPF (Contrastive Language-Image Pre-training with Word Frequency Masking). CLIPF calculates a masking probability score $P(w_i)$ for each word $w_i$ based on its frequency $f(w_i)$ in the training data:
$$P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}}$$
where $t$ is a threshold parameter. Unlike previous frequency-based attempts (e.g., SW-CLIP) that used a hard cutoff to discard words (potentially leaving input slots empty), CLIPF uses these probabilities to prioritize which words to retain, filling the input tokens with the highest-probability words. This approach avoids the computational overhead of POS tagging required by syntax masking.

Key Contributions

1. Analysis of Frequency Impact: The paper demonstrates that the success of a text masking strategy is linked to how much it distorts the word frequency distribution of the training data. Strategies that disproportionately impact frequency (truncation, syntax) underperform those that preserve it (random, block).
2. Introduction of CLIPF: The authors propose CLIPF, a frequency-based masking strategy that directly leverages word frequency statistics without requiring POS tagging.
3. Re-evaluation of Training Epochs: The study highlights that the number of training epochs is a critical variable. While syntax masking appeared superior in short training runs (as reported in CLIPA), it falls behind other strategies when training is extended to convergence (e.g., 30+ epochs).
4. Comprehensive Evaluation: The authors provide extensive quantitative comparisons across multiple datasets (CC3M, CC12M, LAION-400M) and tasks (zero-shot classification, robustness, image-text retrieval).

Results

• Performance vs. Epochs: Experiments on ImageNet-1K show that while syntax masking performs well in early training, its accuracy plateaus or declines relative to random, block, and CLIPF as epochs increase (up to 40 epochs). CLIPF consistently achieves the highest accuracy in the 16–40 epoch range.
• Token Efficiency: CLIPF outperforms syntax masking and other baselines, particularly when the number of input text tokens is reduced (e.g., 4 to 8 tokens). In these aggressive masking scenarios, CLIPF maintains a significant performance gap over syntax masking.
• Robustness and Retrieval: CLIPF demonstrates superior zero-shot robustness on datasets like ImageNet-A, ImageNet-R, and ObjectNet. In image-text retrieval tasks, CLIPF generally outperforms truncation and syntax masking, though it shows comparable performance to random and block masking in some retrieval metrics, with a distinct advantage in high-masking-ratio scenarios (4 tokens).
• Specific Dataset Behavior: On datasets where class names are not predominantly nouns (e.g., Describable Textures Dataset, OxfordPets), syntax masking performs poorly, whereas CLIPF maintains strong performance.
• Computational Efficiency: CLIPF avoids the time-consuming POS tagging required by syntax masking. Experiments show syntax masking takes approximately 2.5 times longer to process the CC12M dataset than CLIPF.

Significance and Claims
The paper claims that the current state-of-the-art finding (that syntax masking is best) is contingent on limited training duration. By extending training, the authors show that syntax masking is not the optimal approach. The significance of this work lies in:

• Correcting the State-of-the-Art: Updating the understanding that syntax masking is the best text masking strategy, showing instead that frequency-preserving strategies are superior.
• Practical Guidance: Providing a clear guideline for practitioners: text masking strategies that disproportionately impact word frequency should be avoided, while those that protect relative frequencies (like CLIPF, random, and block) are preferred.
• Resource Efficiency: CLIPF offers a high-performance alternative that requires less computational overhead (no POS tagging) and achieves better results with fewer training resources (fewer tokens and epochs) compared to unmasked baselines or syntax-based approaches.

The authors conclude that considering word frequency is essential when selecting a text masking strategy, and that CLIPF provides a robust, efficient, and high-performing solution for VLM pre-training.