💻 computer science

Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

This paper introduces DynamiCS, a dynamic cluster-based sampling method that balances semantic data distribution by downsampling large clusters and upsampling small ones at each epoch, thereby reducing computational costs while significantly improving vision-language model performance on long-tail concepts.

Original authors: Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingliang Liang, Zhuoran Liu, Arjen P. de Vries, Martha Larson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world by showing it millions of pictures and their descriptions. This is how "Vision-Language Models" (VLMs) learn. However, the internet is a messy place. If you just grab a random pile of photos, you'll get thousands of pictures of "dogs" and "cars" (the popular stuff), but only a handful of pictures of "sloths" or "rare beetles" (the rare stuff).

If you train your robot on this messy pile, it becomes an expert at recognizing dogs but terrible at recognizing sloths. It's like a student who only studies the most popular chapters of a textbook and fails the exam because the test asks about the obscure ones.

This paper introduces a new method called DynamiCS to fix this problem while also saving a massive amount of money and computer power. Here is how it works, using some simple analogies:

1. The Problem: The "Fat Head" and the "Long Tail"

Think of the data the robot learns from as a crowd of people.

The Fat Head: A huge group of people wearing the same popular T-shirt (e.g., "dogs").
The Long Tail: A few scattered individuals wearing unique, rare outfits (e.g., "sloths").

Previous methods tried to fix this by simply cutting off the "Fat Head." They told the robot, "Ignore the popular dogs; let's only look at the rare stuff so everyone gets equal attention." The problem? This threw away too much useful information about the popular stuff, and the robot still struggled with the rare stuff because it didn't get enough practice.

2. The Solution: The "Smart Librarian" (Cluster Scaling)

DynamiCS acts like a very smart librarian who organizes books into "clusters" (groups of similar topics).

The Old Way: The librarian would just take an equal number of books from every shelf, regardless of how big the shelf was.
The DynamiCS Way: The librarian looks at the shelves and says:
- "This 'Dog' shelf is huge. I'll take a smaller, representative sample from it so we don't waste time reading the same thing over and over."
- "This 'Sloth' shelf is tiny. I'm going to copy the few books we have and show them to the robot more often!"

This is called "Cluster Scaling." It doesn't try to make every topic exactly equal (which is wasteful). Instead, it tries to make the robot useful by ensuring it sees the rare topics enough to learn them, without ignoring the popular ones entirely.

3. The Secret Sauce: "Dynamic Sampling" (The Shuffle)

Here is the second clever trick. Imagine you are studying for a test.

Static Sampling: You pick a specific set of flashcards and study only those cards for the whole week. You memorize them, but you miss out on the rest of the deck.
Dynamic Sampling (DynamiCS): Every single day, you shuffle the deck and pick a different random set of cards to study. Even if you are studying the rare "Sloth" cards, you might see a different picture of a sloth today than you did yesterday.

By shuffling the data every time the robot trains (every "epoch"), DynamiCS ensures the robot sees a wider variety of examples. This makes the "copying" of rare data (upsampling) actually work well, because the robot isn't just memorizing the same few rare images repeatedly; it's seeing different variations of them.

4. The Results: Faster, Cheaper, and Smarter

The paper shows that this approach is a win-win:

Cheaper: The robot learns much faster. The authors claim DynamiCS can achieve results similar to massive, expensive training runs using only about 3% of the computer power (GPU hours).
Better at the Rare Stuff: Because they intentionally "upsample" the rare concepts, the robot becomes much better at recognizing long-tail items (like the "Let It Wag!" test set in the paper) compared to other methods that just try to cut costs.
Still Good at the Popular Stuff: It doesn't lose its ability to recognize common things like dogs or cars; in fact, it often gets better at those too because it's learning more efficiently.

Summary

Think of DynamiCS as a smart study guide for AI. Instead of forcing the AI to read every single page of a massive encyclopedia (which takes forever and costs a fortune), it creates a custom curriculum. It skims the popular chapters quickly but spends extra time reviewing the rare, difficult chapters, and it shuffles the pages every day to keep the learning fresh. The result is a smarter AI that learns the same amount of knowledge in a fraction of the time and cost.

Technical Summary: Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-Training

Problem Statement

Training Vision-Language Models (VLMs) like CLIP requires massive computational resources due to the use of extremely large-scale datasets (e.g., LAION-400M, DataComp). While existing cost-reduction strategies focus on reducing the amount of information per sample (e.g., token reduction, low-resolution training) or filtering data to achieve semantic balance, they often fail to address the "long-tail" problem effectively.

Natural web-scraped datasets exhibit a "fat head and long tail" distribution, where a few semantic concepts dominate while rare concepts are underrepresented. Previous approaches to semantic balancing, such as MetaCLIP and Density-Based Pruning (DBP), often adopt an "aim for even" philosophy. This involves aggressively chopping off the fat head to flatten the distribution, which inadvertently removes or disproportionately reduces rare concepts. Consequently, long-tail concepts remain insufficiently represented, limiting the model's ability to generalize to rare categories. Furthermore, many dual-purpose approaches (combining cost reduction with quality improvement) rely on pre-trained VLMs as filters, adding complexity and dependency.

Methodology: DynamiCS

The authors propose DynamiCS (Dynamic Cluster-based Sampling), a dual-purpose approach that reduces training costs while explicitly enhancing performance on long-tail concepts. The method consists of two core components:

1. Cluster Scaling ("Aim for Utility")

Instead of flattening the semantic distribution, DynamiCS adopts an "aim for utility" philosophy. It acknowledges that while the fat head should be downsampled to reduce cost, it should not be completely eliminated. Conversely, the long tail should be upsampled to ensure representation.

The resampling size $S_i$ for a cluster $i$ is calculated as:
$S_i = \frac{c_i^\alpha}{\sum_{j=1}^N c_j^\alpha} \cdot T$
Where:

$c_i$ is the original size of cluster $i$ .
$N$ is the total number of clusters.
$T$ is the target total number of samples.
$\alpha$ $α$ is a scaling factor ( $0 \le \alpha$ $0 \leq α$ ).
- $\alpha = 0$ : Represents "aim for even" (all clusters equal size).
- $\alpha = 1$ : Represents random sampling.
- $\alpha > 1$ : Favors large clusters (aggressive pruning of long tail).
- Optimal $\alpha$ : The authors find that a moderate $\alpha$ (e.g., 0.2) balances the fat head and long tail, preserving the relative order of cluster sizes while emphasizing the tail.

Clusters are generated using K-means on image embeddings (from DINOv2-ViT-B/16), with centroids merged if their cosine similarity exceeds a threshold to reduce redundancy.

2. Dynamic Sampling

To make upsampling practical without increasing training cost and to improve cross-epoch data diversity, DynamiCS employs dynamic sampling. In each training epoch, a different subset of samples is drawn from each cluster based on the calculated scaling factor $P_i = S_i / c_i$ .

For large clusters ( $S_i < c_i$ ), samples are downsampled.
For small clusters ( $S_i > c_i$ ), samples are upsampled (drawn with replacement).
This ensures that over the course of training, the model sees a diverse set of samples from all clusters, including the rare ones, without requiring a larger total dataset size.

Key Contributions

Reframing Data Sampling: The paper argues that effective data sampling for VLMs must involve both downsampling (for cost) and upsampling (for long-tail performance), moving away from the "aim for even" philosophy toward an "aim for utility" approach.
Dynamic Sampling Mechanism: The authors demonstrate that dynamic sampling complements cluster scaling by improving cross-epoch diversity and making upsampling feasible within a fixed computational budget.
DynamiCS Framework: They propose a complete pipeline that outperforms existing cost-reduction methods and is competitive with full-scale pre-training, achieving these results with approximately 3% of the training costs of full-scale OpenCLIP.

Experimental Results

Experiments were conducted on LAION-400M and DataComp datasets using ViT-B/16 and ViT-L/16 encoders. The models were evaluated on ImageNet-1K (standard), Let It Wag! (long-tail), and various robustness and retrieval benchmarks.

Performance vs. Cost: DynamiCS significantly outperforms cost-saving baselines (RECLIP, FLIP, CLIPA) and dual-purpose approaches (DataComp, DFN, HQ-CLIP) on both standard and long-tail benchmarks.
- On ImageNet-1K, DynamiCS achieves accuracy competitive with full-scale OpenCLIP while using only ~60% of the GPU hours.
- On the Let It Wag! long-tail benchmark, DynamiCS shows substantial improvements (e.g., +10.2% over RECLIP on LAION-400M).
Scaling: The method scales effectively from 0.64B to 2.56B samples seen. DynamiCS-2.56B on DataComp-DFN outperforms full-training baselines (OpenAI-WIT, MetaCLIP, OpenCLIP) on ImageNet-1K and Let It Wag! while using only ~3% of their training costs.
Robustness: DynamiCS demonstrates strong robustness across six robustness datasets (IN-A, IN-O, IN-R, etc.), often outperforming full-scale training models with significantly fewer resources.
Fine-Grained Recognition: The method shows clear advantages on fine-grained tasks like CUB-200-2011 and Flowers102, attributed to better long-tail representation.
Hyperparameter Sensitivity: The method is robust to the choice of $\alpha$ , with $\alpha=0.2$ yielding optimal results.

Significance and Claims

The paper claims that DynamiCS offers a practical solution to the high computational cost of VLM pre-training without sacrificing, and often improving, model performance on rare concepts. By shifting the paradigm from "flattening" distributions to "utility-based" scaling and leveraging dynamic sampling, the authors demonstrate that it is possible to train highly effective VLMs with a fraction of the resources required by current state-of-the-art methods. The approach is presented as a generalizable strategy for imbalanced datasets, potentially extending beyond the specific datasets tested. The authors note that while retrieval performance on DataComp did not surpass LAION-400M (likely due to caption quality differences), the method remains effective for classification and robustness tasks.