💬 NLP

HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

HERMES introduces a reusable, data-derived hierarchical labeling substrate that uses learned semantic transforms and residual vector quantization to annotate documents with multi-granularity codes, enabling the discovery of optimal data mixture strategies across varying resolutions that fixed-granularity methods cannot test.

Original authors: Ziyun Qiao, Yue Min, Ruining Chen, Yujun Li

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziyun Qiao, Yue Min, Ruining Chen, Yujun Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant robot (an AI) how to speak and think by feeding it a massive library of books, websites, and articles. The big question isn't just how much data you feed it, but which books you choose and in what order.

This paper introduces a new tool called HERMES to solve a specific problem: How do we organize this messy library so the robot learns the best possible mix of topics?

Here is the breakdown using simple analogies:

1. The Problem: The "One-Size-Fits-All" Label

Imagine you have a giant pile of mixed-up documents. To teach the robot, you need to sort them into groups.

Old Way: You might use a pre-made list of categories like "Science," "History," or "Cooking." But these are rigid. If you want to look at "Science" in more detail, you can't just zoom in; you have to throw away the old list and make a brand new one from scratch.
The Bottleneck: The paper argues that the problem isn't the robot's ability to learn (the "mixer"); the problem is the labeling system itself. It's too rigid.

2. The Solution: HERMES (The "Russian Nesting Doll" of Labels)

HERMES is a new way to label every single document in the library once, but in a way that lets you "zoom in" or "zoom out" as much as you want without ever re-sorting the books.

Think of HERMES like a set of Russian Nesting Dolls or a hierarchical address system:

Level 1 (The Big Box): Every document gets a broad label, like "Music" or "Cooking." There are about 256 of these big boxes.
Level 2 (The Middle Box): Inside "Music," you can zoom in to see sub-groups like "Rock," "Jazz," or "Hip-Hop." There are about 65,000 of these.
Level 3 (The Tiny Box): Inside "Hip-Hop," you can zoom in even further to specific artists or eras. There are about 130,000 of these tiny groups.

The Magic Trick: You don't need to run three different sorting machines. You run the sorting machine once. The "zoom level" is just a matter of how many digits of the address you read.

Read 1 digit? You get the broad "Music" category.
Read 3 digits? You get the specific "1990s Hip-Hop" category.

This saves massive amounts of computing power because you only have to label the library once, but you can experiment with different levels of detail instantly.

3. The Discovery: It's Not About the "Best" Group, It's About the "Right" Zoom

The researchers tested this on a 1-billion-parameter AI model. They found two surprising things:

Finding A: The "Goldilocks" Zone
They tried two different ways of picking books from the groups:

The "Cover All Bases" approach: Pick a few books from every single sub-group, no matter how small or low-quality.
The "Top Quality" approach: Only pick the top 30% of the best books within each sub-group.

The Result: At the middle zoom level (Level 2, the 65,000 groups), the "Top Quality" approach made the robot significantly smarter. It learned better because the groups were big enough that picking the "best" books actually made sense.

Finding B: The "Too Small" Trap
However, when they zoomed in to the finest level (Level 3, the 130,000 tiny groups), the "Top Quality" approach stopped working.

Why? The groups became so small that there were only a handful of books in them. Trying to pick the "best" book from a group of only 5 books is like trying to pick the best player from a team of 5; the difference is negligible, and you might accidentally pick a bad one just by chance.
The Lesson: You can't just keep zooming in forever. There is a "sweet spot" where the groups are detailed enough to be useful, but big enough to have a good selection of high-quality data.

4. The Conclusion: A New Way to Think About Data

The paper concludes that HERMES isn't just a better sorting algorithm (it actually performs about the same as standard sorting methods when you look at the big picture).

Its real value is that it turns data organization into a "dial" instead of a "switch."

Before: You had to choose between "Coarse Labels" or "Fine Labels" and stick with it.
Now: You can use one single labeling system and adjust the "granularity dial" to find the perfect balance for your specific AI training needs.

In short, HERMES gives researchers a flexible, reusable map of their data, allowing them to find the perfect "zoom level" where high-quality data selection actually improves the AI's brainpower.

Technical Summary: HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures

1. Problem Statement

Current data-mixing strategies for Large Language Model (LLM) pre-training face a bottleneck in the label system rather than the mixing algorithms themselves. Existing approaches typically partition corpora into groups using fixed taxonomies (provenance, topics, formats) or flat embedding clusters (e.g., KMeans). These methods suffer from three primary limitations:

Single Granularity/Axis: They commit to a single semantic axis at a fixed resolution. Changing the granularity (e.g., from broad topics to fine-grained clusters) requires rebuilding the entire label set.
Re-computation Cost: Flat clustering methods (like KMeans) require re-computation to change the number of clusters ( $K$ ), making systematic sweeps of granularity computationally expensive and disjointed.
Inflexibility: The choice of groups dictates what a mixer can express, limiting the ability to test interactions between data resolution and sampling strategies.

The authors argue that the bottleneck is the labeling infrastructure, not the mixer machinery. There is a need for a reusable, data-derived substrate that allows practitioners to navigate a hierarchy of granularities without re-clustering.

2. Methodology: HERMES

HERMES (Hierarchical Embedding-based RVQ for Mixture Selection) is a one-shot, hierarchical labeling substrate designed to annotate documents with coarse-to-fine codes. It consists of two main stages:

A. Learned Semantic Transform (LST)

To prepare embeddings for quantization, HERMES employs a Learned Semantic Transform.

Input: Frozen document embeddings ( $e_i \in \mathbb{R}^{1024}$ ) from a pre-trained encoder.
Mechanism: A single linear layer with normalization ( $h_i = \text{normalize}(W e_i + b)$ ) rotates the embedding space.
Training Objective: The transform is trained jointly with the quantization codebooks to optimize three losses:
1. Pairwise Structure Preservation: Maintains the cosine geometry between raw and transformed embeddings.
2. Quantization-Aware Reconstruction: Minimizes the reconstruction error of the vector quantizer.
3. Orthogonality: Prevents representation collapse by enforcing near-orthogonality of the transformation matrix $W$ .

B. 3-Stage Residual Vector Quantization (RVQ)

The transformed embeddings ( $h_i$ ) are encoded by a cascaded 3-stage RVQ.

Architecture: Three sequential vector quantizers, each with a codebook size $K=256$ .
Process:
1. Stage 1 (Coarse): Encodes $h_i$ to find the nearest centroid $c_1$ , producing a residual $r_2 = h_i - q_1$ .
2. Stage 2 (Intermediate): Encodes $r_2$ to find $c_2$ , producing residual $r_3$ .
3. Stage 3 (Fine): Encodes $r_3$ to find $c_3$ .
Output: Each document receives a hierarchical code $(c_1, c_2, c_3)$ .
Granularity Control: The granularity is defined by the prefix length of the code:
- L1: Code $(c_1)$ $\rightarrow$ 256 buckets.
- L12: Code $(c_1, c_2)$ $\rightarrow$ $\sim$ 65k buckets.
- L123: Code $(c_1, c_2, c_3)$ $\rightarrow$ $\sim$ 130k observed buckets (out of a nominal $1.67 \times 10^7$ ).

C. Sampling Taxonomy

The paper defines a sampler as a composition of two stages operating on the HERMES substrate:

Stage 1 (Outer Weight): Assigns weights to L1 ancestors. The paper uses Uniform (max-entropy) or DoReMi (group-DRO on a proxy model).
Stage 2 (Sub-bucket Sampler): Decides eligibility and selection within L1 ancestors.
- Mass Distribution: Equal (max-entropy) vs. Size-proportional.
- Per-Document Eligibility: All documents vs. Top-30% by quality score (FineWeb-Edu) within the sub-bucket.

3. Key Contributions

HERMES Substrate: A one-shot hierarchical RVQ system that annotates $\sim$ 50M documents into up to $\sim$ 130k cells. It exposes granularity as a prefix length on a single trained codebook stack, eliminating the need for re-clustering when changing resolution.
Granularity-Sampler Interaction: The authors demonstrate that the effectiveness of a sampling rule (e.g., quality-based selection) is dependent on the granularity level.
Candidate Competition Diagnostic: The paper identifies a structural limit where increasing granularity reduces the candidate pool size to a point where within-bucket ranking becomes unstable.

4. Experimental Results

Experiments were conducted on a 1B-parameter LLaMA-style model trained for 25B tokens, evaluated on a 16-task capability macro-average (Avg.).

A. The Clustering Plateau (L1=256)

Finding: At the coarsest granularity (L1=256), HERMES performs on par with standard clustering methods (KMeans, MiniBatchKMeans, Plain RVQ).
Metrics: On intrinsic metrics (compactness, mass-balance, entropy), HERMES sits on a plateau with other 256-way clusterers.
Downstream: When paired with a Uniform + L1 max-entropy sampler, HERMES and KMeans differ by only 0.0002 in Avg. accuracy.
Implication: The performance gain of HERMES is not due to being a superior clustering algorithm, but rather the substrate itself.

B. Finding A: Stage-2 Rule at Intermediate Granularity (L12)

Setup: Fixed DoReMi L1 outer weights, fixed HERMES codebook, granularity at L12 ( $\sim$ 65k buckets).
Comparison: Switching the Stage-2 sampler from max-entropy coverage (equal sub-bucket mass, all docs) to quality top-30% (size-proportional mass, top-30% by FineWeb-Edu score).
Result: This switch increased the 16-task Avg. by +0.0253 (z-score +2.09).
Interpretation: At L12, sub-buckets are large enough (median pool $\sim$ 2,271 documents) that quality ranking is a stable proxy for global quality.

C. Finding B: Collapse at Fine Granularity (L123)

Setup: Same rules applied at L123 ( $\sim$ 130k buckets).
Observation: The median candidate pool contracts 5.3x from L12 (2,271 docs) to L123 (429 docs).
Result: The advantage of the quality top-30% rule collapses. The performance gap between quality top-30% and local random coverage shrinks to +0.0002 (0.3988 vs. 0.3986), which is statistically indistinguishable under single-seed evaluation.
Implication: This supports the candidate competition hypothesis: when sub-buckets become too small, within-bucket ranking is no longer a reliable proxy for global quality, negating the benefit of quality-based selection.

D. Best Configuration

The highest-performing configuration in the sweep was DoReMi-L1 outer weights + L12 quality top-30% (corrected reader), achieving an Avg. of 0.4222.

5. Significance and Claims

The paper modestly claims that HERMES reframes data mixture design. Its primary significance lies in shifting the focus from "choosing among fixed label sets" to "navigating a reusable, data-derived granularity hierarchy."

Substrate over Clusterer: The contribution is the infrastructure that allows granularity to be a free variable (a dial), not the clustering performance itself.
Joint Determination: Granularity and the Stage-2 sampler are jointly determined axes. A rule that works at one granularity (L12) may fail at another (L123) due to structural constraints like candidate pool size.
Limitations: The authors acknowledge that conclusions are limited to a 1B/25B regime on an internal corpus. The optimal granularity depth is bounded by "candidate competition," which may shift with larger models and corpora. They also note that the specific outer-weight choice (DoReMi vs. Uniform) at the optimal Stage-2 rule (L12 quality top-30%) was not fully cross-validated (the Uniform + L12 quality top-30% run was not performed).

In summary, HERMES provides a mechanism to treat data granularity as a continuous, tunable parameter, revealing that the efficacy of data selection strategies is intrinsically linked to the resolution of the underlying label system.