💬 NLP

Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

This paper introduces Quantile-Guided Density Estimation (QGDE), a method that leverages stable token ID-ratio distributions in released BPE tokenizers to accurately estimate the composition ratios of hidden pretraining corpora at both the token and category levels.

Original authors: Qingjie Zhang, Xingzhang Ren, Zixuan Chen, Jinfeng Li, YueFeng Chen, Yitong Yang, Hui Xue, Dayiheng Liu, Han Qiu

Published 2026-08-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Qingjie Zhang, Xingzhang Ren, Zixuan Chen, Jinfeng Li, YueFeng Chen, Yitong Yang, Hui Xue, Dayiheng Liu, Han Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've just bought a brand-new, super-smart robot chef. It can write poetry, solve math problems, and even code video games. But there's a catch: the recipe book it used to learn how to cook is locked in a vault. You can see the robot's final dishes, and you can even see the list of ingredients it might have used, but you have no idea what the actual mix of ingredients was. Was it 90% pizza dough and 10% spices? Or maybe 50% chocolate and 50% broccoli? In the world of Artificial Intelligence, this "recipe book" is called a pretraining corpus, and the "list of ingredients" is the tokenizer vocabulary.

When companies release a new AI model, they often share the final weights (the robot's brain) and the vocabulary list (the ingredient dictionary), but they keep the exact recipe secret. This is a problem because knowing what the AI "ate" helps us understand why it's good at some things and bad at others. For years, scientists have tried to guess the recipe by looking at the robot's behavior or the rules it used to chop words into pieces. But these guesses were often too rough, like saying "it probably ate a lot of fruit" without knowing if it was apples or bananas. The big question was: Can we look at the ingredient dictionary alone and figure out the exact ratio of every single ingredient the robot consumed?

This paper says, "Yes, we can, and here's how." The researchers discovered that the way words are chopped up and numbered in these dictionaries isn't random; it follows a hidden, stable pattern, much like how the frequency of words in any language follows a predictable curve. They realized that if you know the pattern from a "known" recipe (a corpus we have access to), you can transfer that pattern to guess the recipe of a "hidden" one. They built a clever tool called QGDE (Quantile-Guided Density Estimation) to do this. Think of it like a detective who doesn't just look at one clue but uses a whole set of "what-if" scenarios (quantile trends) and weighs them based on how crowded the neighborhood of clues is (local density weighting).

The results are surprisingly precise. In their tests, QGDE could guess the ratio of specific words with a tiny error rate of just 3.00%. When they grouped those words into bigger categories like "Web," "Math," or "Code," the error was still only 3.08%. This is a huge improvement over previous methods, which were like guessing the weather by looking at a single cloud. The paper also tested this on a real-world, released AI model called SmolLM, where the actual recipe was known. Even there, QGDE outperformed other methods, proving that the ingredient dictionary really does hold the secret to the recipe. However, the authors are careful to note that while this works well in their simulations and on SmolLM, we still can't test it on giants like ChatGPT or Qwen because their full training recipes remain locked away. But for now, it suggests that the next time you see an AI's vocabulary list, you might just be looking at a map to its hidden diet.

Technical Summary: Can Released LLM Vocabularies Support Token-Level Estimation of Hidden Corpora?

1. Problem Formulation

The composition of pretraining corpora is fundamental to understanding Large Language Model (LLM) capabilities, yet this information often remains opaque even when model weights are released. While released tokenizer vocabularies (typically trained using Byte-Pair Encoding, or BPE) are often public, they are rarely accompanied by the exact training data statistics.

The core problem addressed is token-level corpus ratio estimation. Given a released target tokenizer vocabulary $V^*$ trained on a hidden corpus $C^*$ , the goal is to estimate the corpus ratio $r^*_i$ for every token $v_i$ in the vocabulary. Unlike prior work that infers coarse-grained category mixtures (e.g., language or domain proportions) or traces specific token groups, this paper aims to estimate the ratio for arbitrary individual tokens.

The authors assume access to one or more known corpora $D$ with observable token ratios. The challenge is to learn an estimator $\hat{f}$ that maps token IDs to ratios in the hidden corpus by leveraging the relationship between token IDs and ratios observed in the known corpora.

2. Key Observation: Transferability of ID-Ratio Distributions

Before proposing a solution, the authors investigate whether the relationship between token IDs and corpus ratios is transferable across different corpora.

Method: They trained BPE tokenizers on various languages (English, French, Japanese, Chinese) and domains (Web, Wikipedia, Code, Math) and plotted the distributions of token IDs versus their corpus ratios in log-log space.
Finding: Despite significant differences in language structure or domain content, the token ID–ratio distributions share a stable global shape.
Quantification: Using a directional transfer similarity score based on Kullback–Leibler (KL) divergence, they demonstrated that these distributions are broadly transferable. For instance, English and French show near-interchangeability, while even structurally distinct pairs like Code and Wikipedia retain substantial similarity.
Implication: This stability motivates transferring the distributional structure from known corpora to a target tokenizer trained on hidden data.

3. Methodology: Quantile-Guided Density Estimation (QGDE)

To exploit this transferability, the authors propose Quantile-Guided Density Estimation (QGDE). The method consists of three main stages:

A. Fitting ID-Ratio Trends with Quantiles

Instead of fitting a single median curve (as done in prior work like PoCTrace), QGDE models the shared ID–ratio relationship using a family of quantile trends.

Log-Log Transformation: Following Zipf's law, token IDs ( $t$ ) and ratios ( $r$ ) are modeled in log-log space, where the relationship approximates a linear trend.
Quantile Regression: For a set of quantile levels $\tau$ , the method fits log-linear curves $q_\tau(x) = a_\tau + b_\tau x$ . This captures not just the central tendency but the full vertical spread of plausible ratios for a given token ID.
Candidate Generation: For a target token with ID $t_i$ , each quantile trend provides a candidate log-ratio estimate $z_{i,\tau}$ .

B. Selecting Quantile Anchors

To avoid redundancy and ensure coverage of the distribution, the method selects a small set of representative quantile anchors $T^*_K$ .

Optimization: Anchors are selected by maximizing Quantile Anchor Coverage (QAC), which counts how many known ID–ratio points fall within a vertical bandwidth of the selected trends.
Result: This process favors anchors whose trends jointly pass through well-supported regions of the distribution rather than spreading uniformly or focusing on sparse areas. Empirical analysis shows that coverage saturates as the number of anchors ( $K$ ) increases (e.g., diminishing returns after $K=14$ ).

C. Local Density Weighting

The final step converts the multiple candidate estimates into a single point estimate using local density weighting.

Mechanism: For a target token ID, the method identifies nearby known ID–ratio points within a local window.
Weighting: It assigns soft weights to each candidate estimate based on how strongly the surrounding points support that specific quantile trend, using Gaussian kernel density estimation.
Final Estimate: The estimated ratio is the density-weighted average of the candidate estimates. This transforms the "range-style" signal of prior work into a fine-grained token-level point estimate.

4. Experimental Results

The authors evaluated QGDE in both controlled settings (using mC4, OSCAR, FineWeb, etc.) and a realistic setting (using the released SmolLM tokenizer).

Controlled Settings

Token-Level Estimation: QGDE significantly outperformed two baselines: direct ID-ratio transfer (copying source ratios by position) and PoCTrace (single median trend).
- Performance: QGDE achieved Mean Relative Errors (MRE) as low as 3.00% for token-level estimation in mixed-source settings.
- Ablation: Increasing the number of quantile anchors ( $K$ ) from 3 to 14 sharply reduced errors, particularly in domain settings, after which gains saturated.
- Source Mixtures: Using mixed known corpora generally yielded better results than single-source settings, though the exact ratio of the mixture mattered less than the diversity of components covered.

Aggregation to Category-Level

Mixture Estimation: Token-level estimates were aggregated to estimate language or domain proportions.
Comparison: QGDE substantially outperformed the Data Mixture Inference (DMI) baseline.
- Error Reduction: In language settings, MRE dropped from 9.09% (DMI) to 3.08% (QGDE). In domain settings, it dropped from ~15.14% to **5.3%**.
- Observation: While increasing $K$ improved token-level accuracy, the gains at the category level were less pronounced due to the smoothing effect of aggregation.

Realistic Setting (SmolLM)

Validation: The method was tested on the SmolLM tokenizer, where ground truth training data proportions were available (FineWeb-edu, Cosmopedia, Python-edu).
Results: QGDE achieved the lowest error for both token-level (5.72–5.78% MRE) and category-level (5.93% MRE) estimation, outperforming both direct transfer and PoCTrace. This confirmed the method's efficacy even when the exact component corpora could not be used as direct training sources.

5. Contributions and Significance

The paper makes three primary contributions:

Discovery of Transferability: It demonstrates that token ID–ratio distributions are transferable across BPE vocabularies trained on different languages and domains, providing a usable signal for estimating hidden corpus ratios.
Methodological Innovation: It introduces QGDE, a general token-level estimator that utilizes multiple quantile trends and local density weighting to approximate the transferable distribution, moving beyond coarse mixture inference or specific token tracing.
Empirical Validation: It shows that QGDE achieves high accuracy (as low as 3.00% MRE) in both controlled and realistic settings, surpassing existing baselines.

Significance:
The authors argue that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation. This capability goes beyond the coarse composition inference possible with current methods, offering a pathway to audit and interpret the data sources of released LLMs even when the training corpora remain hidden. The work suggests that the "fingerprint" of corpus composition is encoded in the tokenizer's vocabulary structure in a way that can be decoded with sufficient statistical modeling.

Limitations:
The authors note that the scarcity of ground truth for released LLMs (e.g., ChatGPT, Qwen, DeepSeek) limits direct evaluation on these models. Validation currently relies on controlled experiments and the rare SmolLM case where training data is public. Broader validation awaits the release of more training corpora.