💻 computer science

Adam's Law: Textual Frequency Law on Large Language Models

This paper introduces "Adam's Law," a framework proposing that prioritizing frequent textual data through paraphrasing, distillation, and curriculum-based fine-tuning significantly enhances the performance of Large Language Models across various tasks.

Original authors: Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, Zhao Hong, Qiqi Xiang, Bowen Cao, Wai Lam

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, Zhao Hong, Qiqi Xiang, Bowen Cao, Wai Lam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but slightly eccentric student (the Large Language Model, or LLM) how to solve problems, translate languages, or write stories. You have a massive library of books to choose from for your lessons.

This paper, titled "Adam's Law," proposes a simple but powerful rule for choosing those books: Always pick the sentences that sound the most "familiar" and "common," even if they mean exactly the same thing as a more complex, rare sentence.

Here is the breakdown of their discovery, explained through everyday analogies:

1. The Core Idea: The "Familiarity" Rule

Imagine you are asking a friend for directions.

Option A (High Frequency): "Go straight, then turn left at the big red bank."
Option B (Low Frequency): "Proceed in a linear trajectory, then execute a leftward rotation at the prominent crimson financial institution."

Both sentences mean the exact same thing. However, your friend (the AI) has heard Option A a million times in their life. They have heard Option B maybe once or twice.

The Paper's Discovery: When you give the AI Option A, it understands it instantly and gives a better answer. When you give it Option B, it gets slightly confused or stumbles, even though the meaning is identical.

The authors call this the Textual Frequency Law (TFL): If you want the AI to work better, feed it the most common, everyday way of saying things, not the fancy, rare way.

2. The Three-Step Toolkit

The authors didn't just say "use common words." They built a three-step system to make this happen:

Step 1: The "Popularity Meter" (Textual Frequency Law)

Since we can't peek inside the AI's brain to see exactly what books it read during training (because many are secret), the authors created a "Popularity Meter."

The Analogy: Imagine you want to know which songs are hits. You can't listen to every radio station in the world, so you check a public chart (online data) to see what's trending.
The Action: They scan the internet to estimate how often a specific sentence appears. If a sentence is rare, they know the AI might struggle with it.

Step 2: The "Rewrite Machine" (Textual Frequency Distillation)

Sometimes, the "Popularity Meter" isn't perfect because the AI might have read secret books we don't know about. So, they built a "Rewrite Machine."

The Analogy: Imagine you have a mystery novel, but you don't know the ending. You ask the AI to finish the story. If the AI finishes it easily, it means the story style is familiar to it. If it struggles, the style is weird to it.
The Action: They ask the AI to complete stories based on their data. If the AI generates a lot of text easily, it confirms that the input style is "high frequency" (familiar). This helps them fine-tune their "Popularity Meter" to be even more accurate.

Step 3: The "School Curriculum" (Curriculum Textual Frequency Training)

Now that they know which sentences are familiar and which are rare, they change how they teach the AI.

The Analogy: Think of learning to play the piano. You don't start with a difficult concerto (rare, complex music). You start with "Twinkle, Twinkle, Little Star" (common, simple music). Once you master that, you move to harder pieces.
The Action: Instead of throwing random sentences at the AI during training, they teach it in order: Start with the most common, familiar sentences first. Once the AI is comfortable, they slowly introduce slightly less common sentences. This is like a "gym routine" for the AI's brain, building strength gradually.

3. The Results: Does it Work?

The authors tested this on four different "sports" for AI:

Math: Solving word problems.
Translation: Turning English into other languages.
Common Sense: Answering questions like "Is a tomato a fruit or a vegetable?"
Tool Use: Asking the AI to use a calculator or search engine.

The Verdict: In almost every case, the AI performed significantly better when fed the "High Frequency" (common) sentences.

Translation: It made fewer mistakes and sounded more natural.
Math: It solved more problems correctly.
Reasoning: It thought more clearly.

Why Does This Happen?

Think of the AI as a giant pattern-matching machine. It has read billions of sentences. When it sees a sentence that looks like the billions it has already seen, it feels "at home." It knows the path forward.

When it sees a sentence with rare, fancy words (low frequency), it feels like it's walking on a path it's never seen before. It gets nervous, overthinks, and makes mistakes.

The Takeaway

This paper teaches us that simplicity and familiarity are superpowers for AI.

If you want an AI to do a great job, don't try to sound like a poet or a professor. Speak like a normal person using common words. If you are training an AI, teach it with the most common examples first. By respecting the "Textual Frequency Law," we can make these powerful machines smarter, faster, and more reliable.

1. Problem Statement

While textual frequency is well-established as a factor in human cognitive processing (e.g., reading speed and neural activation), its specific impact on Large Language Models (LLMs) remains underexplored. Previous research has focused on data quality, quantity, and training order (curriculum learning from easy to hard), but has largely overlooked the frequency of textual expressions when the semantic meaning is identical.

The core problem addressed is: When paraphrasing an input to maintain the same meaning, does the frequency of the specific linguistic expression affect the LLM's performance?

Hypothesis: High-frequency textual expressions (those appearing more often in general corpora) are easier for LLMs to process and predict than low-frequency expressions, even if the meaning is identical.
Challenge: Many LLMs are closed-source, making their exact training data inaccessible for direct frequency calculation. Additionally, simply paraphrasing can introduce semantic drift, requiring careful validation.

2. Methodology

The authors propose a comprehensive framework consisting of three main units: Textual Frequency Law (TFL), Textual Frequency Distillation (TFD), and Curriculum Textual Frequency Training (CTFT).

A. Textual Frequency Law (TFL)

Core Principle: For both prompting and fine-tuning, when multiple inputs share the same meaning, the one with the highest sentence-level frequency should be preferred.
Frequency Estimation: Since training data is often inaccessible, the authors estimate sentence-level frequency ($sfreq$) using open-source online corpora.
- They calculate sentence frequency as the geometric mean of word-level frequencies ($wfreq$) derived from resources like Zipf's law datasets.
- Formula: $sfreq(x, D) = (\prod_{k=1}^{K} wfreq(x_k, D))^{1/K}$ .

B. Textual Frequency Distillation (TFD)

Motivation: Online corpora may not perfectly reflect the specific distribution of a target LLM's training data.
Mechanism: To refine the frequency estimation, the framework queries the target LLM itself to perform "story completion" on the dataset.
Process:
1. The LLM generates continuations for the input sentences.
2. These generated continuations form a distilled dataset ( $D'$ ).
3. A new frequency estimation ( $F_2$ ) is calculated based on $D'$ .
4. The final frequency score ( $F$ ) is a weighted combination of the online estimate ( $F_1$ ) and the distilled estimate ( $F_2$ ), with a mechanism to boost scores for words that were ignored in the initial estimation.

C. Curriculum Textual Frequency Training (CTFT)

Concept: Extending the frequency law to the fine-tuning process.
Strategy: Instead of random shuffling or standard "easy-to-hard" ordering, the training data is sorted by sentence-level frequency.
Execution: The model is fine-tuned in an increasing order of frequency (starting from lower-frequency sentences and moving to higher-frequency ones). This acts as a curriculum where the model first learns diverse/complex expressions before reinforcing common patterns, or conversely, as the paper suggests, leveraging the natural ordering of frequency to optimize convergence.

D. Dataset Construction: Textual Frequency Paired Dataset (TFPD)

To validate these claims, the authors constructed a new dataset, TFPD, containing paraphrased pairs:

Sources: GSM8K (Math Reasoning), FLORES-200 (Machine Translation), and CommonsenseQA.
Generation: GPT-4o-mini generated 20 paraphrases per sentence (10 high-frequency, 10 low-frequency).
Validation: Three human linguists verified that the high-frequency, low-frequency, and original sentences retained identical meanings.
Result: A curated set of pairs where meaning is constant, but textual frequency varies significantly.

3. Key Contributions

Proposal of Textual Frequency Law (TFL): The first formal proposal suggesting that high-frequency paraphrases yield better performance in LLMs for both prompting and fine-tuning.
Textual Frequency Distillation (TFD): A novel method to estimate sentence-level frequency for closed-source models by leveraging the model's own generation capabilities to refine frequency metrics.
Curriculum Textual Frequency Training (CTFT): A new fine-tuning paradigm that orders training data by sentence frequency, demonstrating superior results compared to random or difficulty-based ordering.
TFPD Dataset: A high-quality, human-validated dataset specifically designed to isolate the variable of textual frequency from semantic meaning.

4. Experimental Results

The framework was evaluated on four tasks: Math Reasoning, Machine Translation (100+ languages), Commonsense Reasoning, and Agentic Tool Calling.

Prompting Performance:
- Math Reasoning: Using high-frequency prompts improved accuracy across all tested models (DeepSeek-V3, GPT-4o-mini, LLaMA-3.3-70B). For example, LLaMA-3.3-70B accuracy rose from 80.49% to 88.75%.
- Machine Translation: High-frequency inputs consistently outperformed low-frequency inputs across BLEU, chrF, and COMET metrics. On DeepSeek-V3, 99/100 language pairs showed BLEU improvements.
- Commonsense Reasoning: High-frequency partitions achieved higher accuracy (0.6974 vs. 0.6747 on GPT-4o-mini).
Fine-Tuning Performance:
- CTFT Efficacy: Fine-tuning with CTFT (sorted by frequency) yielded the best results. For example, on the pag_Latn language pair, BLEU scores improved from 3.7781 (standard HF fine-tuning) to 4.9102 (CTFT), a ~30% relative improvement.
- Comparison: High-frequency fine-tuning data outperformed even the original ground-truth data from the source datasets (FLORES-200).
Ablation Studies:
- Removing TFD caused performance drops, validating the need for model-specific frequency estimation.
- Correlation analysis showed a strong positive correlation between textual frequency and final translation performance, while showing weak correlation with traditional complexity metrics (e.g., dependency tree depth), proving TFL is a distinct and valuable signal.

5. Significance and Theoretical Foundation

Theoretical Proof: The paper includes a formal proof (Appendix) demonstrating that under Zipf's Law and standard cross-entropy training assumptions, higher-frequency sentences incur lower Negative Log-Likelihood (NLL) loss. This provides a mathematical basis for why LLMs "prefer" high-frequency inputs.
Practical Impact:
- Prompt Engineering: Offers a concrete strategy for prompt optimization (rephrasing to common terms) without changing the task logic.
- Data Efficiency: Suggests that training resources can be optimized by prioritizing high-frequency expressions or ordering training by frequency, potentially reducing the need for massive data volumes.
- Closed-Source Adaptation: The TFD method allows researchers to adapt frequency-based strategies even when they cannot access the proprietary training data of major LLMs.

In conclusion, "Adam's Law" establishes that linguistic frequency is a critical, yet overlooked, factor in LLM performance, providing a new axis for optimizing both inference (prompting) and training (fine-tuning) strategies.