💬 NLP

Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decoder

This paper demonstrates that in a controlled, attention-only decoder setting, training on just 30% of the dataset yields approximately 90% of the full-data performance, offering practical guidance for balancing dataset size and computational costs in resource-constrained environments.

Original authors: Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Götz-Henrik Wiegand, Lorena Raichle, Rico Städeli, Tomas Hrycej, Bernhard Bermeitinger, Siegfried Handschuh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but tiny, robot how to write like a human. You have a massive library of books (the data) and a limited amount of electricity to run your robot (the compute cost).

The big question in the world of AI right now is: "Do we need to feed the robot every single book in the library to make it smart, or is there a point where reading more books just wastes electricity?"

This paper, presented at a major AI conference in 2026, says: "You probably don't need all the books."

Here is the breakdown of their experiment and findings, explained with some everyday analogies.

1. The Experiment: Building a "Tiny" Robot

Most AI research uses massive, expensive supercomputers to train giant models. This paper wanted to see if the rules of "bigger is better" still apply when you are working with a tiny, budget-friendly model.

To do this, they built a simplified robot:

The "Brain" (Self-Attention): They kept the part of the robot that learns how words relate to each other.
The "Memory" (Embeddings): They gave the robot a pre-loaded dictionary so it didn't have to waste time learning what a "cat" or "dog" means from scratch.
The "Muscles" (MLP): They removed the heavy lifting muscles (the complex math layers) to keep the robot small and fast.

The Analogy: Imagine you are training a chess player. Instead of letting them build their own chess pieces and learn the rules of the board from scratch, you give them a pre-made board and pieces (frozen embeddings) and remove their ability to build furniture (removing MLPs). You only let them practice strategy (self-attention). This ensures that any improvement they make is purely because they are learning the game, not because they are building a better board.

2. The Method: The "Power-of-Two" Ladder

They didn't just throw random amounts of data at the robot. They created a ladder of training sets, doubling the size each time:

Run 1: 128 articles.
Run 2: 256 articles.
Run 3: 512 articles.
...all the way up to 131,000 articles.

They wanted to see: At what point does reading another book stop making the robot smarter?

3. The Big Discovery: The "Diminishing Returns" Curve

The results were very clear and followed a pattern known as "Scaling Laws," but with a twist for small models.

The Analogy of the Coffee Cup:
Imagine you are filling a coffee cup with hot water (data).

The First Pour: The cup goes from empty to half-full. The coffee is now drinkable! (Huge improvement).
The Second Pour: The cup is 90% full. It's almost perfect.
The Third Pour: You are just trying to fill the last 10% to the brim. You have to pour a lot of water to get that tiny bit of extra fullness.

The Finding:
The researchers found that their tiny robot reached 90% of its maximum possible intelligence after reading only 30% of the total data.

To get that final 10% of perfection, they had to use 70% more data and spend 70% more electricity.
It's like paying $100 to get a 90% grade, but then paying another $300 just to squeeze out a 95% grade. For most people, the first $100 is the best deal.

4. Why This Matters: The "Budget" Reality

In the real world, small research labs and startups don't have the money to train models on the entire internet. They have to be smart about their budget.

Old Thinking: "We need more data and bigger computers to get better AI."
New Insight: "If we stop training early and use a smaller, representative chunk of data, we get 90% of the results for a fraction of the cost."

The paper proves that for small, specialized models, quality and timing matter more than sheer volume. You don't need to read the whole encyclopedia to learn the basics; you just need the right chapters.

5. The "Robustness" Check

They also worried: "What if the 30% of data we picked just happened to be really easy or really hard?"
They ran the experiment multiple times with different random chunks of data. The results were the same every time. This means the "30% rule" isn't a lucky accident; it's a reliable pattern.

The Takeaway

If you are building a small AI model in a small lab:

Don't panic about getting every single data point.
Stop training earlier than you think.
Save your money. You can get a very capable model by using a moderate amount of data, rather than burning your budget trying to reach the theoretical "perfect" 100%.

In short: More data is good, but at a certain point, it's just expensive overkill.

1. Problem Statement

Training Large Language Models (LLMs) typically follows scaling laws where performance improves with increased model size, dataset size, and computational budget. However, most existing scaling law studies (e.g., Kaplan et al., 2020; Hoffmann et al., 2022) operate at an industrial scale where these factors are scaled jointly, making it difficult to isolate the specific contribution of dataset size.

Furthermore, it remains unclear whether the "diminishing returns" observed in massive models persist in tiny, resource-constrained models (e.g., small research labs or exploratory development). In these settings, training on the full dataset is often computationally prohibitive. The core research question is: Does increasing dataset size yield the same predictable, diminishing returns in a tiny, controlled architecture, and at what point does additional data cease to be cost-effective?

2. Methodology

The authors designed a highly controlled experimental setup to isolate the effects of dataset size from architectural complexity and representation learning.

A. Architecture: Attention-Only Tiny Decoder

To minimize confounding variables, the authors constructed a reduced-capacity Transformer decoder based on the 117M GPT-2 design but with strict constraints:

Frozen Embeddings & Output: The pretrained token embeddings and the output projection layer (weight-tied to embeddings) are frozen. This prevents the model from relearning token representations, ensuring that performance gains are driven by the self-attention mechanism rather than representation learning.
MLP Removal: All Multi-Layer Perceptron (MLP) sublayers are removed. The model relies solely on the self-attention mechanism for non-linearity and learning.
Parameter Count: The resulting model has ~79.5M total parameters, but only ~2.36M are trainable (the self-attention layers).
Rationale: This design isolates the learning dynamics to the self-attention mechanism, allowing for a clean analysis of how dataset size affects learning without the noise of full-scale model training.

B. Dataset and Subset Construction

Corpus: The experiments used the All the News 2.0 dataset (2.7M news articles), which was not part of the original GPT-2 pretraining corpus to avoid data leakage.
Nested Subsets: The authors created a sequence of nested training subsets ( $D_k$ ) ranging from $2^7$ to $2^{17}$ sequences (approx. 128 to 131,072 sequences).
Control: Subsets were constructed using a fixed random seed permutation, ensuring that larger subsets strictly extend smaller ones without introducing sampling variability.
Preprocessing: Articles were tokenized with the GPT-2 tokenizer (context length 1024). A Minimum Article Length (MAL) of 500 tokens was applied to reduce padding ratios to ~28%.

C. Experimental Regimes

The study evaluated performance under two distinct regimes:

Fixed Training Schedule: All models trained for 150 epochs. As dataset size increased, the total compute (optimization steps) increased linearly.
Fixed Compute Budget: All models trained for the same total number of optimization steps, isolating the effect of data volume from compute volume.

3. Key Contributions

Isolated Scaling Analysis: A novel architecture that freezes embeddings and removes MLPs to strictly isolate the self-attention mechanism's response to dataset scaling.
Dataset Representativeness Validation: Demonstrated via Jensen-Shannon divergence that moderate-sized subsets (e.g., $2^{12}$ sequences) are statistically indistinguishable from the full dataset in terms of token distribution, validating the use of subsets for scaling studies.
Cost-Performance Characterization: Quantified the trade-off between dataset size and training cost, identifying the "sweet spot" where diminishing returns set in.
Regime Identification: Showed that in tiny models, performance is primarily data-limited rather than model-limited, supporting the use of subset-based pretraining.

4. Key Results

A. Diminishing Returns

Smooth Improvement: Validation token-level accuracy improved smoothly as dataset size increased, following a power-law trend consistent with industrial-scale scaling laws.
Rapid Saturation: The most significant gains occurred early.
- Using only ~30% of the training data (Subset level $2^{15}$ ) was sufficient to achieve ~90% of the full-data validation accuracy.
- Using ~56% of the data achieved ~95% of the accuracy.
Cost Efficiency: To reach 90% of the full-data performance, the training time required was only 26% of the full-dataset training time.

B. Fixed Compute vs. Fixed Schedule

Under a fixed compute budget (same number of optimization steps), performance initially improved with larger datasets but quickly saturated. Larger datasets did not yield consistent improvements without a corresponding increase in training steps, confirming that data volume alone is insufficient without adequate compute.

C. Scaling Law Analysis

The authors compared their empirical results against the Kaplan et al. (2020) scaling law formula.
Data-Limited Regime: The predicted loss for their model size and dataset size was much closer to the "infinite model capacity" limit than the "infinite data" limit. This indicates that the model's performance ceiling is determined by data availability, not model capacity.
Implication: Adding more model parameters (increasing capacity) would yield negligible gains unless more data is provided. Conversely, the observed diminishing returns are a natural property of data scaling, not a failure of the tiny architecture.

5. Significance and Implications

Practical Guidance for Small Labs: The findings provide actionable advice for researchers with limited compute. Training on a moderate subset (e.g., 30% of the data) is often sufficient to reach near-saturated performance, drastically reducing costs and energy consumption.
Validation of Tiny Models: The study confirms that scaling laws observed in billion-parameter models also apply to "tiny" models (millions of parameters), provided the training dynamics are controlled.
Efficient Pretraining Strategy: The paper advocates for subset-based pretraining with early stopping as a principled strategy for base model development. It suggests that chasing the final few percentage points of accuracy via full-dataset training is often inefficient.
Architectural Simplification: The results suggest that for data-constrained scenarios, complex architectures with MLPs may be unnecessary, as the self-attention mechanism alone captures the majority of the learning signal when data is the limiting factor.

In conclusion, the paper demonstrates that "more data" is not always "worth the cost." In controlled, tiny-model settings, a significant portion of performance can be achieved with a fraction of the data, offering a highly efficient pathway for exploratory model development and resource-constrained research.