💬 NLP

Neural Neural Scaling Laws

This paper introduces Neural Neural Scaling Laws (NeuNeu), a neural network-based approach that outperforms traditional parametric models by framing downstream task performance prediction as a time-series extrapolation problem, achieving significantly lower error rates and zero-shot generalization across diverse model families and tasks.

Original authors: Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie, Kyunghyun Cho

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie, Kyunghyun Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how well a student will do on their final exam based on how they are doing in their daily homework.

For a long time, researchers have used a simple rule of thumb (called "Scaling Laws") to make these predictions. They assumed that if you just look at the student's average homework score, you can draw a smooth, predictable curve to guess their final grade. They thought, "If the average score goes up, the final grade will go up in a nice, straight line."

But the authors of this paper, Michael Y. Hu and his team, say that this simple rule is broken. Here is why, and what they did to fix it.

The Problem: The "Average" Lies

The paper argues that looking at an average score hides the truth.

The Analogy: Imagine two students both have an average score of 80%.
- Student A got every single question right except one. Their performance is steady and reliable.
- Student B got half the questions 100% and the other half 60%. Their performance is chaotic and unpredictable.
- If you only look at the "80% average," you can't tell the difference. But if you look at the individual questions (the "token-level" data), you see that Student A is likely to do better on the final exam than Student B.

The old methods (called "Logistic Scaling Laws") threw away all that detailed information. They just took the average, smoothed it out, and tried to fit a simple mathematical curve to it. The authors found that this fails when the student (or AI model) starts acting weirdly—sometimes getting better, sometimes getting worse, or hitting a plateau.

The Solution: NEUNEU (The "Neural" Predictor)

The team built a new tool called NEUNEU (Neural Neural Scaling Laws). Think of NEUNEU not as a simple calculator, but as a super-observant tutor.

Instead of just looking at the average score, NEUNEU looks at:

The History: How the student's grades have changed over time.
The Details: The specific performance on every single question, not just the average.

How it works (The Metaphor):
Imagine you are trying to guess how a car will perform on a long road trip.

The Old Way: You check the car's average speed so far and assume it will keep going at that speed forever.
The NEUNEU Way: You look at the engine's vibration, the fuel mixture, the driver's reaction time, and the specific bumps in the road the car just hit. You use a smart computer (a neural network) to learn from thousands of other cars to predict exactly how this specific car will handle the rest of the trip.

What They Found

The team trained NEUNEU on data from many open-source AI models (like students from different schools). They tested it on 66 different tasks (like answering science questions, writing stories, or solving logic puzzles).

Here are their main discoveries:

It's Much More Accurate: NEUNEU was 44% more accurate than the old "average-based" methods. It made fewer mistakes when guessing how well the AI would do in the future.
It Handles the Weird Stuff: Sometimes, making an AI bigger or training it longer makes it worse at certain tasks (a phenomenon called "inverse scaling"). The old methods couldn't predict this; they just assumed things would always get better. NEUNEU learned to spot these patterns and predict them correctly.
It's a "Zero-Shot" Genius: NEUNEU was trained on a specific set of models and tasks. But when they showed it a brand new type of model or a brand new task it had never seen before, it still worked better than the old methods. It's like a tutor who can teach a new subject they've never studied before just by looking at the student's study habits.
It Knows When It's Guessing: NEUNEU doesn't just give one answer; it gives a range of possibilities (like saying, "There's a 90% chance the score is between 75 and 85"). This helps researchers know when they can trust the prediction and when they should be careful.

The Bottom Line

The paper claims that trying to predict the future performance of AI models using simple, pre-written formulas (like the old "average score" method) is a dead end. Instead, we should use a smart, flexible computer system (NEUNEU) that learns directly from the messy, detailed data of how models actually behave.

By doing this, they can predict future AI performance with much higher accuracy, helping researchers decide which AI models are worth the time and money to build.

Technical Summary: Neural Neural Scaling Laws (NEUNEU)

Problem Statement

Neural scaling laws traditionally predict how language model (LM) performance improves with increased compute, data, and parameters, typically modeling the relationship between training loss and downstream performance using simple parametric forms (e.g., power laws or logistic functions). However, the authors identify two critical limitations in current approaches:

Signal Obscuration: Aggregate metrics like average validation loss average out rich token-level loss distributions, obscuring signals that may be predictive of downstream capabilities.
Functional Rigidity: Individual downstream tasks exhibit diverse scaling behaviors (monotonic improvement, plateauing, or inverse scaling) that cannot be captured by a single parametric family.

Consequently, predicting downstream performance from validation loss using fixed functional forms suffers from high bias and poor generalization to unseen model architectures, parameter counts, or tasks.

Methodology: NEUNEU

The authors propose Neural Neural Scaling Laws (NEUNEU), a neural network approach that frames scaling law prediction as a time-series extrapolation problem. Unlike parametric methods, NEUNEU learns to predict future performance without assuming a specific functional form.

Architecture and Input Representation

NEUNEU combines temporal context from observed accuracy trajectories with token-level validation information.

Loss Representation: To preserve distributional information and handle unbounded cross-entropy losses, the model converts token-wise losses ( $\ell$ ) into token-wise probabilities ( $p = e^{-\ell}$ ).
Input Sequence: The model processes a sequence of historical downstream accuracies ( $y_{1:t}$ $y_{1 : t}$ ) and compute gaps ( $g_{1:t}$ $g_{1 : t}$ ), alongside a vector of token probabilities ( $p_t$ $p_{t}$ ) from the validation set.
- Gaps: Compute intervals between evaluation steps are abstracted to ensure invariance across different model scales.
- Token Probabilities: The model ingests a vector of $N=256,000$ token probabilities (roughly one minibatch size), rather than a single averaged loss value.
Model Components:
1. Loss Encoder: A hierarchical 1D CNN (4 layers) processes the token probability vector to produce an embedding $e$ .
2. Transformer Backbone: A standard BERT-style transformer (6 layers, bidirectional attention, RoPE) processes a sequence containing a [CLS] token, the loss embedding $e$ , and context tokens formed by concatenating projected accuracies and gaps.
3. Prediction Head: The output from the [CLS] token is projected to predict a distribution over future accuracy using quantile regression. The model outputs 5 quantiles ( $\hat{q}_{0.1}, \hat{q}_{0.25}, \hat{q}_{0.5}, \hat{q}_{0.75}, \hat{q}_{0.9}$ ).

Training Strategy

Data: The model is trained entirely on open-source language model training trajectories from HuggingFace, specifically the DataDecide model suite. This includes 144 unique pretraining trajectories across 6 model sizes (90M to 1B parameters) and 24 datasets.
Objective: The model is trained using pinball loss summed across the 5 quantiles to produce calibrated uncertainty estimates.
Augmentation: To ensure robustness to missing data, the training process randomly drops tuples from the accuracy sequence, absorbing the gap into the preceding tuple.

Key Contributions

Parametric-Free Prediction: NEUNEU is the first model to predict downstream scaling performance without parametric assumptions or prior knowledge of specific functional forms.
Superior Accuracy: On 66 downstream tasks, NEUNEU achieves a Mean Absolute Error (MAE) of 1.99%, representing a 44% reduction in error compared to widely used logistic scaling laws (3.56% MAE).
Uncertainty Calibration: By utilizing quantile regression, NEUNEU provides calibrated uncertainty estimates. Its predicted 10%-90% interquantile interval captures 77.6% of ground truth data, closely matching the expected 80%.
Zero-Shot Generalization: The model generalizes zero-shot to:
- Unseen random seeds.
- Unseen pretraining datasets (e.g., C4).
- Unseen model families and architectures, including non-transformer models (e.g., OLMo-Hybrid-7B) and larger parameter counts (up to 12B) not seen during training.
- Unseen downstream tasks (held-out tasks during meta-training).

Results and Evaluation

The authors evaluated NEUNEU against several baselines:

Logistic Scaling Laws: Parametric fits per task.
BNSL (Broken Neural Scaling Laws): Parametric models allowing for regime breaks.
LC-PFN: A meta-learned transformer for in-context learning curve extrapolation.
Ablations: Variants of NEUNEU using only average loss (AVERAGE) or no loss information (NOLOSS).

Key Findings:

Generalization: NEUNEU significantly outperforms all baselines on out-of-distribution (OOD) evaluations, particularly on new architectures (Pythia, OLMo-Hybrid) where parametric laws fail catastrophically (incurring ~4x higher error).
Task Agnosticism: Unlike logistic scaling laws which require fitting a separate model per task, a single NEUNEU model predicts accuracy for any task, including those never seen during training, often outperforming logistic laws fitted directly to those specific tasks.
Decision Utility: NEUNEU correctly ranks the final performance of competing model configurations with 76.6% accuracy, a 12.9% improvement over logistic scaling laws. This suggests NEUNEU can effectively guide resource allocation and hyperparameter selection.
Information Value: The AVERAGE and NOLOSS ablations perform worse than full NEUNEU, confirming that token-level loss distributions contain predictive signal that averaging discards.

Significance and Claims

The paper argues that the limitations of existing scaling laws are not fundamental to the prediction problem but are artifacts of restricted hypothesis classes (i.e., assuming specific functional forms). By replacing parametric bias with the flexibility of a neural network trained on diverse open-source trajectories, NEUNEU demonstrates that predicting downstream scaling directly from data outperforms parametric alternatives.

The authors position NEUNEU as a step toward "Foundation Models of Training Dynamics." Rather than treating language models as black boxes, NEUNEU acts as a nascent model of training dynamics, capable of simulating future states given current performance and compute actions. This capability could allow practitioners to explore hyperparameter spaces and architecture choices without the prohibitive cost of real-world training runs. The work aligns with the principle that methods scaling with compute and data tend to outperform those relying on human-designed structures.