💻 computer science

Quiet Feature Learning in Algorithmic Tasks

This paper reveals that Transformer models trained on algorithmic tasks undergo "quiet feature learning," where critical intermediate computations are acquired during periods of stagnant loss, challenging the reliability of cross-entropy as a sole indicator of learning progress.

Original authors: Prudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan Paturi

Published 2026-01-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prudhviraj Naidu, Zixian Wang, Leon Bergen, Ramamohan Paturi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Silent Construction" Phase

Imagine you are watching a construction crew build a skyscraper. You are standing far away, looking at the building through a telescope. For a long time, the building looks exactly the same as it did yesterday. Nothing is changing. The "progress meter" (which measures how much of the building is finished) stays flat.

Then, suddenly, overnight, the entire top floor appears. The progress meter jumps from 0% to 100% instantly.

This paper argues that when AI models (specifically Transformers) learn to solve math and logic puzzles, they behave exactly like that construction crew. They spend a long time doing "invisible work" where their performance score (called loss) doesn't improve at all. Then, suddenly, they "click" into place and the score drops dramatically.

The authors call this invisible work "Quiet Feature Learning."

The Experiment: Teaching AI to Do Math and Logic

The researchers didn't ask the AI to write poetry or chat about feelings. Instead, they gave it ten very specific, strict logic puzzles, such as:

Adding two binary numbers (like 101 + 011).
Finding the shortest path through a maze (Graph search).
Picking the best schedule of events so they don't overlap (Activity selection).

They trained the AI on these tasks using different amounts of computer power (compute). They expected the AI to get slightly better and slightly better as they gave it more power, following a smooth, predictable curve.

The Surprise: The "Flat" Phase and the "Jump"

Instead of a smooth curve, they found a Phase Transition.

The Slow Phase (The Flat Line): As they added more computer power, the AI's error rate barely moved. It looked like the AI wasn't learning anything. The "progress meter" was stuck.
The Fast Phase (The Jump): Suddenly, at a specific point, the error rate plummeted. The AI went from being terrible to being perfect almost instantly.

This happened not just when they made the AI bigger, but also when they gave it more data or just let it train for longer.

The Secret: "Quiet Features"

Here is the most interesting part. The researchers didn't just look at the final score; they looked inside the AI's "brain" (its internal representations) while it was in that Slow Phase.

They found that the AI was learning, even though the score didn't show it. It was learning specific, intermediate steps required to solve the puzzle.

Analogy: Imagine learning to drive a car. You spend weeks learning how to turn the key, press the clutch, and check the mirrors. If you were graded only on "how fast you drove down the highway," you would get a zero for weeks because you haven't started driving yet. But inside your brain, you are mastering the "quiet features" of driving.
The Paper's Findings: The AI learned things like "carry bits" (a step in addition) or "queue management" (a step in searching a map) before it could actually solve the whole problem correctly.

The authors call these "Quiet Features." They are the internal building blocks that don't immediately make the final answer better, but are absolutely necessary for the final answer to exist.

Proving It: The "Amnesia" Test

To prove these quiet features were actually important, the researchers did a "sabotage" experiment.

They took a model that had learned these quiet features but hadn't yet solved the whole puzzle.
They surgically removed (ablated) just the specific "quiet feature" (like the ability to remember a carry bit).
The Result: The model's performance crashed. It couldn't solve the problem anymore.

This proved that the AI wasn't just "guessing" or memorizing; it had genuinely learned the logic steps, but those steps were sitting dormant, waiting for the rest of the system to catch up.

Why Does This Matter?

The paper concludes that we have been looking at AI training the wrong way.

The Old Way: We watch the "loss curve" (the error score). If the line is flat, we think the AI isn't learning.
The New Way: The paper suggests that a flat line might actually mean the AI is doing a massive amount of heavy lifting internally. It's like a caterpillar spinning a cocoon. From the outside, it looks like nothing is happening. But inside, a butterfly is being built.

Summary

The paper shows that AI models often learn the steps to a solution long before they can execute the solution. They accumulate "quiet" knowledge that stays hidden until a critical moment, when everything clicks together, and the model suddenly becomes smart. This challenges the idea that we can only judge an AI's learning by how well it performs on the final test.

Technical Summary: Quiet Feature Learning in Algorithmic Tasks

Problem Statement

The paper addresses the gap between established scaling laws in large language models (LLMs) and the observed "emergent abilities" where performance improves abruptly rather than smoothly. While scaling laws typically predict a smooth, power-law relationship between compute and performance, recent debates suggest that "emergent" jumps might be artifacts of measurement or averaging over heterogeneous tasks. The authors posit that these abrupt transitions may mask significant internal learning progress that remains invisible to standard loss metrics. To investigate this, the study focuses on ten foundational algorithmic tasks with precisely defined solutions, allowing for the isolation of specific features and the avoidance of memorization or heuristic artifacts.

Methodology

Experimental Setup

The authors trained Transformer++ models (a decoder-only architecture with enhancements based on Llama and PALM) on ten algorithmic tasks spanning binary arithmetic, graph algorithms, and sequence-based optimization. The tasks include:

Binary Arithmetic: Addition and Multiplication.
Graph Algorithms: Breadth First Search (BFS), Depth First Search (DFS), Shortest Path, Topological Sorting, and Minimum Spanning Tree.
Sequence Optimization: Maximum Subarray, Majority of Majorities, and Activity Selection.

Scaling Law Estimation:
Models were trained across a wide range of compute budgets ( $10^9$ to $10^{15}$ FLOPs). For each budget, a grid search was conducted over model sizes, batch sizes, and learning rates to identify the "compute-optimal" configuration (minimum validation loss). All models were trained for at most a single epoch to ensure learning rather than memorization.

Feature Probing:
To understand internal representations, the authors employed linear probing on the residual streams of the models.

Target Features: They defined specific "intermediate algorithmic computations" relevant to each task (e.g., carry bits for addition, queue state for BFS, max ending here for Maximum Subarray).
Probe Training: Separate linear probes (logistic regression for binary features, linear regression for continuous features) were trained on held-out data to detect these features at every token position and layer.
Ablation: To test causality, the authors ablated specific "quiet features" by removing their directional component from the residual stream and measuring the resulting degradation in task performance compared to random ablations.

Key Findings

1. Phase Transitions in Loss Curves

The study observes distinct phase transitions in the validation loss curves across three scenarios: varying model and dataset size, fixing model size while varying dataset size, and within individual training runs.

Slow Phase: A period where validation loss remains stagnant or decreases very slowly despite increasing compute.
Fast Phase: An abrupt, rapid decrease in loss, often leading to near-perfect accuracy.
Observation: This behavior deviates from the smooth power-law scaling trends typically reported in heterogeneous settings.

2. Quiet Feature Learning

A critical finding is that models learn meaningful internal representations before the loss drops (during the "slow phase").

Quiet Features: These are intermediate algorithmic computations (e.g., carry bits, adjacency lists) that the model learns to represent internally but which do not immediately translate into improved task loss.
Loud Features: Features learned during or after the phase transition, coinciding with rapid loss reduction.
Temporal Precedence: Probing reveals that quiet features emerge well before the phase transition. For tasks like addition and multiplication, features for early token positions are learned first. For BFS, later token positions are learned first.
U-Shaped Trajectory: In some cases, the probing loss for quiet features increases slightly after the phase transition, suggesting the model may be shifting to alternative representations in high-compute regimes, though performance remains above random baselines.

3. Causal Necessity

Ablation experiments confirm that quiet features are causally necessary for task performance.

Removing quiet features (e.g., the carry bit in addition) causes a significant drop in test accuracy (e.g., from 100% to ~25-50% depending on input size), far exceeding the degradation caused by ablating random directions.
This demonstrates that the model accumulates partial competence (subroutines) that remains dormant until all necessary components align to produce a correct output.

Significance and Claims

The paper argues that substantial representational progress can remain hidden beneath an apparently flat loss curve. The authors make the following claims regarding the significance of their findings:

Challenge to Loss as a Proxy: The results challenge the prevailing assumption that improvements in cross-entropy loss necessarily coincide with improvements in feature representations. Relying solely on loss curves may lead practitioners to underestimate a model's internal capabilities or misjudge when a model is nearing a capability threshold.
Mechanism of Emergence: The "quiet period" suggests that in over-parameterized models, the network accumulates necessary subroutines in the background. The abrupt transition occurs only when these partial solutions are fully aligned, allowing the model to execute the correct overall procedure.
Diagnostic Implications: The authors motivate the use of richer diagnostics, such as probe-based or circuit-based analysis, to monitor training dynamics. These methods could provide earlier warnings of capability acquisition than loss metrics alone.
Theoretical Gap: The findings highlight a need for theoretical frameworks that explain why models accumulate latent subroutines before they manifest in observable metrics, particularly in the context of algorithmic tasks where solutions are discrete and compositional.

The paper concludes by noting that while these phenomena were observed in algorithmic tasks, they raise questions about whether similar "quiet phases" exist in more complex natural language settings, suggesting that hidden phases of progress may precede sudden improvements in capability even in larger, more complex models.