🤖 machine learning

The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

This paper introduces the Generalization Spectrum, a novel evaluation framework that measures per-sample generalization across varying transfer distances, revealing that while reinforcement learning converts memorization into near-transfer more efficiently than supervised fine-tuning, various optimization techniques often fail to expand the generalization radius or may even reduce far-transfer capabilities.

Original authors: Jinghan Zhang, Zerui Cheng, Shiqi Chen, Ge Zhang, Wenhao Huang, Jiashuo Liu, Junxian He, Tianle Cai

Published 2026-06-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinghan Zhang, Zerui Cheng, Shiqi Chen, Ge Zhang, Wenhao Huang, Jiashuo Liu, Junxian He, Tianle Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Report Card" Illusion

Imagine you are a teacher grading students. Traditionally, we give every student a single score on a final exam. If Student A gets a 90% and Student B gets a 90%, we assume they are equally smart.

But this single score hides a huge secret: How did they get that score?

Student A might have just memorized the exact questions from the practice test. If you change the wording slightly, they fail.
Student B might have understood the underlying logic. If you change the wording or the context, they still get it right.

Current AI evaluation is like that single report card. It tells us if an AI learned something, but not how well it can apply that learning to new situations. This paper introduces a new way to look at AI learning, called the Generalization Spectrum.

The Analogy: Chromatography (The Ink Test)

The authors compare their method to chromatography, a chemistry technique used to separate ink into its different colors.

Traditional AI Testing: Looking at a black dot of ink and saying, "It’s black."
The Generalization Spectrum: Running that black dot through a solvent to see it separate into blue, red, and yellow streaks.

In this analogy:

The black dot is the AI’s overall performance score.
The solvent is the "transfer distance"—how much we change the problem from the original training example.
The colors are the different types of learning: rote memorization, surface-level adaptation, and deep understanding.

By watching how the "ink" spreads (or fades) as we change the problems, we can see exactly what kind of learning the AI is doing.

The Experiment: The "Distance" Ladder

To test this, the researchers used competitive programming (coding puzzles) because the answers are either strictly right or strictly wrong. They created a "ladder" of five levels, moving from the easiest (closest to the training data) to the hardest (furthest away).

Level D0 (Exact Recall): The exact same problem the AI studied. Did it memorize it?
Level D1 (Implementation Transfer): The same problem, but the AI must write the code in a different language (e.g., Python to C++). Did it learn the logic, or just the syntax?
Level D2 (Context Transfer): The same math problem, but wrapped in a completely different story (e.g., instead of "calculating train speeds," it’s "calculating rocket trajectories"). Did it understand the math, or just the story?
Level D3 (Category Match): A different problem that uses the same type of algorithm. Can it apply the technique to a new puzzle?
Level D4 (Unpaired Baseline): A random new problem. Can it handle the unknown?

The "Fair Fight" Rule: Matched Memorization

A major problem in AI research is that some methods are just better at memorizing. If Method X remembers 90% of the training data and Method Y remembers 50%, Method X will naturally look smarter on new problems, even if it’s not actually "learning" better.

To fix this, the researchers used Matched Memorization. They only compared the AI models when they had memorized the exact same amount of the original training data. This isolates the true ability to transfer knowledge, rather than just repeating it.

What They Found: Three Different Learners

They tested three main ways of teaching AI:

ICL (In-Context Learning): Showing the AI an example right before asking the question (like giving a student a cheat sheet).
SFT (Supervised Fine-Tuning): Training the AI to copy expert solutions (like a student copying a textbook).
RL (Reinforcement Learning): Training the AI by rewarding it when it gets the right answer, regardless of how it got there (like a student learning by trial and error).

The Results:

ICL (The Cheat Sheet): It was amazing at Levels D0–D2. If the problem looked similar to the example, it nailed it. But as soon as the problem changed significantly (Levels D3–D4), its performance crashed. It relied too heavily on the specific example provided.
SFT (The Copycat): It was good at memorizing (D0) and okay at changing languages (D1). But it failed badly when the story changed (D2) or when faced with new problems (D3–D4). It tended to memorize the surface of the code (like Python syntax) rather than the logic, so it struggled when things changed.
RL (The Problem Solver): This was the surprise winner for generalization. Even though it memorized the same amount as the others, it performed better at the far end of the spectrum (D3–D4). Because it was rewarded for correct answers rather than copying code, it learned the underlying logic better. It didn't just mimic the style; it understood the structure.

The "Variants" Test: Tweaking the Methods

The researchers also tested "upgraded" versions of these methods to see if they made the AI smarter or just faster at memorizing.

Adding Hints to ICL: Giving the AI extra clues (like "this is the same algorithm, just different language") helped it do better on the easy levels (D1–D2) but didn’t help it solve the hard, new problems (D3–D4). It lifted the local performance but didn’t expand the radius of understanding.
Self-Distillation (SDFT): This is where the AI tries to teach itself by mimicking its own successful attempts. It got better at the easy levels but actually got worse at the hard levels compared to standard training. It became too focused on the specific training examples.
Hint-Assisted RL: Giving the RL AI hints helped it learn the training data faster, but it ended up with a narrower range of understanding. It solved the training problems quickly but struggled more with new, unseen problems than the standard RL method.

The Takeaway

The paper concludes that not all learning is created equal.

Aggregate scores are misleading. Two AIs can have the same test score but have completely different abilities to handle new situations.
RL is currently better at "far transfer." Reinforcement Learning seems better at converting memorization into flexible, adaptable knowledge than copying expert code (SFT).
Shortcuts have limits. Adding hints or making training faster often improves performance on familiar tasks but can actually hurt the AI's ability to handle truly new challenges.

In short, this paper provides a new "microscope" for AI researchers. Instead of asking "Did the AI pass the test?", we can now ask, "How far does the AI's understanding stretch?"

Technical Summary: The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

Problem Statement
Traditional evaluations of learning algorithms reduce performance to a single aggregate score on an i.i.d. test set. This approach obscures a fundamental question: to what extent does learning from a specific training example generalize to others? Standard benchmarks fail to capture "per-sample generalization"—the distance knowledge extracted from one example can transfer—conflating distinct capabilities such as exact memorization, near-transfer under minor perturbations, and far-transfer across distributional shifts. Consequently, it is difficult to determine whether a method’s superior aggregate performance stems from genuine generalization or merely enhanced memorization.

Methodology: The Generalization Spectrum
The authors introduce the Generalization Spectrum, an evaluation framework that treats generalization as a function of transfer distance rather than a binary in-distribution/out-of-distribution split. The core methodology involves constructing paired test variants derived from the same seed instance, ordered by increasing distance from the original. This "chromatographic" approach separates learning components into distinct profile shapes.

The framework defines five levels of transfer distance ( $D_0$ – $D_4$ ) based on the progressive removal of shared attributes between a seed problem and its variant:

$D_0$ (Exact Recall): The original problem. Measures memorization.
$D_1$ (Implementation Transfer): Same problem logic and narrative, but different implementation format (e.g., Python to C++). Removes surface code realization.
$D_2$ (Context Transfer): Same executable specification and solution logic, but completely reframed natural-language narrative. Removes narrative context.
$D_3$ (Category-Matched): Different problem sharing the same algorithmic family/tag. Retains only the algorithmic category.
$D_4$ (Unpaired Baseline): Recent in-domain problems with no enforced structural or tag relationship to the seed.

Matched-Memorization Comparison
To isolate "transfer efficiency" (the ability to convert memorization into generalization), the authors propose comparing methods at checkpoints where they achieve comparable performance on $D_0$ . This controls for the confounding factor where stronger training simultaneously increases both memorization and transfer.

Benchmark Instantiation
The framework is instantiated on competitive programming, a domain chosen for unambiguous correctness (pass/fail on test cases) and the ability to construct controlled variants. The benchmark consists of 64 seed problems, generating 256 evaluation instances across $D_1$ – $D_4$ . $D_2$ variants are generated via a multi-stage pipeline involving cross-model generation and verification to ensure narrative reframing preserves mathematical structure.

Key Contributions

Framework: Introduction of the Generalization Spectrum and the concept of Generalization Profiles (curves of performance vs. transfer distance).
Metric: Definition of metrics including Gain, Normalized Gain, Area Under Spectrum (AUS), and Normalized Near-Far Gap (N-Fn) to quantify transfer decay.
Benchmark: Construction of a paired competitive programming benchmark with 64 seeds and 256 transfer instances.
Diagnostic Analysis: Application of the spectrum to compare canonical paradigms (ICL, SFT, RL) and within-family variants (RFT, SDFT, Hint-assisted RL).

Results

Paradigm Comparison (ICL vs. SFT vs. RL):
- Under matched memorization ( $D_0$ ), RL (GRPO) converts memorization into near-transfer more efficiently than SFT and preserves a positive far-transfer tail. SFT gains concentrate near implementation transfer ( $D_1$ ) and largely exhaust after narrative recontextualization ( $D_2$ ), often showing negative gains at far distances ( $D_3$ – $D_4$ ).
- Paired ICL exhibits strong transfer at instance-paired levels ( $D_0$ – $D_2$ ), including context transfer, but drops sharply at category-matched ( $D_3$ ) and unpaired ( $D_4$ ) levels. This reveals a "correspondence bottleneck" rather than uniformly stable transfer. Random ICL performs worse than the base model, highlighting the brittleness of context-based adaptation without precise retrieval.
- Failure Mode Analysis: SFT suffers from high compile-error rates in $D_1$ (cross-lingual transfer) due to imitation of surface Python patterns. RL maintains stable compilation. In $D_2$ , SFT’s algorithm-selection gain is tied to recognizing the original statement wording, whereas RL preserves structural solution-family identification under narrative recontextualization.
Within-Family Variants:
- ICL Content Variants: Adding abstractions (pseudocode, key insights) or hints lifts performance at paired levels ( $D_0$ – $D_2$ ) but does not extend transfer to far distances ( $D_3$ – $D_4$ ). The interventions reshape the local profile without expanding the generalization radius.
- SFT Target-Source Variants: Rejection Fine-Tuning (RFT), which uses self-generated correct targets, preserves a smoother far-transfer tail compared to demonstration SFT (which imitates external reference solutions). Demonstration SFT shows weaker semantic transfer and higher compile errors in cross-lingual tasks.
- Self-Distillation (SDFT): SDFT improves local transfer ( $D_1$ – $D_2$ ) over vanilla SFT but performs worse at far distances ( $D_3$ – $D_4$ ), behaving like an "ICL-distillate" that writes demonstration-conditioned local structure into weights without preserving benefits beyond instance-level correspondence.
- Hint-Assisted RL: Training-time hints (pseudocode or key insights) improve sample efficiency (faster convergence to matched $D_0$ ) but compress the far-transfer tail. Hint-assisted GRPO falls below verifier-only GRPO at $D_3$ – $D_4$ , suggesting scaffolds tie learning to the scaffolded seed relation rather than expanding generalization.

Significance and Claims
The paper claims that the Generalization Spectrum provides a diagnostic tool for understanding learning methods beyond single-score comparisons. It reveals that different post-training methods change different parts of the generalization profile even when aggregate gains appear similar. Specifically, it demonstrates that auxiliary signals (hints, abstractions, distillation) do not uniformly expand generalization; instead, they reshape specific parts of the spectrum—lifting local transfer, preserving far-transfer tails, or accelerating optimization speed—often involving trade-offs between near and far transfer. The framework enables practitioners to evaluate not just whether an algorithm learns, but how far that learning extends.