🤖 AI

SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization

This paper introduces SE-Bench, a diagnostic environment that obfuscates the NumPy library to rigorously benchmark self-evolution by isolating knowledge internalization from prior knowledge and reasoning complexity, revealing that closed-book training and self-play are more effective than open-book training or standard RL for embedding new knowledge into model weights.

Original authors: Jiarui Yuan, Tailin Jin, Weize Chen, Zeyuan Liu

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiarui Yuan, Tailin Jin, Weize Chen, Zeyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant student how to use a brand-new, completely alien programming language. The catch? This language looks and acts exactly like a language they already know (like Python's NumPy), but every single word and command has been scrambled into nonsense.

For example, instead of writing numpy.mean, they must write zwc.kocito.

This is the core idea behind SE-BENCH, a new "exam" created by researchers at Tsinghua University to test if AI agents can truly learn and remember new skills on their own, rather than just guessing or relying on a cheat sheet.

Here is a breakdown of their findings using simple analogies:

1. The Problem: The "Fake New" Test

Current AI tests are flawed. If you ask an AI to solve a math problem using a new library, the AI might just be remembering it from its training data (like a student who memorized the textbook before the test). Or, if the AI fails, you don't know if it's because it forgot the library or because the math problem was too hard.

The SE-BENCH Solution:
The researchers created a "scrambled" version of a common library called NumPy.

The Scramble: They renamed every function to random nonsense words (e.g., mean becomes kocito).
The Rule: The AI is not allowed to use the original library. It must use the nonsense words.
The Goal: If the AI can solve the problem using the nonsense words, it proves it actually learned the new rules. If it can't, it failed to internalize the knowledge.

2. The Three Big Discoveries

The researchers tested different ways to teach the AI and found three surprising truths:

A. The "Open-Book Paradox" (Don't let them cheat!)

The Setup: They tried teaching the AI in two ways:
1. Open-Book: The AI could see the dictionary (the documentation) while studying and while taking the test.
2. Closed-Book: The AI could see the dictionary while studying, but it had to throw the dictionary away before taking the test.
The Result: The "Open-Book" students failed completely when the dictionary was removed. They hadn't actually learned the material; they were just looking it up.
The Lesson: To make the AI truly learn and store the knowledge in its "brain" (its internal weights), you must force it to study without the dictionary during the training updates. It needs to compress the information into itself.

B. The "RL Gap" (Reinforcement Learning is a bad teacher for facts)

The Setup: They tried using Reinforcement Learning (RL), a method where the AI learns by trying things and getting rewards for correct answers (like a dog learning tricks).
The Result: Even with the "Closed-Book" method, RL failed. The AI couldn't learn the new nonsense words.
The Reason: The researchers found that the "safety brakes" built into RL (called PPO clipping) prevent the AI from making the big, bold changes needed to memorize new vocabulary. It's like a teacher who is too afraid to let the student make big mistakes, so the student never learns the new rules.
The Lesson: RL is great at polishing behavior, but it is terrible at teaching brand-new facts from scratch.

C. The "Self-Play" Success (Teaching yourself works, if you use the right method)

The Setup: They let the AI generate its own practice problems and try to solve them (Self-Play).
The Result:
- If they used RL for self-play, the AI failed (0% success).
- If they used Supervised Fine-Tuning (SFT) (a more direct teaching method) on the AI's own generated problems, it succeeded.
The Lesson: AI can teach itself new skills, but only if the "teaching method" (SFT) is strong enough to force the knowledge into its brain. RL is not a "one-size-fits-all" solution for self-evolution.

Summary Analogy

Think of the AI as a student preparing for a surprise exam on a new subject.

The Test: The exam uses a made-up language that looks like English but has different words.
The Failure: If the student is allowed to hold the dictionary during the exam, they pass, but they haven't learned anything. If they try to learn by just guessing and getting "points" for right answers (RL), they get stuck because the rules for guessing are too strict.
The Success: The only way the student truly learns is if they practice with the dictionary, but then are forced to take the exam without it (Closed-Book Training). This forces their brain to actually memorize the new words.

The Bottom Line:
SE-BENCH proves that for AI to truly "evolve" and learn new skills on its own, we need to stop letting them rely on external cheat sheets during training and stop using Reinforcement Learning for teaching new facts. We need to force them to memorize the knowledge internally.

Technical Summary: SE-BENCH: Benchmarking Self-Evolution with Knowledge Internalization

Problem Definition

The paper addresses a critical gap in evaluating self-evolving agents: the ability to internalize novel knowledge and skills for lifelong learning. While current benchmarks evaluate transient adaptations (e.g., inference-time refinement or iterative self-modification), they fail to rigorously measure the foundational capability of knowledge internalization—the process where an agent actively learns from experience to permanently update its parameters.

Existing evaluations suffer from two fundamental confounds:

Entanglement of Prior Knowledge: It is difficult to distinguish whether an agent solved a task involving "novel" knowledge by learning it during the experiment or by recalling it from pre-training data.
Entanglement of Reasoning Complexity: Failures on complex tasks are ambiguous; they could stem from an inability to internalize the necessary knowledge or simply from the logical difficulty of the reasoning required.

To resolve this, the authors argue for a "Needle in a Haystack" test for self-evolution: an environment where tasks are algorithmically trivial if the knowledge is internalized, but impossible without it.

Methodology: SE-BENCH Construction

The authors introduce SE-BENCH, a diagnostic environment designed to obfuscate the NumPy library into a pseudo-novel package named ZWC. The construction pipeline involves three stages:

Obfuscation:
- Renaming: 268 common NumPy functions are mapped to randomized, semantically void identifiers (e.g., numpy.mean $\rightarrow$ zwc.kocito).
- Wrapper Implementation: A custom ZWCArray class wraps all inputs and outputs to prevent models from bypassing the obfuscation by calling standard methods on underlying NumPy arrays.
- Documentation Rewriting: The API documentation is rewritten by an LLM to describe the new ZWC package, ensuring the logic remains isomorphic to standard NumPy but the syntax is alien.
Question Generation:
- Single-Function Tasks: Problems requiring exactly one specific ZWC function to solve, ensuring 100% coverage of the library.
- Multi-Function Tasks: Complex problems requiring the composition of at least three functions from a sampled set to test generalization beyond simple memorization.
Filtering:
- Tasks are validated via a strict consensus protocol using three state-of-the-art models (Qwen3-Coder, Gemini-2.5-Pro, GPT-OSS-120B). A task is retained only if all three can solve it using standard NumPy.
- Human verification is performed on a random subset to ensure clarity.

Evaluation Protocol:

Training Phase: Agents are provided with the problem description and the relevant ZWC API documentation.
Testing Phase: Agents are provided only with the problem description. They must rely entirely on knowledge internalized during training.
Success Metric: A solution is correct only if it passes all test cases, uses ZWC APIs (verified via AST analysis), and contains zero imports of the original NumPy library.

Key Contributions and Insights

The paper presents three primary insights derived from evaluating various self-evolution strategies (Memory-based, SFT, RL, and Self-Play) on SE-BENCH:

1. The Open-Book Paradox

The presence of reference documentation during parameter updates inhibits long-term retention.

Finding: Standard Supervised Fine-Tuning (Open-SFT), where documentation is present during both trajectory collection and training, results in 0% performance on the test set (without docs).
Mechanism: Removing documentation during the training phase (Closed-SFT) forces the model to compress external logic into its weights rather than relying on context. Closed-SFT significantly outperforms Open-SFT, proving that true internalization requires "information starvation" during optimization.

2. The RL Gap

Standard Reinforcement Learning (RL) fails to internalize new knowledge, even under the Closed-Book setting.

Finding: Closed-RL achieves 0% performance.
Mechanism: The failure is attributed to two specific mechanisms in standard RL (specifically PPO/GRPO):
- PPO Clipping: Prevents the large probability mass shifts required to learn new vocabulary mappings (e.g., np.mean $\rightarrow$ zwc.kocito).
- Negative Gradients: The use of normalized advantages introduces negative reinforcement for below-average responses, which erodes tentative associations before they can solidify during the fragile early stages of memorization.
Hybrid Success: A hybrid approach (Closed-SFT followed by RL) achieves state-of-the-art performance. Here, SFT establishes the foundational memory, and RL acts as a behavioral optimizer, pruning "lazy" guesses and improving execution discipline without correcting fundamental memory errors.

3. Viability of Self-Play

Self-evolution via self-generated data is viable, but the optimization mechanism is critical.

Finding: Pure RL self-play (Absolute-Zero) fails completely (0%). However, applying Closed-SFT to self-generated tasks and responses (Closed-SFTself) yields significant performance (22.5% on single-function tasks).
Implication: Models are capable of generating valid curricula to teach themselves; the failure of standard self-play is due to the unsuitability of RL for knowledge internalization, not the self-play paradigm itself.

Results

Baseline Performance: Memory-based methods (Expel) achieved non-trivial results (up to 47.1%) by mapping identifiers to known concepts, but failed to achieve perfect retention.
Parameter Optimization: Only Closed-SFT and the Hybrid Closed-SFT-RL methods succeeded. All other parameter-update methods (Open-SFT, Open-RL, Closed-RL, Absolute-Zero) failed completely.
Generalization: The findings (Open-Book Paradox and RL Gap) were validated across different model families (Qwen3 and Llama3) and scales (1.7B to 30B parameters).
Error Analysis: RL significantly reduced "Attribute Hallucinations" (guessing non-existent methods) by encouraging fallback to robust primitives, but did not correct "Function Hallucinations" (wrong API names) or "Signature Misalignments," confirming RL's role as a behavioral optimizer rather than a knowledge internalizer.

Significance

The paper positions SE-BENCH as a rigorous diagnostic platform for the self-evolving agent community. Its significance lies in:

Isolating Internalization: It provides a clean, controlled environment to separate knowledge internalization from reasoning complexity and prior knowledge leakage.
Diagnostic Tool: It serves as a "unit test" for future self-evolving agents, suggesting that agents must pass SE-BENCH to be trusted with complex, open-ended evolution.
Mechanistic Insight: It challenges the assumption that RL is a universal solution for self-improvement, demonstrating that SFT is superior for factual internalization while RL is better suited for optimizing the utilization of already-internalized knowledge.
Future Research: It establishes a foundation for studying the fundamental mechanisms of knowledge internalization, potentially guiding the design of more targeted hybrid algorithms for AGI development.