Technical Summary: Continual Learning Bench (CL-BENCH)

Problem Statement

While there is substantial interest in building AI systems that improve through sequential experience (continual learning), no high-quality benchmark exists to rigorously evaluate this capability. Existing evaluation protocols for language agents and memory systems largely rely on proxies such as recall fidelity over long contexts, question-answering on prior data, or offline task accuracy. These metrics fail to test whether a system genuinely learns online by discovering and exploiting latent structure shared across related tasks. Furthermore, current benchmarks often conflate performance improvements with underlying model capability, making it difficult to isolate true learning from pre-trained knowledge.

Methodology

The CL-BENCH Framework

The authors introduce Continual Learning Bench (CL-BENCH), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve through sequential experience. The benchmark spans six diverse real-world domains:

1. Blind Spectrum Monitoring: Mapping persistent RF transmitters from partial, noisy scans.
2. Codebase Adaptation: Solving GitHub issues across two repositories, learning file layouts and test patterns.
3. Cohort Studies: Estimating patient survival curves across five distinct epidemiological studies with varying schemas and biases.
4. Database Exploration: Answering natural language questions about an unknown SQLite database with obfuscated schemas and data formats.
5. Exploitable Poker: Playing against deterministic opponents with fixed, exploitable strategies.
6. Sales Prediction: Forecasting demand across stores and products with rotating data visibility and distractor columns.

Task Design Criteria

To ensure tasks measure learning rather than static capability, CL-BENCH tasks are constructed to satisfy three strict criteria:

• Headroom: Initial performance must be well below the maximum achievable. The latent structure to be learned (e.g., schema conventions, opponent strategies) must be task-specific and not recoverable from general offline training alone.
• Shared Latent Structure: Instances within a task must share discoverable structure (e.g., codebase layout, disease dynamics) that a system can exploit to improve performance over time. This structure is not explicitly communicated but must be inferred from experience.
• Learning Mechanism: The environment must provide a feedback loop (e.g., error messages, intermediate predictions) allowing accumulated experience to translate into actionable knowledge for future instances.

Tasks also incorporate concept drift (e.g., database migrations, changing opponent strategies) to test whether systems can adapt and discard stale knowledge rather than blindly memorizing.

Validation Pipeline

Each task undergoes a rigorous validation process:

1. Author Review: Two authors verify the task meets the three design criteria.
2. Expert Panel: 2–3 domain experts per task review the specification for Realism (reflecting real-world workflows), Reusable Knowledge (patterns transferring across steps), and Learning Improvement (measurable performance gains).
3. Decision: Tasks are accepted only if they pass validation on all axes.

Evaluation Metrics

The paper introduces two primary metrics to analyze performance:

1. Reward: The cumulative outcome across a sequence of instances (e.g., query efficiency, prediction accuracy).
2. Gain: A metric designed to isolate learning from base capability. It is defined as the difference between a system's stateful performance ($r_{sf}$) and its performance if run statelessly ($r_{sl}$) on the same instances: $g_t = r_{sf,t} - r_{sl,t}$.
   • Normalized Gain: To allow cross-task comparison, gain is normalized by the system's specific "headroom" (the difference between the task maximum and the stateless baseline).

Key Contributions

1. CL-BENCH Benchmark: A suite of six expert-validated, difficult tasks across real-world domains specifically designed to isolate online learning from base model capability.
2. Metric Innovation: The introduction of the Gain metric and its normalized variant, which allows researchers to distinguish between a system that is simply "strong" and one that effectively learns from sequential experience.
3. Empirical Evaluation: A large-scale evaluation of frontier models (including Claude Opus 4.7, Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro) across various agent architectures.

Results

The evaluation of frontier models across CL-BENCH reveals significant gaps in current continual learning capabilities:

• Naive ICL Outperforms Dedicated Memory: Surprisingly, In-Context Learning (ICL) with full context retention (without specialized memory modules) consistently outperforms systems with dedicated memory architectures (e.g., Mem0, ACE, ICL Notepad). For instance, ICL with Claude Sonnet 4.6 achieved the highest normalized reward (22.3%) and gain (25.4%).
• Dedicated Memory Systems Struggle: Systems designed specifically for memory management (like ACE or ICL Notepad) often performed worse than naive ICL, sometimes ranking at the bottom of the leaderboard despite higher costs.
• Limited Online Adaptation: Even the best-performing systems achieved only a 25.4% normalized gain over their stateless baselines. This indicates that current systems leave substantial headroom unrealized.
• Failure Modes:
  • Overfitting: Agents frequently overfit to immediate observations or fail to reuse knowledge across instances.
  • Stale Beliefs: Systems struggle to update incorrect beliefs even when provided with feedback indicating errors (e.g., failing to detect database migrations).
  • Plasticity vs. Stability: Decomposition of gain shows that while some systems adapt quickly to new task variants (plasticity), they often fail to retain useful structure across variants (stability), or vice versa.

Significance and Claims

The paper claims that CL-BENCH is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks, successfully isolating online learning from underlying model capability.

The authors conclude that reliable online adaptation remains an open problem. The finding that naive ICL outperforms dedicated memory mechanisms suggests that current approaches to memory management and stateful agent design are not yet effective. The paper posits that accumulated state frequently hurts rather than helps, as memory modules introduce spurious generalizations and stale beliefs.

The authors position CL-BENCH as a foundation for tracking progress in this field, with clear criteria provided for future contributors to propose new tasks. They explicitly state that the benchmark is modest in its current scope (covering six domains and tens of instances) but aims to grow with community contributions to address the broader challenge of building AI systems that genuinely improve through sequential experience.