💬 NLP

Chain-of-Experience for Continual LLM Improvement

This paper introduces Chain-of-Experience (CoE), a framework enabling large language models to continuously improve during inference by accumulating iterative experiential traces from self- and environmental feedback, which consistently outperforms zero-shot baselines across math, coding, and knowledge tasks while reducing API costs.

Original authors: Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie, Shen Yan

Published 2026-08-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoqin Tu, Yunhao Fang, Yizhong Wang, Cihang Xie, Shen Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human beings are built to learn from their mistakes. When we solve a difficult problem, fail, and then try again with the knowledge of what went wrong, we do not simply restart from zero; we carry the lesson forward, refining our approach until we succeed. This continuous loop of action, feedback, and adjustment is the engine of human intelligence. For decades, the most advanced computer programs known as large language models have operated quite differently. Once these systems are trained, they are deployed in a fixed state, treating every new question as an isolated event. They generate an answer and stop, ignoring the rich feedback that exists within the problem-solving process itself. They do not learn from the experience of the moment.

This gap between how humans learn and how machines currently operate has led researchers to ask a fundamental question: can these digital minds be taught to learn while they are working, using the experience they gather in real time? The answer lies in a new approach called the Chain-of-Experience. This method treats the entire history of a model's attempts at a problem as a single, evolving story. Instead of discarding a failed attempt, the system feeds the result back into the model, allowing it to read its own previous errors and the feedback it received, then generate a new, improved response. It is a continuous cycle where the model learns from its own journey, updating its understanding with every step it takes.

In a recent study, researchers set out to test whether this concept could actually work across a wide range of difficult tasks. They took eight of the most powerful language models available today, including systems from major technology companies, and placed them in a series of challenging environments involving mathematics, computer programming, and complex knowledge questions. The goal was to see if these models could improve their performance simply by interacting with feedback during the test, without any changes to their underlying training. The researchers designed a system where the models could receive different types of guidance. Sometimes, the feedback came from the model itself, acting as a critic to point out flaws in its own logic. Other times, the feedback came from an external environment, such as a computer program that could instantly tell the model if a piece of code ran successfully or if a mathematical answer was correct.

The results were striking. When these models were allowed to use this iterative process of learning from experience, they consistently outperformed their standard versions that were not allowed to reflect on their mistakes. The improvement was not marginal; the models achieved a significant boost in accuracy across the board. In the realm of computer coding, where the feedback was precise and immediate, the models saw their success rates jump by 8.6 percentage points on average. Even in more abstract tasks like solving complex math problems, where the feedback was less direct, the models still managed to improve their scores by over five percentage points. This suggests that the ability to learn from experience is a powerful tool that can be unlocked even without retraining the model's core brain.

Perhaps more surprising was the efficiency of this new method. The researchers found that these models did not just get better; they got better while spending less money and computing power. By using feedback to guide their thinking, the models reached higher levels of accuracy with fewer attempts and shorter responses. In fact, the study showed that the models could achieve their best results with a nineteen percent reduction in the cost of running them compared to traditional methods that do not use this feedback loop. This indicates that the models were not just guessing more times; they were thinking more effectively, using the information they gathered to avoid dead ends and focus on the right path.

The study also revealed a fascinating pattern regarding the models' own capabilities. The researchers observed that the models which were already the strongest at a task were also the ones that improved the most when given the chance to learn from experience. It appears that a model's capacity to learn from its own history is tied to its initial intelligence; the smarter the model starts, the better it becomes at digesting feedback and evolving its strategy. This correlation was strong across the different tasks tested, suggesting that the ability to learn from experience is not a separate skill but an emergent property of a powerful reasoning system.

Even when the feedback was imperfect or misleading, the models showed a remarkable resilience. In experiments where the researchers deliberately gave the models false information, such as telling them a wrong answer was correct, the models did not collapse. While their performance did dip slightly, the strongest models were able to recover and often still find the right solution. This robustness suggests that these systems are not blindly following instructions but are actively reasoning through the information they receive, weighing the feedback against their own internal logic. The study also found that most of the learning happened very quickly. The models made the vast majority of their improvements in the first few rounds of feedback, after which their performance tended to level off. This implies that the initial moments of reflection are the most critical for learning.

The researchers also explored whether combining different types of feedback could lead to even better results. They found that when a model received both its own internal critique and an external signal confirming the correctness of an answer, the two sources of information worked together to produce the highest scores. This combination allowed the models to benefit from the broad perspective of self-reflection and the precise verification of an external check. However, when the researchers tried to simplify the process by summarizing past experiences into a short memory, the models performed worse. This suggests that the detailed, step-by-step history of the model's attempts contains crucial information that gets lost when the experience is compressed too tightly.

Ultimately, this work demonstrates that large language models are capable of a form of learning that was previously thought to be impossible without retraining. By allowing them to accumulate experience and interact with feedback during the problem-solving process, they can evolve and improve in real time. The study confirms that this approach is not only effective but also efficient, offering a way to get more out of these powerful tools without the massive computational cost of traditional training. As these systems continue to develop, the ability to learn from experience may become a standard feature, bridging the gap between how machines think and how humans learn.

Technical Summary: Chain-of-Experience for Continual LLM Improvement

Problem Statement

Conventional Large Language Models (LLMs) operate in a fixed state post-training, treating every inference as an isolated event. This paradigm ignores the rich feedback embedded in the problem-solving process itself, failing to leverage the model's ability to improve through iterative interaction during test time. While existing test-time strategies like Chain-of-Thought (CoT) or search-based methods (e.g., Tree-of-Thought) explore multiple reasoning paths, they often discard accumulated experience after generating a single answer. Similarly, self-refinement methods (e.g., Reflexion, Self-Refine) utilize iterative loops but often lack the capacity to accumulate deep, continuous experience or effectively integrate diverse feedback signals. The paper addresses the gap in enabling LLMs to evolve from accumulated experience and interactions to improve performance dynamically at inference time.

Methodology: Chain-of-Experience (CoE)

The authors propose Chain-of-Experience (CoE), a framework where models engage in iterative problem-solving by accumulating experiential traces through interactions with self or environmental feedback.

Formal Definition

In the traditional setting, an LLM generates a response $A$ given a question $Q$ sampled from $P(A|Q)$ . CoE extends this into a sequential decision process where the response $a_t$ at step $t$ depends on the full history of prior attempts and their corresponding environmental feedback.
The generative process is defined as:
$a_t \sim P(a_t | Q, e_0, e_1, \dots, e_{t-1})$
where each experience $e_i$ consists of a pair $(a_i, f_i)$ , with $a_i$ being the action (response) and $f_i$ being the feedback.

Feedback Spectrum

The study investigates four distinct feedback types to characterize how experience shapes model evolution:

No Feedback: The environment provides no signal ( $f_i = \emptyset$ ). Improvements must rely solely on internal reflection of prior attempts.
Execution Feedback: For executable tasks (e.g., coding), feedback is generated by running the model's response in an environment (e.g., an interpreter), providing traces, error messages, or test outcomes.
Model Feedback: An auxiliary model (which can be the same model) acts as a judge or critic, providing textual critiques, preference scores, or structured evaluations.
Correctness Feedback: An oracle-like binary signal ( $f_i \in \{0, 1\}$ ) indicating whether the answer is correct. This serves as a high-signal upper bound for iterative refinement.

Experimental Setup

The framework was evaluated across eight state-of-the-art LLMs (including GPT-5, o3, o4-mini, Gemini-2.5 Pro, and Claude 4.5 Sonnet) on six benchmarks spanning three domains:

Math: AIME 2025, OmniMath.
Coding: LiveCodeBench (V6), LiveBench (Code).
Knowledge: EvaLearn, GPQA Diamond.

Baselines included standard Few-shot In-Context Learning (ICL), Dynamic Reference Sheet (DRS), Agentic Context Engineering (ACE), and varying levels of built-in reasoning depth (low vs. high).

Key Contributions and Results

1. Performance Gains via Iterative Feedback

CoE with feedback consistently outperforms feedback-free baselines and other test-time scaling strategies.

Self-Feedback: Using only self-feedback, models achieved an average improvement of 7–9% over baselines like ICL, DC, and ACE (rising from ~62.9% to 71.0% average accuracy).
Executor/Correctness Feedback: For coding tasks, programmatic executor feedback drove an average 8.6% accuracy gain (66.4% to 75.0%). Correctness feedback established the upper bound for performance.
Comparison: CoE with feedback significantly outperformed methods that rely on experience from other tasks (like DRS and ACE), demonstrating that same-task iterative refinement is more effective for these models.

2. Efficiency and Cost-Effectiveness

Contrary to the intuition that iterative methods increase costs, CoE improves the accuracy-per-token ratio, though cost reductions vary by domain.

API Cost: The study reports an overall 5.6% improvement in accuracy. Regarding costs, the paper highlights specific reductions across different tasks: coding tasks saw a 13.4% reduction in API calls with self-feedback compared to executor feedback (e.g., $70.7 vs. $81.6); AIME 2025 saw a 47.3% reduction; and EvaLearn saw a 7.0% reduction compared to no-feedback counterparts. The paper notes a 19% lower API cost specifically in the context of coding tasks when comparing feedback-based variants to non-iterative baselines.
Token Efficiency: CoE with correctness/executor feedback achieved higher accuracy with token counts comparable to other multi-round methods, yielding a significantly higher return per token. For instance, on AIME 2025, correctness feedback reached 84.6% accuracy with 108K tokens, whereas DRS used 11K tokens but only achieved 74.7%.

3. Correlation Between Base Ability and Learning Capacity

The study observed a positive correlation between a model's base zero-shot performance and its capacity to improve via CoE.

Across five benchmarks, the average Pearson correlation was +0.5.
Stronger models (e.g., GPT-5, Claude 4.5) demonstrated significantly higher learning gains from feedback compared to weaker variants, suggesting that the ability to "learn from experience" at test time is an emergent property that scales with model capacity.

4. Robustness and Improvement Patterns

Spurious Feedback: Models remained robust under weak or spurious feedback (e.g., always being told an answer is incorrect). While performance dipped slightly on average, stronger models showed minimal degradation. Selective Majority Voting (SelMV) further helped maintain performance under adversarial signals.
Timing of Gains: Most performance gains occurred early in the iteration process (within the first 20 steps), with diminishing returns in later stages.
Attribution: Analysis of 6,630 incorrect-to-correct transitions revealed that 47.7% of improvements were directly driven by feedback fidelity, while 30% in coding tasks stemmed from specification recall. Self-generated feedback was found to be more contextually aligned than external signals.

5. Complementary Feedback Channels

Combining complementary feedback channels (e.g., Model Feedback + Correctness/Executor signals) yielded additional gains over single-channel approaches, particularly on AIME 2025 and LiveBench (Code). However, memory-based selection methods (compressing experience into a "reference sheet") did not outperform full experience trails, suggesting that aggressive compression may discard critical intermediate reasoning steps.

Significance and Claims

The paper claims that Chain-of-Experience (CoE) represents a shift from static inference to dynamic, test-time learning. Its primary significance lies in demonstrating that:

LLMs can autonomously improve at inference time without parameter updates, provided they are given iterative feedback loops.
Feedback is a critical driver of this improvement, with even self-generated feedback offering substantial gains over zero-shot or static few-shot approaches.
Efficiency is achievable: The framework allows models to reason more efficiently, achieving higher accuracy with lower computational costs compared to existing test-time scaling methods, with specific cost reductions observed across coding and math domains.
Model Capacity Matters: The ability to learn from experience is not uniform; it scales with the base capability of the model, implying that future advancements in model architecture will further enhance test-time adaptability.

The authors conclude that CoE offers a general and effective framework for test-time learning, enabling LLMs to evolve through accumulated experience and interaction, bridging the gap between static deployment and continuous learning.