💬 NLP

Learning from Supervision with Semantic and Episodic Memory: A Reflective Approach to Agent Adaptation

This paper proposes a memory-augmented framework that leverages LLM-generated critiques stored in episodic and semantic memory to improve agent adaptability and reduce inference costs without parameter updates, while introducing a "suggestibility" metric to explain performance variations across different models and domains.

Original authors: Jackson Hassell, Dan Zhang, Hannah Kim, Tom Mitchell, Estevam Hruschka

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jackson Hassell, Dan Zhang, Hannah Kim, Tom Mitchell, Estevam Hruschka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI Without Rewiring Its Brain

Imagine you have a brilliant but stubborn student (the AI). This student has read almost everything in the library (pre-trained on massive data), but when you give them a specific test on a new topic, they sometimes get it wrong.

Usually, to fix this, you have to send the student back to school for months to relearn everything (this is called fine-tuning). It's expensive, slow, and you might accidentally make them forget what they knew before.

This paper proposes a different approach: Don't rewire the brain; just give the student a better study guide.

Instead of changing the student's internal knowledge, the researchers built a "memory system" where the AI can look up past mistakes and the specific reasons why they were mistakes. They call this Reflective Learning.

The Two Types of Memory: The Sticky Note vs. The Textbook

The researchers gave the AI two types of memory to help it learn from labeled examples (questions with correct answers):

Episodic Memory (The "Sticky Note" or "Case File"):
- What it is: This is a collection of specific past experiences. Imagine a detective's case file. When the AI faces a new question, it looks back at 5 similar past cases it solved (or failed) and reads the notes attached to them.
- The Twist: It doesn't just look at the question and the answer. It looks at a Critique. A Critique is a note written by a "Teacher AI" that says: "You got this wrong because you missed this specific detail. Here is exactly why the right answer is right."
- Analogy: It's like checking your old math homework before a test, but instead of just seeing the answer key, you have a teacher's red pen explaining exactly where your logic went wrong.
Semantic Memory (The "Textbook" or "Cheat Sheet"):
- What it is: This is a high-level summary of all the mistakes and lessons learned from the entire dataset.
- The Twist: The AI takes all those specific "Sticky Notes" and condenses them into a single, short list of rules or general advice.
- Analogy: If the Sticky Notes are individual math problems, the Textbook is the "Top 10 Rules for Solving Algebra" that the teacher wrote after grading 1,000 tests. It gives the AI a broad strategy without needing to read every single past case.

The Winning Strategy: The paper found that using both is best. The AI gets the broad rules (Textbook) and the specific examples (Case Files) at the same time.

The "Teacher" and the "Student"

The system works like this:

The Student (Performance Agent): Tries to answer a question.
The Teacher (Critic Agent): Looks at the Student's answer and the correct answer. It writes a detailed critique explaining the error.
The Memory: The critique is saved into the Episodic (specific) and Semantic (general) memory banks.
The Next Test: When the Student faces a new question, it checks its memory banks for similar past mistakes and the Teacher's advice before answering.

Why This Matters: Three Big Discoveries

1. It Actually Works (and is Better than just showing examples)

Usually, if you want an AI to learn a new task, you just show it a few examples of questions and answers (like a flashcard). This paper shows that showing the AI critiques (explanations of errors) is much more powerful.

The Result: On average, this method improved the AI's accuracy by 8.1% over doing nothing, and 4.6% over just showing it standard examples. It's like the difference between a student who just memorizes answers and one who understands why the answers are correct.

2. It Saves "Thinking Time" (The Efficiency Hack)

This is a surprising finding. Some modern AI models are "reasoning models"—they think out loud, generating a long chain of thoughts before answering.

The Problem: Sometimes these models "overthink." They wander down rabbit holes, get confused, and run out of time (or computer tokens) before giving an answer.
The Fix: When the AI has a pre-written critique in its memory, it doesn't need to figure out the logic from scratch. It can just say, "Oh, I remember this type of problem; the Teacher said to do X."
The Result: The AI stopped "thinking" as much. It reduced its internal thinking tokens by nearly 32%. It was faster, cheaper to run, and actually got more answers right because it didn't get lost in its own thoughts.

3. Not All AI is Equally "Suggestible"

The researchers noticed that some AI models improved a lot, while others barely changed. They invented a new metric called Suggestibility.

What it is: How willing is the AI to listen to new information provided in the prompt, even if it contradicts what it "knows" from its training?
The Finding: Models that are highly "suggestible" (willing to listen to the Teacher's critique) improved the most. Models that are stubborn (relying only on their internal training) didn't improve much.
The Catch: High suggestibility is a double-edged sword. If the AI is too suggestible, it might believe a lie if the Teacher gives it bad advice. The paper notes this is a safety concern: the same trait that helps it learn quickly also makes it vulnerable to being tricked.

Summary

The paper demonstrates that you don't need to retrain an AI to make it smarter at new tasks. Instead, you can give it a memory system that stores specific past mistakes and general lessons learned. By reading these "critiques" before answering, the AI becomes more accurate, thinks less (saving money and time), and adapts quickly—provided it is willing to listen to the feedback.

1. Problem Statement

Large Language Model (LLM) agents currently face significant challenges when adapting to new tasks or domains without parameter updates:

Fine-tuning Limitations: Traditional fine-tuning is computationally expensive, requires large labeled datasets, risks "catastrophic forgetting," and lacks interpretability.
In-Context Learning (ICL) Limitations: Standard few-shot learning (RAG) often relies on raw input-output examples. This leads to shallow pattern mimicking rather than deep conceptual understanding.
Static Knowledge: Pre-trained models cannot easily incorporate new supervised signals (labeled data) in real-time to improve performance on unseen tasks.

The authors propose a solution that enables agents to learn continuously from labeled supervision without updating model weights, leveraging a memory-augmented architecture grounded in self-critique.

2. Methodology

The framework introduces a Reflective Approach where an agent learns from labeled data by generating and storing structured critiques. The system consists of two main agents and a dual-memory architecture:

A. Agent Roles

Performance Agent (PA): The primary LLM tasked with solving the problem (e.g., answering questions).
Critic Agent (CA): An LLM (often the same model as the PA) that generates feedback. It takes the PA's initial prediction, the ground-truth label, and the input to generate a structured critique.

B. Critique Structure

To prevent the critic from reinforcing its own biases, the critique is structured into three fields:

Assertion: Explicitly restates the correct answer and judges the PA's initial prediction.
Rationale (Local): An instance-specific explanation of why the correct answer is right and the PA's answer was wrong.
Reflection (Global): A generalized insight applicable to similar future questions.

C. Memory Architecture

The framework utilizes two distinct memory types to store these critiques:

Episodic Memory (Instance-Level):
- Stores specific past experiences (Question + Critique pairs).
- Uses Retrieval-Augmented Generation (RAG) to retrieve the top $K=5$ most similar past instances based on semantic embeddings.
- Provides context-specific reasoning and "revisits" specific failure/success scenarios.
Semantic Memory (Task-Level):
- Distills all critiques from a dataset into a unified, abstract knowledge representation (e.g., a bulleted list of rules or advice).
- Serves as a high-level guide or "guardrail" for the agent, summarizing generalizable patterns.

D. Learning Strategies Evaluated

The paper compares several strategies:

Zero-Shot: No memory.
EP_LABEL: RAG with only raw question-answer pairs (standard few-shot).
EP_CRIT: RAG with episodic memory containing full critiques.
SEM_CRIT: Prompt augmentation with semantic memory (summary of critiques).
EP+SEM_CRIT: Hybrid approach combining both episodic examples and semantic summaries.

3. Key Contributions

Memory-Augmented Reflective Learning: A novel framework that separates learning into episodic (specific examples) and semantic (general rules) memories, grounded in label-driven self-critiques.
The "Suggestibility" Metric: A new metric ( $S$ ) introduced to quantify how receptive a model is to external reasoning. It measures the performance difference when an agent follows truthful vs. misleading critiques. This explains why some models benefit more from memory augmentation than others.
Efficiency in Reasoning Models: The discovery that pre-computed critiques significantly reduce inference-time computation (thinking tokens) for reasoning models by offloading the reasoning process to the retrieved memory.
Comprehensive Empirical Analysis: Evaluation across 7 diverse datasets (fact-based and preference-based) and 6 different LLMs (including non-reasoning and reasoning models like GPT-5 and Qwen3-Thinking).

4. Key Results

Performance Gains:
- The hybrid EP+SEM_CRIT strategy achieved an average improvement of 8.1 percentage points over the zero-shot baseline and 4.6 percentage points over the strong RAG-based (EP_LABEL) baseline.
- Gains were most pronounced on preference-based tasks (e.g., Book/Movie preferences) where models lack pre-trained parametric knowledge.
- PubMed (highly specific factual knowledge) showed the least improvement, suggesting that memory augmentation struggles when cross-instance transfer of specific facts is limited.
Model Variance & Suggestibility:
- Suggestibility correlates positively with performance gains (Spearman $\rho = 0.402$ ). Models with low suggestibility (e.g., Llama 4 Scout) showed limited gains, while highly suggestible models (e.g., GPT-5, Qwen3) benefited significantly.
- High suggestibility presents a safety trade-off: models may be overly deferential to incorrect contextual information.
Computational Efficiency:
- For reasoning models, using critiques reduced thinking tokens by an average of 31.95%.
- Critiques act as a substitute for internal reasoning, preventing "overthinking" and reducing token timeouts (e.g., on the Multi-Condition Ranking task, token timeouts dropped from 60% to 12% for GPT-oss-20B).
Cost-Benefit:
- While generating memory incurs a one-time training cost, the inference-time savings (fewer tokens generated) lead to a "break-even" point within a modest number of queries for most datasets, making it cost-effective for production.

5. Significance and Implications

Lightweight Adaptation: This approach offers a computationally efficient, interpretable alternative to fine-tuning, allowing agents to adapt to new tasks instantly without retraining.
Interpretability: Unlike black-box weight updates, the memory artifacts (critiques and summaries) are human-readable, providing transparency into why an agent makes a decision.
Efficiency for Reasoning Models: The finding that external critiques can reduce internal reasoning load is crucial for deploying large reasoning models in cost-sensitive environments.
Future Directions: The work highlights the importance of "suggestibility" in agent design and suggests that future systems must balance the ability to learn from feedback with robustness against adversarial or erroneous inputs.

In conclusion, the paper demonstrates that reflective learning via structured memory is a powerful mechanism for enhancing LLM adaptability, offering significant gains in accuracy and efficiency while providing a clear path for interpretability and continuous learning.