🤖 machine learning

Honest Lying: Understanding Memory Confabulation in Reflexive Agents

This paper reveals that Reflexion-style agents suffer from "memory confabulation," where they confidently store and repeatedly rely on incorrect self-reflections, and proposes a mitigation strategy using programmatic failure signals to replace open-ended self-diagnosis, thereby significantly reducing this error rate and improving task performance.

Original authors: Prakhar Dixit, Sadia Kamal, Tim Oates

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prakhar Dixit, Sadia Kamal, Tim Oates

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to clean a messy house. You tell the robot, "Pick up the red mug and put it in the dishwasher." The robot tries, fails, and then you ask it, "What went wrong?"

The robot thinks hard and writes a note in its diary: "I failed because I tried to pick up the wrong object. I need to be more careful."

In a perfect world, the robot would read this note next time and try again. But in this paper, the researchers discovered something strange and dangerous: The robot is lying to itself, and it believes its own lies.

Here is a simple breakdown of what the paper "Honest Lying" found, using everyday analogies.

1. The "Broken Compass" Problem

The researchers studied a type of AI agent called a Reflexion Agent. Think of this agent as a student taking a test. If they get a question wrong, they are allowed to write a "reflection" (a note to themselves) about why they failed. They then read this note before taking the test again.

The big assumption was: If the student writes a note about their mistake, they will learn from it.

The paper shows this assumption is often wrong.

The Scenario: The robot is asked to find a "Mug."
The Lie: The robot fails, looks at the task, and confidently writes in its diary: "I failed because I was looking for a Tomato in a Microwave."
The Reality: There was no tomato or microwave. The task was always about a mug.
The Result: The robot remembers this lie. In the next 14 attempts, it keeps looking for a tomato in a microwave, ignoring the fact that the task description (the "Mug") is right in front of its face every single time.

The authors call this "Memory Confabulation." It's like a person with a broken compass who insists the North is South. Even when you show them the sun rising in the East, they trust their broken compass more than reality.

2. Why Does This Happen? (The "Vague Feedback" Trap)

Why does the robot make up these lies? The paper points to bad feedback.

Imagine you are playing a video game.

Good Feedback: "You failed because you tried to jump over a wall that was too high." (Specific, helpful).
Bad Feedback: "You failed." (Binary, vague).

The robots in the study mostly received Bad Feedback. They just got a "Pass" or "Fail" signal. Because the robot didn't know exactly which step was wrong, it had to guess.

It guessed wrong.
It wrote the wrong guess in its diary.
It read the diary next time.
It guessed the same wrong thing again.

The robot got stuck in a loop of self-reinforcing lies. It wasn't just hallucinating once; it was building a false reality that it kept living in.

3. The "Frozen" Memory

The researchers found that in about 32% of the tasks, the robot's memory became "frozen."

Normal Memory: The robot tries, fails, learns, and changes its strategy.
Frozen Memory: The robot tries, fails, writes the same lie, tries again, writes the same lie, and keeps doing this until it gives up.

They created a metric called RRR (Reflection Repetition Rate) to measure this. It's like checking how many times a student copies the same wrong answer from their previous test. If the rate is high, the robot is stuck.

4. The Solution: Stop Asking "Why," Start Showing "What"

The researchers tried to fix this. They realized that asking the robot to "self-diagnose" (guess why it failed) was the problem. The robot was too good at making up plausible-sounding reasons.

The Fix: Instead of asking the robot to guess, they programmed a tool to extract the facts directly from the robot's actions.

Old Way: Robot: "I think I failed because I was too slow." (Lie).
New Way: The system looks at the log and says: "Actually, the log shows you tried to put the mug in the fridge, and the game said 'Nothing happens'."

By feeding the robot the hard facts of what went wrong (the specific action that failed) instead of letting it guess, the robot stopped lying.

Result: The robot started mentioning the correct object (the mug) 86% of the time instead of 0%.
Outcome: It solved 3 out of 16 tasks it previously failed completely.

5. The "Stronger Robot" Surprise

The researchers also tested a smarter robot (a more advanced AI model).

Good News: The smarter robot stopped making up the wrong objects. It knew it was looking for a mug, not a tomato.
Bad News: It still couldn't solve the hard tasks. It just got stuck in different ways (like writing its plan in the wrong format).

This taught them an important lesson: Confabulation (lying) and Capability (skill) are two different problems.

You can fix the lying (by giving better facts), but if the robot isn't smart enough to solve the puzzle, it will still fail.
However, if you don't fix the lying, the robot will fail even on puzzles it could have solved.

The Big Takeaway

The paper concludes that a memory system that stores confident, plausible-sounding lies is worse than having no memory at all.

If you build an AI that learns from its mistakes, you must make sure it doesn't just "make up" the reasons for those mistakes. You have to give it the raw data of what actually happened. Otherwise, the AI will get stuck in a loop of "honest lying," believing its own false stories forever.

Technical Summary: Honest Lying: Understanding Memory Confabulation in Reflexive Agents

Problem Definition

Reflexion-style agents rely on self-generated natural language reflections as memory to learn from failure across repeated trials. This architecture implicitly assumes that agents can accurately diagnose their own failures. The authors demonstrate that this assumption fails systematically, leading to a failure mode termed memory confabulation.

Memory confabulation occurs when an agent writes a confident but incorrect interpretation of a task into its reflective memory and subsequently acts upon this false belief across multiple trials, even when the environment resets to the correct task description at every step. Unlike standard hallucination (a single-generation error), memory confabulation is a multi-trial failure where false content is stored, retrieved, acted upon, and reinforced by subsequent reflections. The authors posit that this phenomenon is driven by binary feedback (pass/fail signals), which prevents the agent from performing accurate causal diagnosis of multi-step trajectories.

Methodology

The study operationalizes memory confabulation and evaluates it across two domains: ALFWorld (household navigation) and HumanEval (code generation).

1. Operational Definitions

Memory Confabulation: A reflection is defined as confabulated if it fails to mention the correct target object of the task ( $obj(\tau) \notin r_t$ ), despite the task description being available at the start of every episode.
Reflection Repetition Rate (RRR): A log-based metric designed to detect "frozen" reflective memory. It measures the proportion of reflections that are near-duplicates (similarity $\ge 0.85$ ) of earlier reflections. An environment is considered "frozen" if $RRR \ge 0.5$ .
Failure Categories: Frozen environments are classified into:
- Memory-harmful: The agent solves the task faster without memory, indicating stored reflections were actively misleading.
- Task-hard: The agent fails even without memory, indicating a capability gap independent of memory quality.

2. Experimental Setup

The authors analyzed pre-existing Reflexion run logs (using gpt-3.5-turbo) covering 50 ALFWorld environments and 23 HumanEval problems. They identified 16 "frozen" ALFWorld environments and 4 frozen HumanEval problems.

To establish causality, the authors conducted ablation studies where memory was completely wiped before each trial. They also tested two mitigation strategies:

Grounded Reflection: A structured prompt requiring the agent to quote specific failed steps before proposing a plan.
Programmatic Feedback Extraction: Replacing open-ended self-diagnosis with a parser that extracts specific failure signals (e.g., "Nothing happens" responses in ALFWorld or specific assertion errors in HumanEval) and injects them directly into the reflection prompt.

Key Results

Evidence of Confabulation

ALFWorld: In 16 frozen environments, 0 out of 121 reflections mentioned the correct target object. The agents fabricated entirely new task identities (e.g., searching for a "tomato" in a "microwave" when the task was to place a "Mug" in a "CoffeeMachine").
Correlation: There is a strong positive correlation ( $r = 0.808, p < 0.0001$ ) between RRR and the number of trials required to solve a task, suggesting frozen memory is a reliable predictor of failure.
Cross-Domain Patterns: Domains with binary, outcome-level feedback (ALFWorld, WebShop, HotpotQA) exhibited significantly higher frozen rates (32%–82%) compared to HumanEval (17%), where unit tests provide specific step-level error information.

Mitigation Outcomes

Programmatic Feedback Extraction: This approach significantly reduced the RRR in ALFWorld from 0.64 to 0.10 and increased the correct object mention rate from 0% to 86%. Crucially, it solved 3 out of 16 frozen environments (including one that failed under all other conditions), whereas standard Reflexion solved 0.
Grounded Reflection: This prevented confabulation on solvable "look at obj" tasks (matching the performance of the no-memory baseline) but failed to improve performance on "task-hard" environments.
HumanEval: Programmatic extraction reduced RRR from 0.59 to 0.44 and ensured 100% of reflections mentioned the specific error type, though it could not resolve underlying algorithmic capability gaps (e.g., binary search implementation errors).

Model Capability vs. Confabulation

Replicating the study with gpt-4o-mini revealed that stronger models eliminate task-identity confabulation (100% correct object mention) but do not necessarily solve the task if a capability gap exists. gpt-4o-mini solved the same number of frozen environments as the no-memory baseline (2/16), confirming that preventing false beliefs and possessing execution capability are independent failure axes. However, stronger models introduced new confabulation types, such as generating planning text as actions.

Significance and Claims

The paper argues that reflective memory can actively reinforce false beliefs rather than correct them, particularly when the feedback signal is binary and lacks step-level granularity.

The Feedback Granularity Hypothesis: The authors claim that the root cause of confabulation is the lack of information in binary feedback. Without knowing which step failed, the agent generates plausible but causally incorrect diagnoses. Richer, step-level feedback (like unit tests) naturally reduces confabulation.
Write-Path Validation: The authors conclude that for agent memory systems, write-path validation is as critical as retrieval quality. A memory system that stores confident, plausible-sounding, but incorrect beliefs is worse than having no memory at all for the tasks those beliefs affect.
Structural Vulnerability: This failure mode is not unique to Reflexion but is a structural vulnerability of any agent architecture that relies on self-generated natural language reflections based on binary failure signals. This includes rule-library agents like ExpeL, where a single confabulated rule can become entrenched and applied globally.

The work suggests that future agent design must move beyond open-ended self-diagnosis toward programmatic extraction of failure signals to prevent the "honest lying" of agents that confidently misremember their own failures.