💬 NLP

ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

This paper introduces ForgetBench, a novel benchmark designed to systematically evaluate the long-term forgetting dynamics and retention stability of large language models under continual knowledge editing, revealing that current methods struggle to balance effective knowledge updates with the preservation of previously acquired information.

Original authors: Ruxi Gu, Zhenliang Zhang, Wei Wang

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruxi Gu, Zhenliang Zhang, Wei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your brain could learn a new fact every second, but every time you learned something new, you accidentally erased an old memory. This isn't just a sci-fi nightmare; it's a growing concern for the artificial intelligences we call Large Language Models (LLMs). These digital brains are incredibly good at reading books and having conversations, but they struggle with something humans do naturally: keeping old memories safe while making room for new ones. Scientists have long studied how human memory works, splitting it into "working memory" (what you're thinking about right now) and "long-term memory" (facts you've stored away for years). While we know how to teach these AI models new tricks, we don't fully understand how they handle the messy business of updating their internal knowledge over time without losing everything they knew yesterday. If we want AI to be a helpful lifelong companion rather than a forgetful friend who needs constant reminders, we need to figure out how to make their memories stick.

Enter ForgetBench, a new "stress test" designed by researchers to see exactly how and why AI models forget things when they are constantly being updated. Think of ForgetBench not as a simple quiz, but as a time-lapse camera for an AI's brain. Instead of just checking if an AI knows a fact right now, the researchers set up a scenario where the AI has to learn a stream of new information, one after another, like a student taking a new history test every day for a month. The big question they asked was: "If we teach this AI a new fact today, will it still remember the fact we taught it a week ago, or will the new lesson have overwritten the old one?"

To answer this, the team built two different kinds of memory challenges. The first, called Concept-based QA, is like a game of flashcards with isolated facts. Imagine you tell the AI, "Donald Rodriguez is 22 years old," and then later, "Actually, Donald is 79." The researchers wanted to see if the AI could swap the age without getting confused or forgetting other people's ages. The second challenge, Scenario-based QA, is much more like a complex role-playing game. Here, the AI has to track relationships between characters, like "Who followed Bill Gates and retweeted a post about space exploration?" This tests if the AI can hold onto a web of connected stories or if the new updates cause the whole story to collapse.

The researchers ran these tests on several different AI models using various methods to update their knowledge. What they found was a bit of a bummer for the current state of AI: existing methods are terrible at balancing the two. When the models learned new facts, they often forgot the old ones, or they became so confused that they started making up nonsense. The study measured this using something called a "forgetting curve," which tracks how quickly a memory fades as more new information is added. They found that for most current models, the "half-life" of a memory (the time it takes for the AI to forget half of what it learned) is very short. In other words, the AI is great at learning now, but it's terrible at remembering later.

The paper suggests that while we have gotten really good at teaching AI new things in a single session, we haven't solved the problem of how to keep those memories safe over a long period of constant updates. The researchers didn't just find a problem; they provided a new way to measure it, showing that the current tools for updating AI knowledge are like trying to write a new chapter in a book by erasing the previous pages. Until we can build better "memory mechanisms" that allow AI to grow without losing its past, these digital brains will remain brilliant but incredibly forgetful.

Technical Summary: ForgetBench

Problem Statement

Large Language Models (LLMs) have demonstrated significant capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains poorly understood. Existing evaluation paradigms primarily focus on working memory (e.g., in-context reasoning, multi-hop QA within a single inference) or static knowledge editing (evaluating single-step edit success, generalization, and locality). These approaches fail to capture the temporal dynamics of knowledge retention and degradation during continual model modification.

Current knowledge editing benchmarks (e.g., zsRE, UnKE) treat edits as independent operations, overlooking the cumulative interference caused by subsequent updates. Consequently, they cannot answer a fundamental question for reliable lifelong adaptation: whether knowledge acquired today can survive future modifications. Furthermore, existing long-term memory benchmarks often rely on externally maintained memory (retrieval-augmented generation), whereas this work focuses on intrinsic parametric memory, where forgetting emerges from cumulative interference within model parameters.

Methodology: ForgetBench

The authors propose ForgetBench, a unified benchmark designed to systematically characterize forgetting behavior in LLMs under continual knowledge editing. The methodology consists of three core components:

1. Data Generation Paradigms

ForgetBench introduces two complementary QA construction paradigms to disentangle isolated factual retention from structured relational knowledge preservation:

Concept-based QA: Focuses on atomic factual attributes. Each subject is associated with a single property and independently perturbed over time. This setting removes inter-instance dependencies, providing a controlled environment to analyze localized parameter interference.
Scenario-based QA: Constructs QA pairs over structured relational knowledge induced by a dynamic interaction graph. A heterogeneous knowledge graph is generated through simulated multi-agent interactions (agents, items, and edges). QA instances are derived from sampled subgraphs, requiring reasoning over interconnected entities. This evaluates structured knowledge understanding under continual editing.

2. Sequential Editing Framework

Unlike static benchmarks, ForgetBench employs a sequential editing paradigm. New knowledge is continuously injected over time through a temporally ordered knowledge stream. The model undergoes repeated editing stages, and performance is evaluated periodically on both newly edited instances and previously edited ones. This allows for the construction of forgetting curves that reflect memory preservation, degradation, or overwriting over time.

3. Unified Evaluation Framework

To quantitatively analyze long-term retention dynamics, the authors define a spatiotemporal evaluation matrix $Z[t, i]$ , where $t$ represents the editing round and $i$ the knowledge instance. Based on this, they introduce a set of metrics capturing three key dimensions:

Pointwise Editing Performance:
- Edit Success (ES): Measures immediate effectiveness of new edits.
- Retention (Ret): Evaluates the average preservation of all past knowledge.
Temporal Decay Dynamics:
- Forgetting Curve ( $F(\Delta)$ ): Models the relationship between memory age ( $\Delta$ , the number of subsequent edits) and retention.
- Memory Half-life ( $\Delta_{1/2}$ ): The number of editing steps required for retention to decay to half its initial level.
- Area Under the Forgetting Curve (AUF): Measures average retention over the knowledge's lifetime.
- Forgetting Speed: Quantifies the local variation rate of the curve.
Cross-Instance Stability:
- Temporal Consistency (TC): Quantifies the stability of individual memory trajectories across successive editing steps, distinguishing between stable degradation and abrupt memory disruption.
Generalization and Fluency:
- Gen: Measures generalization to rephrased queries.
- Flu: Monitors language generation stability to detect text degeneration or model collapse.

Key Results

Extensive experiments conducted across diverse LLMs and editing methods (including causal intervention, null-space constraints, hypernetworks, and locate-then-edit approaches) yield the following findings:

Imbalance in Performance: Existing approaches fail to strike a balance between long-term retention and generalization quality. Methods that achieve high immediate edit success often suffer from rapid temporal decay or poor generalization.
Structural Differences: The evaluation matrices reveal distinct spatiotemporal structures. Concept-based QA shows highly dispersed correctness (independent vulnerability to interference), while scenario-based QA exhibits vertical coherence, suggesting that contextual dependencies can induce more stable prediction trajectories.
Limitations of Current Paradigms: Current knowledge editing methods are insufficient for reliable lifelong adaptation, as they cannot effectively preserve knowledge against the cumulative interference of subsequent updates.

Significance and Claims

The paper claims that ForgetBench provides the first systematic analysis of long-term parametric forgetting dynamics in LLMs under continual knowledge editing. By moving beyond static, single-step evaluations, the benchmark enables the temporal analysis of knowledge acquisition, retention, and forgetting.

The authors emphasize that their findings highlight a critical gap in current LLM development: the lack of robust memory mechanisms capable of effectively acquiring, updating, and preserving knowledge over time. ForgetBench serves as a diagnostic tool to motivate future research toward more robust memory mechanisms that can sustain knowledge integrity in dynamic, continually updating environments. The work does not propose a new editing algorithm but rather establishes the necessary evaluation infrastructure to assess and improve such mechanisms.