💻 computer science

Amnesia: A Stealthy Replay Attack on Continual Learning Dreams

This paper introduces Amnesia, a stealthy replay attack on continual learning systems where a limited-privilege insider manipulates only replay sample selection to maximize performance degradation while remaining within auditable statistical constraints, thereby exposing a practical threat surface in index-controlled learning pipelines.

Original authors: Ahmed Sharshar, Naveen Kumar Kummari, Mohsen Guizani

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Sharshar, Naveen Kumar Kummari, Mohsen Guizani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Robot That Forgets

Imagine a robot learning to do new jobs one by one. First, it learns to sort red blocks. Then, it learns to sort blue blocks. Finally, it learns to sort green blocks.

The problem is Continual Learning: When the robot learns to sort green blocks, it often accidentally "overwrites" its memory of how to sort red and blue blocks. This is called Catastrophic Forgetting.

To fix this, engineers use a trick called Experience Replay. The robot keeps a small "notebook" (a buffer) of examples from its past jobs. Every time it learns something new, it also flips through a few pages of its old notebook to remind itself of the past. This keeps the old memories alive.

The Attack: The "Dream" Saboteur

The paper introduces a new type of attack called Amnesia.

The Setup:
Imagine the robot's notebook is managed by a librarian. The librarian's job is to pick which pages (examples) to show the robot during its "rehearsal" sessions.

The Attacker: A "grey-box insider" (someone with limited access) who controls the librarian.
The Limitation: The attacker cannot change the pictures in the notebook, cannot change the robot's brain (weights), and cannot change the robot's current lesson. They can only decide which pages the librarian pulls out to show the robot.

The Goal:
The attacker wants the robot to forget its old jobs (red and blue blocks) as much as possible, while making it look like everything is normal.

How It Works: The "Tilt and Project" Strategy

The attacker uses a two-step process to manipulate the robot's memory without getting caught by the boss (the auditor).

1. The "Tilt" (Choosing the Poison)

The attacker looks at the notebook and calculates which past examples are most dangerous to the robot's memory.

Analogy: Imagine the robot is learning to drive. The attacker notices that if the robot practices "parking" too much, it forgets how to "drive on the highway." So, the attacker decides to tilt the selection toward "parking" examples.
The Math: They assign a "harm score" to every class of data. They want to show the robot more of the "harmful" classes and less of the "safe" ones.

2. The "Project" (The Stealth Mask)

Here is the tricky part. If the attacker just shows the robot only parking examples, the boss (auditor) will immediately notice: "Hey, the notebook usually has a mix of driving, parking, and turning. Why is it all parking today?"

To avoid detection, the attacker must stay within strict Auditable Budgets:

The Mass Budget: The total number of pages shown must stay the same.
The Visibility Budget: The mix of pages must look statistically similar to the normal mix. If the normal mix is 50% driving and 50% parking, the attack can't change it to 90% parking. It can only nudge it slightly, like 55% parking and 45% driving.

The Solution:
The attacker uses a mathematical "projector" to take their harmful plan and squash it back into the "safe zone."

They calculate the perfect, slightly dangerous mix that is just barely close enough to the normal mix to fool the auditor's logs.
They use two methods:
- KL (Kullback-Leibler): A smooth, gentle nudge. It's very hard to detect but slightly less damaging.
- TV (Total Variation): A sharper, more aggressive nudge. It causes more damage but is easier for the auditor to spot if they look closely.

The Result: "Amnesia"

When the robot trains with this manipulated selection:

It rehearses the "harmful" examples just enough to confuse its memory of the old tasks.
It rehearses the "safe" examples just enough to pass the auditor's check.
The Outcome: The robot ends up with Amnesia. It forgets the old tasks (like sorting red blocks) much faster than usual, even though the auditor's logs show that the "mix of pages" looked perfectly normal.

Why This Matters

The paper proves that you don't need to hack the robot's brain or poison the data files to break a learning system. You just need to control which old memories are brought up for review.

Real-world analogy: Imagine a student studying for a history exam. The teacher (the system) gives them a list of topics to review. If a saboteur (the attacker) subtly changes the list so the student reviews "World War II" 10% more often and "The Renaissance" 10% less often, the student might fail the Renaissance section later. If the saboteur keeps the total number of review topics the same and the list looks mostly normal, the teacher won't notice the sabotage until the student fails the test.

Key Takeaways from the Experiments

It Works: The attack successfully made robots forget old tasks across many different datasets (like CIFAR-10, CORe50, and Tiny-ImageNet).
It's Stealthy: The "KL" version of the attack was so subtle that standard checks (looking at the logs of what was reviewed) didn't catch it.
It's Fast: The attack adds almost no extra time to the training process.
The Trade-off: The more damage the attacker wants to do, the harder it is to stay hidden. The "TV" method caused more damage but was more likely to trigger an alarm.

Summary

Amnesia is a "sneaky librarian" attack. By carefully choosing which past memories to rehearse—just enough to break the robot's brain but not enough to look suspicious—the attacker can make a learning system forget its past, all while staying within the rules of the audit logs.

Technical Summary: Amnesia – A Stealthy Replay Attack on Continual Learning

Problem Statement

Continual Learning (CL) systems rely on experience replay to mitigate catastrophic forgetting, typically by interleaving a small memory of past examples with current data. While existing security research in CL has focused on poisoning (modifying inputs to degrade performance) and backdoors (embedding triggers for targeted failures), these approaches often lack explicit, auditable constraints found in monitored production environments.

In deployed CL pipelines, replay samplers are often instrumented for compliance, logging telemetry such as replay index sets, per-batch label counts, and replay rates. These logs enable "auditable budgets" where monitors verify that the realized replay class histogram remains close to a nominal baseline and that the replay rate is unchanged. The paper identifies a critical robustness gap: the vulnerability of CL systems to a limited-privilege insider who controls only the replay index selection (the "dreaming" stage) without access to model parameters, pixel data, or labels. This attacker aims to maximize forgetting of past tasks while strictly adhering to these auditable visibility and mass constraints, a threat surface previously unexplored under such strict constraints.

Methodology: The Amnesia Attack

The authors introduce Amnesia, a sampler-level replay composition attack that biases which indices are drawn from the replay buffer. The attack operates under two explicit constraints:

Visibility Budget ( $\delta$ ): The divergence (Total Variation or Kullback-Leibler) between the attacked replay histogram and a nominal baseline ( $p_0$ ) must not exceed a threshold.
Mass Budget ( $f$ ): The total replay rate (keep fraction) remains fixed.

The attack follows a two-step "tilt-then-project" procedure:

1. Preference (Tilt)

The attacker utilizes lightweight, sampler-readable utility signals (e.g., Exponential Moving Average (EMA) of cross-entropy loss or confidence scores logged as metadata) to identify "harmful" classes—those whose over-representation maximizes forgetting. These utilities ( $u$ ) are used to tilt the nominal class histogram $p_0$ toward a target distribution $\tilde{p}$ that favors high-utility (harmful) classes.

2. Projection (Stealth)

The tilted distribution $\tilde{p}$ is projected back into the auditable "stealth ball" defined by the divergence constraint $\delta$ . The paper provides exact, efficient optimizers for two divergence types:

KL Divergence: Uses a single-parameter exponential tilt ( $p^*(c) \propto p_0(c) \exp(\alpha u_c)$ ) with a monotone 1-D root search to find the multiplier $\alpha$ that saturates the budget. This preserves positive mass across all classes, resulting in smoother discretization.
Total Variation (TV) Divergence: Uses a greedy two-sided water-filling algorithm that transfers probability mass from low-utility to high-utility classes until the $\ell_1$ budget is exhausted. This can push low-utility classes toward zero, creating sparser distributions.

3. Realization and Scheduling

The continuous optimal distribution $p^*$ is converted into integer quotas ( $q$ ) for the batch size $m$ . The system employs a windowed scheduler using a ring buffer to enforce a tightened per-step budget ( $\delta_t$ ), ensuring that the average divergence over any rolling window $W$ remains within the global limit $\delta$ . This handles the discretization noise introduced by rounding quotas to integers.

Key Contributions

Amnesia Attack: The first sampler-level replay composition attack that manipulates index selection to maximize forgetting while remaining compliant with logging-based audits (rate and histogram checks). It leaves pixels, loss computations, and model parameters untouched.
Principled Formulation: A formal optimization framework maximizing harm ( $\langle u, p \rangle$ ) subject to auditable constraints ( $Div(p \| p_0) \le \delta$ and fixed mass $m$ ). The authors prove that their projection step is the exact best-response class composition, weakly dominating any other feasible sampler-level attack under equivalent constraints.
Audit Realization: A practical pipeline converting theoretical distributions into integer quotas with "audit-and-fix" mechanisms and a windowed scheduler to guarantee compliance with both per-batch and rolling-window audit checks.
Comprehensive Evaluation: Systematic analysis of the trade-off between impact (forgetting), visibility (divergence), and budget across multiple datasets (Split CIFAR-10/100, CORe50, Tiny-ImageNet) and strong replay baselines (ER, ER-ACE, SCR, DER++).

Experimental Results

The authors evaluated Amnesia on four CL benchmarks against strong replay baselines:

Impact: Amnesia consistently depresses final accuracy (ACC) and significantly worsens backward transfer (increases $-BWT$). For example, on Split CIFAR-10 with ER, the attack reduced ACC from 50.4% to 29.3% while increasing forgetting ($-BWT$) from 55.1 to 82.7.
Stealth vs. Impact Trade-off:
- The KL variant achieves high impact while remaining largely undetected by audits, with low tail divergence ( $r_{batch@95}$ ) and zero window violations. Its exponential tilt distributes mass more smoothly, avoiding the "chunkiness" that triggers audit flags.
- The TV variant is often more damaging (higher forgetting) but more easily detected, particularly under tight per-class constraints, due to its tendency to create sparse, extreme allocations that are harder to realize under integer quotas.
Comparison to Poisoning: Despite operating under strict audit-aware constraints and having no access to model internals or data pixels, Amnesia achieves comparable or superior forgetting levels compared to data-poisoning baselines like BrainWash and PACOL, highlighting index-only control as a potent threat surface.
Robustness: The attack remains effective across different backbone architectures (ResNet-18, ViT-Tiny, ConvNeXt-Base) and replay methods, though methods like ER-ACE show slightly higher robustness.
Overhead: The attack introduces minimal computational overhead (approx. 3.5% for KL and 7.1% for TV) relative to the baseline training time.

Significance and Claims

The paper claims to expose index-only replay control as a practical and dangerous threat surface in Continual Learning systems. By formalizing the trade-off between impact and visibility under auditable budgets, the authors demonstrate that an insider with limited privileges (sampler access only) can significantly degrade model performance without triggering standard compliance checks.

The work establishes a principled impact-visibility-budget trade-off, showing that maximizing harm under strict divergence constraints is an exact optimization problem rather than a heuristic. The authors argue that current CL defenses are insufficient against such "stealthy" composition attacks, as they often assume threats involve data corruption or parameter manipulation, overlooking the subtle manipulation of sampling distributions. The results suggest that monitoring replay statistics alone may not be sufficient to detect sophisticated, divergence-constrained attacks, necessitating new defense mechanisms that account for the strategic manipulation of replay indices.