🤖 machine learning

When Generator Replay Degrades: Projected Rehearsal Orchestration for Heterogeneous Federated Class-Incremental Learning

This paper introduces PRO and PRO-MAX, a server-light framework for heterogeneous federated class-incremental learning that replaces fragile input-space generator replay with projected rehearsal orchestration to effectively mitigate representation drift and supervision imbalance across diverse clients and modalities.

Original authors: Thinh T. H. Nguyen, Khoa D. Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong

Published 2026-06-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Thinh T. H. Nguyen, Khoa D. Doan, Binh T. Nguyen, Danh Le-Phuoc, Kok-Seng Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Global Classroom with a Broken Chalkboard

Imagine a massive, global classroom where students (clients) are learning new subjects (classes) over time. They can't share their personal notebooks (raw data) with the teacher (server) because of privacy rules. Instead, they send their homework updates to the teacher, who combines them to create a "Global Brain" that knows everything.

The Problem:
In the real world, this classroom is chaotic.

Different Paces: Student A is learning about "Cats" while Student B is already on "Dogs."
Uneven Help: Student C only has pictures of "Siamese cats," while Student D has pictures of "Persian cats."
The Forgetting Curse: As the Global Brain learns new things, it starts forgetting the old things (like how to recognize a cat once it learns about dogs).

The Old Solution (Generator Replay):
To stop forgetting, previous methods tried to use a "Magic Photocopier" (a Generator). When the brain needed to remember "Cats," the photocopier would try to create fake cat pictures from scratch based on what it remembered.

The Flaw: If the student who taught the brain about cats was confused or had bad notes (weak supervision), the photocopier would create bad fake cats. When the brain studied these bad fake cats later, it got even more confused. It's like trying to learn French from a dictionary written by someone who barely speaks French.

The New Solution (PRO & PRO-MAX):
The authors propose a new way to remember things that doesn't rely on creating fake pictures. Instead, they use Projected Rehearsal.

Part 1: PRO (The "Summary Note" Approach)

Instead of trying to recreate the exact photo of a cat (input space), PRO asks the students to send in a summary note (projected memory) of what a cat looks like in their minds.

The Analogy: Imagine instead of sending a photo of a cat, the student sends a small card that says: "A cat has pointy ears, whiskers, and a tail. The average cat is about this size."
How it works:
1. The Server's Library: The teacher keeps a library of these summary cards for every animal learned so far. These cards are tiny and easy to send over the internet.
2. The Rehearsal: When a student is learning about "Dogs," the teacher sends them the "Cat Summary Card." The student doesn't look at a fake cat photo; they look at the mathematical description of a cat.
3. The Benefit: Even if the student's notes on cats were a little shaky, the summary card is a stable, compact representation. It doesn't matter if the student is learning cats or dogs; the summary card works for everyone. It's like using a universal translation of a concept rather than trying to draw the concept perfectly.

Part 2: PRO-MAX (The "Moving Target" Fix)

There is one problem with the summary cards: As the students learn more, their way of thinking changes. A "Cat" in the brain of a student who just learned about "Lions" might look slightly different than a "Cat" in the brain of a student who just learned about "Fish." The summary card might become slightly outdated (stale).

PRO-MAX adds a smart adjustment mechanism.

The Analogy: Imagine the "Cat Summary Card" is a sticky note on a wall. As the student learns new things, the wall itself shifts slightly. The sticky note might end up in the wrong spot.
How it works:
1. The Calibration: Before the student starts their new lesson, they take a quick snapshot of their current thinking (a "calibration subset").
2. The Shift: They compare their old thinking to their new thinking. They calculate exactly how much the "Cat" concept has moved or shifted in their mind.
3. The Alignment: They tell the teacher, "Hey, the Cat card has shifted 5 inches to the left." The teacher then moves all the Cat cards in the library to match this new position.
The Result: The summary cards stay perfectly aligned with the students' current way of thinking, preventing the brain from getting confused by outdated information.

Why This Matters (The "Weak Task" Discovery)

The paper discovered a critical flaw in the old "Magic Photocopier" method.

The Scenario: If a student learns a topic poorly (e.g., they only see 3 bad pictures of a "Zebra"), the Photocopier will generate terrible fake Zebras.
The Consequence: When the brain tries to learn the next topic, it studies these terrible fake Zebras, which poisons the learning process. The more fake Zebras you study, the worse you get at recognizing real animals later.
The PRO Advantage: Because PRO uses summary cards (statistics) instead of fake photos (generations), it is much more robust. Even if the initial notes were a bit weak, the summary card is a stable average. It doesn't amplify the errors the way a photocopier does.

The Results in Plain English

The researchers tested this on images (like CIFAR-100), text, and graphs (like social networks).

Better Memory: PRO and PRO-MAX forgot less than the old methods, especially when students were learning at different speeds or had different amounts of data.
Cheaper: The "summary cards" are much smaller to send over the internet than the huge files needed for "fake photo" generators.
More Robust: Even when they gave the old methods more fake photos to study (a bigger budget), those methods still failed in chaotic classrooms. This proves that quality of memory matters more than quantity.

Summary of the Metaphor

Old Method (Generator): Trying to remember a friend by trying to paint a perfect portrait of them every time you see them. If you had a bad day when you first met them, your painting is wrong, and you keep painting over the wrong picture.
New Method (PRO): Remembering a friend by keeping a small index card with their key features (height, hair color, job). It's fast, easy to share, and hard to mess up.
New Method Plus (PRO-MAX): Realizing that your friend's style changes over time, so you update the index card to reflect their new haircut or job title, ensuring the note is always accurate.

The paper concludes that by switching from "painting fake pictures" to "updating summary notes," we can build smarter, more private, and more stable AI systems that learn continuously without forgetting.

Technical Summary: Projected Rehearsal Orchestration for Heterogeneous Federated Class-Incremental Learning

1. Problem Definition

The paper addresses Federated Class-Incremental Learning (FCIL), a setting where a global model must learn new classes over time while retaining knowledge of previous classes, all under privacy constraints that prevent centralizing raw data. The study specifically targets heterogeneous FCIL, a realistic but challenging scenario where clients exhibit:

Supervision Imbalance: Different clients observe different subsets of labels for the same semantic task or provide uneven sample counts.
Stage Misalignment: Clients progress through the sequence of semantic tasks at different rates and times (i.e., $\pi_c[p] \neq \pi_{c'}[p]$ ).

Existing FCIL methods often rely on generator-based replay or data-free synthesis to preserve old knowledge by generating artificial input-space samples. The authors argue that under heterogeneous streams, these methods become fragile for two primary reasons:

Error Compounding: If a task is learned weakly due to sparse or biased supervision, the generated replay signals derived from that state are unreliable. Reusing these poor signals in later stages causes downstream degradation.
Modality Coupling: Input-space generators are tightly coupled to specific data types (e.g., images), making them difficult to transfer across modalities (e.g., text or graphs) without re-engineering the generator architecture.

2. Methodology

The authors propose PRO (Projected Rehearsal Orchestration), a generator-free framework that replaces input-space synthesis with compact projected memories stored in the feature space.

Core Architecture

The model is decomposed into:

$h_\theta$ : A modality-specific base encoder.
$a_\psi$ : A lightweight adapter.
$W$ : A unified classifier head.
The system operates under a single-head protocol where task identity is not provided at inference.

The PRO Framework

Shared Warmup: All methods undergo a federated in-domain self-supervised warmup to initialize $h_\theta$ without external pretraining, ensuring fair comparison.
Projected Memory Storage: The server maintains compact class-level statistics for each known class $y$ : the mean ( $\mu_y^u$ ) and diagonal standard deviation ( $\sigma_y^u$ ) of the base features $u = h_\theta(x)$ . No raw data or input-space synthetic samples are stored.
Client-Side Rehearsal:
- Controlled Plasticity: Clients first perform a short training phase on current-task data with a proximal regularizer to prevent excessive drift from the global model.
- Balanced Pseudo Multi-Task Training: Clients sample "old" pseudo-features from the server's projected memories ( $\tilde{u}_{y,j} = \mu_y^u + \gamma \sigma_y^u \odot \epsilon_j$ ) and combine them with real current-task features.
- Integration Point: Crucially, these old projected features are fed before the adapter ( $a_\psi$ ). This ensures that gradients from old classes update both the adapter and the classifier, preventing the adapter from drifting solely toward the current task.
Server Aggregation: The server aggregates model updates (FedAvg) and refreshes memory statistics using the aggregated global model to ensure consistency in the feature space.

PRO-MAX: Memory Alignment Extension

To handle stronger representation drift in heterogeneous settings, PRO-MAX introduces a memory alignment mechanism:

Local Transport Estimation: Clients cache a small calibration subset before and after local updates to estimate feature drift ( $\delta = u^+ - u^-$ ).
Neighborhood-Weighted Alignment: Clients compute a weighted transport vector for old memories based on nearest neighbors in the calibration set.
Confidence-Weighted Aggregation: The server aggregates these transport vectors using confidence scores (based on how close the client's current data is to the old class). This shifts the old projected memories to align with the evolving representation space without requiring server-side gradient descent.

3. Key Contributions

Identification of Generator Fragility: The paper identifies that generator-based replay fails under heterogeneous FCIL not just due to resource limits, but because weak task learning produces poor synthetic signals that compound errors downstream.
Modality-Agnostic Memory Interface: PRO introduces a generator-free framework using compact class-level projected statistics, enabling seamless transfer across images, text, and graphs without modality-specific generators.
Robustness to Heterogeneity: By performing balanced pseudo multi-task learning on clients and maintaining server-light memory statistics, PRO and PRO-MAX achieve superior retention and utility under supervision imbalance and stage misalignment.
Memory Alignment: PRO-MAX demonstrates that explicitly aligning old memories with representation drift via confidence-weighted transport significantly reduces forgetting in heterogeneous streams.

4. Experimental Results

The methods were evaluated on CIFAR-100 (image), THUCNews-10 (text), and Cora (graph) under both homogeneous and heterogeneous protocols.

Heterogeneous Performance: Under severe heterogeneity, generator-based baselines (TARGET, MFCL, HR) degraded significantly even when given expanded replay budgets. PRO and PRO-MAX maintained high Final Average Accuracy (FAA) and low Average Forgetting (AF). For example, on CIFAR-100, PRO-MAX achieved 46.12% FAA compared to 27.86% for TARGET.
Modality Transfer: The approach successfully transferred to text and graph benchmarks, outperforming modality-specific baselines (e.g., FedSeIT for text, POWER/MOTION for graphs) while maintaining lower communication costs.
Resource Efficiency: PRO and PRO-MAX operate with a fixed projected-memory budget (comparable to a single class of features), whereas generator-based methods often require significantly higher communication overhead to transmit synthetic samples or larger replay buffers.
Generator Failure Diagnostics: Experiments showed a strong positive correlation between replay-real mismatch (measured by MMD) and downstream degradation. Methods with higher mismatch (generators) suffered more from weak-task interventions, whereas PRO/PRO-MAX maintained lower mismatch and less downstream drop.

5. Significance and Claims

The paper claims that replay quantity alone does not resolve replay-quality failures. In heterogeneous federated settings, the quality of the replay signal is contingent on the quality of the underlying task learning. When supervision is weak or misaligned, input-space generators amplify these errors.

The significance of the work lies in shifting the memory abstraction from input-space synthesis to feature-space projection. This approach:

Decouples memory mechanisms from specific data modalities.
Avoids the computational and communication costs of generating and transmitting synthetic inputs.
Provides a server-light solution where the server performs only aggregation and memory accounting, avoiding server-side gradient-based optimization or generator training.

The authors modestly acknowledge limitations: projected memories depend on the quality of the learned representation and may be noisy for extremely rare classes with sparse support. They also note that while the method reduces privacy risks associated with raw data storage, it does not offer formal privacy guarantees against representation inversion without additional mechanisms like differential privacy.