C^2Prompt: Class-aware Client Knowledge Interaction for Federated Continual Learning

This paper proposes C²Prompt, a novel federated continual learning method that mitigates temporal and spatial forgetting by introducing a local class distribution compensation mechanism and a class-aware prompt aggregation scheme to enhance class-wise knowledge coherence across distributed clients.

The Gist

Imagine a massive, global school where students (the clients) are scattered all over the world, and a principal (the server) tries to teach them all at once. However, there are two big rules:

1. Privacy: Students cannot share their notebooks or homework with anyone; they must keep their data private.
2. Continual Learning: The curriculum changes every week. One week it's about cats, the next about cars, then about airplanes.

The goal is for the principal to create a "Super Brain" that knows everything about all these topics without forgetting the old ones, even though the students are learning different things at different times and in different ways.

This is the challenge of Federated Continual Learning (FCL). The paper you provided, C²Prompt, proposes a brilliant new way to solve the two biggest headaches in this scenario: Forgetting (forgetting old lessons) and Confusion (mixing up new lessons with old ones).

Here is how the paper solves it, using simple analogies:

The Problem: The "Noisy Classroom"

In previous methods, when the principal tried to combine the students' knowledge, two things went wrong:

1. The "Different Dialects" Problem (Intra-class Distribution Gap):
   Imagine the "Cat" lesson. Student A in Japan only sees fluffy Persian cats. Student B in Brazil only sees stray street cats. When they try to teach the principal what a "cat" is, their descriptions are so different that the principal gets confused. They can't agree on what a cat actually looks like.

   • Result: The "Cat" knowledge becomes weak and fuzzy.

2. The "Wrong Teacher" Problem (Inter-prompt Relevance):
   Imagine the principal is trying to combine notes on "Cats" and "Cars." Because the students are confused about what a "cat" is, they accidentally mix in notes about "cars" when talking about cats. The principal then tries to merge these messy notes, and the "Cat" knowledge gets contaminated by "Car" knowledge.

   • Result: The model forgets the old tasks because the new information is clashing with it.

The Solution: C²Prompt (The "Class-Aware" System)

The authors, Kunlun Xu and his team, invented a system called C²Prompt. Think of it as giving the students two special tools to fix the classroom chaos.

Tool 1: The "Universal Translator" (Local Class Distribution Compensation)

The Analogy:
Before the students start their weekly lesson, the principal sends them a "Global Cheat Sheet." This sheet says, "Okay, globally, a cat is 50% fluffy and 50% scruffy."

• How it works: If Student A (who only sees fluffy cats) gets this sheet, they realize, "Oh, I need to adjust my understanding to include scruffy cats too." They tweak their mental model to match the global average before they even start learning the new task.
• The Benefit: Now, when Student A and Student B talk to the principal, they are both speaking the same "Cat Language." The gap between their different dialects is closed.

Tool 2: The "Smart Librarian" (Class-Aware Prompt Aggregation)

The Analogy:
After the students finish their lessons, they send their notes back to the principal. In the old days, the principal just dumped all the notes into a pile and mixed them together (like a smoothie). If a student accidentally wrote about "Cars" in the "Cat" pile, the whole smoothie tasted like gasoline.

• How it works: The new system uses a Smart Librarian. Before mixing the notes, the librarian checks the "tags" on every page.
  • "Is this note about Cats?" -> Yes, put it in the Cat bin.
  • "Is this note actually about Cars?" -> No, keep it out of the Cat bin.
  • The librarian only mixes notes that are truly relevant to the specific topic.
• The Benefit: The "Cat" knowledge stays pure. It doesn't get contaminated by "Car" knowledge. This prevents the model from getting confused and forgetting what it learned yesterday.

The Result: A Super-Student

By using these two tools, the C²Prompt system achieves three amazing things:

1. It remembers everything: It doesn't forget old lessons (Temporal Forgetting) because the knowledge isn't getting mixed up.
2. It understands everyone: It handles students from different backgrounds (Spatial Forgetting) because the "Universal Translator" aligns their views.
3. It learns faster: Because the notes are clean and organized, the principal can learn new topics much more efficiently.

Why This Matters in the Real World

Imagine a hospital network where different hospitals are learning to diagnose new diseases.

• Privacy: They can't share patient records.
• Continual Learning: A new virus might appear tomorrow, and they need to learn it immediately without forgetting how to treat last year's flu.
• C²Prompt: This system ensures that Hospital A (in a rural area) and Hospital B (in a city) can learn together effectively, even if their patient populations look different, without mixing up the symptoms of the flu with the new virus.

In short: C²Prompt is like a super-organized study group that ensures everyone speaks the same language and only shares the right notes, resulting in a group that learns faster, remembers more, and never gets confused.

Technical Summary

1. Problem Statement

Federated Continual Learning (FCL) aims to enable distributed clients to learn from continuously emerging task streams while preserving data privacy. However, FCL faces a dual challenge:

1. Temporal Forgetting: The model forgets previously learned tasks as new ones are introduced.
2. Spatial Forgetting: The model struggles to adapt to heterogeneous data distributions across different clients (Non-IID data).

Existing prompt-based FCL methods attempt to mitigate these issues by communicating task-specific prompts between clients and a central server. However, the authors identify a critical flaw in current approaches: Class-wise Knowledge Coherence. This lack of coherence manifests in two ways:

• Intra-class Distribution Gap: Different clients have varying local distributions for the same class. When prompts are aggregated, this gap degrades the learned semantics, causing the global prompt to be a "confused" average that fits no specific client well.
• Inter-prompt Class-wise Relevance: Prompts from different clients often contain conflicting knowledge about class relationships. Aggregating them without regard for class relevance leads to knowledge confusion and interference with historical prompts, exacerbating both spatial and temporal forgetting.

2. Methodology: C²Prompt

The authors propose C²Prompt (Class-aware Client Knowledge Interaction), a framework designed to explicitly enhance class-wise knowledge coherence during prompt communication. The method operates in two main phases per training stage and consists of four key modules (illustrated in Figure 2 of the paper):

A. Global Class Distribution Estimation

At the beginning of a new task stage (Round 0), clients compute local class statistics (mean and standard deviation) for their data. These are uploaded to the server, which aggregates them using probability theory to estimate a Global Class Distribution for each class. This global distribution serves as a reference to align local clients.

B. Local Class Distribution Compensation (LCDC)

To address the intra-class distribution gap, C²Prompt introduces a Local Class Distribution Compensation mechanism.

• Mechanism: Clients learn a set of "compensation prompts" ($P^c_{t,k}$) specifically designed to shift the local feature distribution of a class to align with the estimated Global Class Distribution.
• Loss Function: A distribution-based cross-entropy loss ($L_c$) is used to maximize the likelihood of local features under the global Gaussian distribution.
• Effect: This ensures that the semantic representation of a class (e.g., "Car") is consistent across all clients before discriminative learning begins, reducing the "distributional gap" that causes knowledge conflicts.

C. Local Discriminativity Learning

Once the compensation prompts are trained and frozen, clients learn Discriminativity Prompts ($P^d_{t,k}$) to acquire new task knowledge.

• Process: The model uses both the compensation prompts (to ensure global consistency) and the discriminative prompts (to learn specific task features).
• Histogram Recording: During local training, clients record an Instance Histogram ($H_k$) for each prompt, tracking the affinity (similarity score) between prompts and specific classes. This captures how relevant a prompt is to a specific class locally.

D. Class-aware Prompt Aggregation (CPA)

To address inter-prompt class-wise relevance, the server performs a Class-aware Prompt Aggregation scheme instead of simple averaging.

• Mechanism: The server collects client histograms to compute an Inter-prompt Correlation Matrix. This matrix estimates the relevance of prompts from different clients based on their class affinities.
• Aggregation: Prompts are aggregated using dynamic weights derived from this correlation matrix. Prompts that share high class-relevance are strengthened, while conflicting or irrelevant knowledge is suppressed.
• Effect: This selectively strengthens class-relevant knowledge aggregation, preventing the fusion of conflicting semantics and reducing interference with old tasks.

3. Key Contributions

1. C²Prompt Framework: An exemplar-free method that simultaneously mitigates temporal and spatial forgetting by explicitly modeling class-wise knowledge coherence.
2. Local Class Distribution Compensation (LCDC): A novel mechanism that transfers local semantics to the global domain, significantly improving intra-class consistency across heterogeneous clients.
3. Class-aware Prompt Aggregation (CPA): A scheme that utilizes class affinities to estimate inter-prompt relevance, generating dynamic weights to alleviate knowledge conflicts during server-side aggregation.
4. State-of-the-Art Performance: The method achieves superior results on multiple FCL benchmarks without requiring raw data storage or generative models.

4. Experimental Results

The authors evaluated C²Prompt on three standard benchmarks: ImageNet-R, DomainNet, and CIFAR-100.

• Performance Metrics: The method outperforms state-of-the-art baselines (including Powder, Fed-CODAP, Fed-L2P, and LoRA-based methods) across all metrics:
  • Average Accuracy (Avg): Improved by 2.51% on ImageNet-R and 2.90% on DomainNet compared to the previous best (Powder).
  • Forgetting Measure (FM): Achieved negative forgetting rates (indicating new tasks helped old tasks) on small datasets and the lowest forgetting on large datasets.
  • Transferability: Showed significant improvements in Forward Transfer (FT) and Combined Transfer (CT), demonstrating better knowledge accumulation.
• Ablation Studies: Removing either the LCDC or CPA modules resulted in performance drops, confirming that both distribution compensation and class-aware aggregation are necessary and complementary.
• Efficiency: C²Prompt maintains a low communication overhead (comparable to Powder) and introduces minimal parameter overhead, as the compensation prompts are few in number compared to discriminative prompts.
• Visualization: Attention maps show that C²Prompt focuses more effectively on discriminative regions of images compared to baselines, which often attend to class-irrelevant features.

5. Significance

This paper addresses a fundamental bottleneck in Federated Continual Learning: the knowledge conflict arising from the aggregation of prompts across clients with non-IID data.

• Theoretical Insight: It highlights that standard prompt aggregation fails because it ignores the class-wise coherence of the knowledge being shared.
• Practical Impact: By decoupling distribution alignment (LCDC) from discriminative learning and using class-aware aggregation (CPA), C²Prompt enables robust, long-term learning in privacy-preserving, distributed environments without the heavy computational cost of generative replay or full fine-tuning.
• Broader Application: The approach is particularly relevant for edge computing scenarios (e.g., IoT, mobile devices) where data is highly heterogeneous, privacy is paramount, and models must adapt continuously to new tasks.