💬 NLP

WAVE++: Capturing Within-Task Variance for Continual Relation Extraction with Adaptive Prompting

WAVE++ is a novel prompt-based approach for Continual Relation Extraction that mitigates catastrophic forgetting and handles task variations by utilizing task-specific prompt pools, label descriptions, and a generative model to consolidate knowledge without the need for explicit data storage.

Original authors: Bao-Ngoc Dao, Minh Le, Quang Nguyen, Luyen Ngo Dinh, Nam Le, Linh Ngo Van

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bao-Ngoc Dao, Minh Le, Quang Nguyen, Luyen Ngo Dinh, Nam Le, Linh Ngo Van

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a professional translator who is constantly learning new languages.

At first, you learn Spanish. Then, you learn French. Then, Japanese. The problem is that as you get deeper into Japanese, you start to forget the specific nuances of Spanish. Even worse, if someone asks you a question in Spanish, you might accidentally try to answer it using Japanese grammar rules. This is what scientists call "Catastrophic Forgetting" in AI.

This paper introduces WAVE++, a new way to teach AI to learn new things (specifically, how to identify relationships in text) one after another without losing its old skills.

Here is how WAVE++ works, explained through three simple metaphors:

1. The "Specialized Toolboxes" (Task-Specific Prompt Pools)

Imagine you are a handyman. In older AI methods, you had one giant, messy toolbox for every job. When you tried to fix a sink (Task A) and then move on to wiring a house (Task B), you’d throw new electrical tools into the same box, making it impossible to find your plumbing wrenches later.

WAVE++ gives you a dedicated, organized toolbox for every single job. When it’s time to do plumbing, you grab the "Plumbing Box." When it’s time for electricity, you grab the "Electrician Box." Because the tools don't get mixed up, the "Plumbing" skills stay perfectly preserved even while you are learning "Electricity."

2. The "Detailed Instruction Manuals" (Label Descriptions)

Sometimes, a job is confusing. If someone says, "Fix the connection," you might not know if they mean a water pipe or a wire.

To prevent this confusion, WAVE++ doesn't just learn a label like "Connection." Instead, it reads a detailed manual that explains exactly what a "Water Pipe Connection" looks like versus an "Electrical Connection." By reading these "manuals" (generated by a smart AI like Gemini), the model understands the essence of the task, making it much harder to confuse two similar-looking jobs.

3. The "Democratic Voting System" (Cascade Voting)

When a new piece of text comes in, the AI has to decide: "Which toolbox should I use for this?"

Older AIs used a single "Manager" (a classifier) to make this decision. But if the Manager gets tired or confused, they pick the wrong box, and the whole process fails.

WAVE++ uses Cascade Voting. Instead of one Manager, it asks a group of "Experts" (the different toolboxes) to vote.

The first expert says, "I think this is Spanish!"
The second expert says, "I agree, it's Spanish!"
If they agree, the job is done.
If they disagree, they call in more experts to hold a vote until a clear winner emerges. This "democracy" makes the AI much more reliable and less likely to make a silly mistake.

4. The "Memory Sketchbook" (Generative Replay)

Usually, to remember old tasks, AI has to keep a massive "storage unit" full of old data. This is expensive and can be a privacy risk (like keeping everyone's private letters in a warehouse).

Instead of keeping the actual old data, WAVE++ uses a "Sketchbook." It learns the patterns of the old data and then "sketches" new, synthetic examples that look just like the old ones. It’s like remembering how to bake a cake by remembering the recipe and the smell, rather than having to keep 100 actual cakes in your fridge just to remind yourself how to do it.

Summary

WAVE++ is like a student who:

Has a separate notebook for every subject (No mixing up math and history).
Reads deep explanations for every concept (No shallow understanding).
Uses a group of friends to vote on which subject they are studying (No single point of failure).
Re-draws old lessons from memory to keep them fresh (No need for a massive library of old textbooks).

The result? An AI that gets smarter and smarter over time without ever "forgetting" what it learned on Day 1.

Technical Summary: WAVE++: Capturing Within-Task Variance for Continual Relation Extraction with Adaptive Prompting

1. Problem Statement

Continual Relation Extraction (CRE) aims to train models to learn new relational patterns sequentially without losing previously acquired knowledge (catastrophic forgetting). Existing approaches generally fall into two categories, both with significant drawbacks:

Rehearsal-based methods: These use a memory buffer to store past samples. While effective, they suffer from high memory overhead and raise critical data privacy concerns.
Prompt-based methods: These use learnable parameters (prompts) to guide pre-trained models, avoiding data storage. However, they face three core challenges:
1. Inaccurate Task Identification: Misclassifying the current task leads to the use of incorrect prompts.
2. Catastrophic Forgetting in Shared Parameters: Shared components (like a general prompt or a shared classifier) degrade as new tasks are learned.
3. Inability to Capture Variance: Current methods often use a single prompt per task, failing to account for the high within-task variability (different contexts for the same relation) and cross-task similarity (different relations appearing in identical contexts).

2. Methodology

The authors propose WAVE++, a framework inspired by the connection between Prefix-tuning and Mixture of Experts (MoE). They posit that prefix-tuning is essentially adding new "experts" to a pre-trained model. WAVE++ introduces four key technical innovations:

Task-Specific Prompt Pools (Addressing Within-Task Variance): Instead of one shared prompt pool or one prompt per task, WAVE++ assigns a dedicated pool of $M$ prompts to each task. By setting the prompt length $L=1$ , each prompt acts as an independent "expert" with its own unique key. This allows the model to adaptively select different experts for different instances within the same task, effectively capturing within-task variance.
Label Descriptions & Contrastive Learning (Enhancing Robustness): To improve the semantic richness of prompts, the authors use an LLM (Gemini 1.5) to generate detailed, diverse descriptions and examples for each relation. They employ a contrastive loss ( $\mathcal{L}_{cl}$ ) to align input representations with these description embeddings, ensuring the model learns global, relation-specific context rather than just local patterns.
Cascade Voting Mechanism (Improving Task-Identity Inference): To solve the task-prediction problem without a dedicated (and error-prone) MLP classifier, they use a voting strategy. Each task-specific prompt pool acts as an "expert" that votes on which task the input belongs to based on Mahalanobis distance. A cascade mechanism aggregates these votes, resolving conflicts between experts to identify the correct task identity.
Generative Latent Replay (Mitigating Forgetting): To protect the shared relation classifier without storing raw data, WAVE++ uses a generative model. It stores only the mean and covariance (Gaussian distributions) of the latent representations of previously learned relations. During new task training, it "replays" synthetic representations from these distributions to reinforce the classifier's memory.

3. Key Contributions

Theoretical Insight: Establishes a formal link between prompt-based continual learning and MoE architectures, proving that prefix-tuning is a specialized instance of MoE.
Architectural Innovation: Introduces task-specific prompt pools and a training-free cascade voting mechanism for robust task identification.
Semantic Augmentation: Integrates LLM-generated label descriptions to provide global context, improving the discriminative power of the model.
Privacy-Preserving Memory: Implements a generative replay mechanism that uses latent distributions instead of raw data, addressing both forgetting and privacy.

4. Results

The authors evaluated WAVE++ on two benchmark datasets: FewRel and TACRED.

State-of-the-Art Performance: WAVE++ outperformed both existing prompt-based methods (like L2P, HiDe-Prompt, and EoE) and traditional rehearsal-based methods (like RP-CRE and CRL).
Superiority over WAVE-CRE: Compared to its predecessor, WAVE++ showed significant gains (approx. 2.0% on FewRel and 4.0% on TACRED), proving the effectiveness of label descriptions and cascade voting.
Robustness: Ablation studies confirmed that the prompt pool, label descriptions, and generative replay are all critical components. Specifically, removing generative replay caused a massive performance drop (over 22% on FewRel), highlighting its necessity in preventing forgetting.

5. Significance

WAVE++ represents a significant step forward in rehearsal-free continual learning. By shifting the focus from "storing data" to "capturing variance and semantics," it provides a scalable, privacy-compliant solution for complex NLP tasks. The ability to handle both within-task and cross-task variations through an MoE-inspired architecture makes it highly adaptable to real-world scenarios where data distributions shift constantly.