💬 NLP

Continual Visual Learning under Evolving Semantic Concept Shift

This paper introduces SemReWrite, a framework designed to handle evolving semantic concept shifts in visual foundation models by selectively updating obsolete visual-semantic mappings while preserving valid knowledge, validated through a new benchmark (EvoShift-Bench) and specialized evaluation metrics.

Original authors: Ismail Lamaakal, Chaymae Yahyati, Yassine Maleh, Khalid El Makkaoui, Ibrahim Ouahbi

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ismail Lamaakal, Chaymae Yahyati, Yassine Maleh, Khalid El Makkaoui, Ibrahim Ouahbi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where the rules for naming things change, but the things themselves do not. A car remains a car in every photograph, yet a new traffic law might suddenly require us to distinguish between a "private vehicle," a "commercial truck," and an "emergency responder" based on subtle details that were previously ignored. In the same way, a bird that was once simply called a "sparrow" might be reclassified into two distinct species by a new scientific study, or a factory defect might be redefined by a stricter safety standard. This is the reality for long-lived computer vision systems: the visual world stays the same, but the meaning we assign to it evolves. For decades, artificial intelligence researchers have focused on teaching computers to recognize objects even when the lighting changes or the camera angle shifts. However, a new challenge has emerged: how do we teach a computer to update its dictionary of meanings without erasing the knowledge it has already learned?

A team of researchers has tackled this problem by developing a new framework called SemReWrite. Their work addresses a specific and difficult scenario known as "evolving semantic concept shift." In this situation, the visual evidence an AI sees does not change, but the definition of what that evidence represents does. Consider an autonomous driving system that initially treats all moving vehicles as a single category. Later, a new policy requires the system to separate passenger cars from emergency vehicles because they need different actions. The images of the cars look identical, but the correct answer for the computer has fundamentally changed. Traditional methods for updating AI models often fail here. If you simply retrain the model on the new rules, it might learn the new distinctions but forget how to handle the old, still-valid categories. If you try to protect the old knowledge too strictly, the model might refuse to learn the new rules at all. The researchers found that the solution is not to choose between learning and remembering, but to do both selectively.

The core of their approach is a system that acts like a careful editor rather than a blank slate. When the semantic rules change, the system first analyzes the difference between the old definition and the new one. It uses a small number of new examples, labeled with the updated rules, to pinpoint exactly which parts of the visual world are affected by this change. For instance, if the rule changes to distinguish between types of vehicles, the system identifies that the visual features of the vehicle's size or shape are now critical, while the features of the road or the weather remain irrelevant to the new definition. Once it knows where the change matters, it applies a targeted update only to those specific areas of the model's decision-making process. The rest of the model, which handles the parts of the world that have not changed, is left untouched. This ensures that the system learns the new definitions without accidentally unlearning the old, correct ones.

To test this idea, the researchers created a new benchmark called EvoShift-Bench, which simulates various ways that meanings can evolve. They tested scenarios where a single category splits into two, where two categories merge into one, where the boundary between categories shifts, and where entirely new categories are inserted. They also tested situations where the visual appearance of the world changes at the same time as the definitions, such as when a system trained on clear photos must adapt to sketches or low-light images. In these tests, the new framework consistently outperformed existing methods. While other approaches either failed to learn the new rules or forgot the old ones, this selective rewriting method achieved a high score in both areas. It successfully learned the new distinctions while preserving the accuracy of the unchanged concepts.

The results show that the system is particularly efficient. It can learn these new semantic rules using very few examples—sometimes as few as one or two images per new category. This is crucial because in the real world, getting a large number of new, correctly labeled images is often difficult and expensive. The researchers also found that the system could handle a sequence of changes over time. As the rules evolved through multiple stages, the system continued to adapt without accumulating errors or losing its ability to recognize stable concepts. Crucially, the system was able to suppress the old, incorrect associations. It did not just learn the new answer; it actively stopped predicting the old, now-wrong answer for the affected images. This distinction is vital: a good adaptation system must not only know what is true now but also know what is no longer true.

The study suggests that the future of long-lived visual systems lies in this kind of selective updating. Instead of treating all knowledge as equally permanent or equally disposable, these systems must be able to identify which parts of their understanding have become obsolete and which parts remain valid. By combining a clear understanding of how the definitions have changed with a precise mechanism for updating only the necessary parts of the model, the researchers have demonstrated a path forward for AI that can grow and change alongside the world it observes. The work does not claim to solve every problem in artificial intelligence, but it offers a concrete and effective method for handling the specific, complex reality of changing meanings in a static visual world.

Technical Summary: Continual Visual Learning under Evolving Semantic Concept Shift

1. Problem Definition: Evolving Semantic Concept Shift

The paper addresses a critical limitation in current visual foundation models and adaptation strategies: the assumption that while data appearance may change (covariate shift), the semantic meaning of the prediction task remains fixed. In long-lived systems (e.g., autonomous driving, industrial inspection, ecological monitoring), the definitions of concepts themselves evolve due to changes in taxonomies, policies, or decision criteria.

The authors define Evolving Semantic Concept Shift as a scenario where the conditional probability $P(Y|X)$ changes even when the marginal distribution of visual evidence $P(X)$ remains constant. In this setting:

The same visual input may require a different semantic interpretation over time.
Conventional domain adaptation fails because it assumes the task semantics are static.
Standard continual learning (CL) fails because its primary goal is to prevent all forgetting, whereas semantic evolution requires the selective forgetting of obsolete associations while preserving valid ones.
Crucially, this shift is often unidentifiable from unlabeled visual streams alone; if the image distribution does not change, statistical tests cannot detect that the labeling rule has changed. Therefore, adaptation requires explicit side information regarding the semantic revision.

2. Methodology: The SEMREWRITE Framework

The authors propose SEMREWRITE, a framework designed to selectively update obsolete visual-semantic mappings while preserving valid knowledge. The framework operates through four main stages:

A. Semantic Change Representation

The system compares the old semantic specification ( $S_t$ ) and the revised specification ( $S_{t+1}$ ) using a frozen text encoder (e.g., from CLIP).

Prototypes: Semantic definitions are encoded as normalized text prototypes ( $z_c$ ).
Semantic Delta: For persistent concepts, the change is represented as a displacement vector $\Delta z_c = z_{t+1}^c - z_t^c$ . This vector captures the direction of semantic movement in the language embedding space.
Graph Memory: A structured semantic concept memory ( $G_t$ ) maintains relationships (e.g., is-a, part-of) to handle complex operations like splitting, merging, and redefining concepts.

B. Affected Concept Localization

Since the visual stream alone cannot reveal the shift, the framework combines the semantic specification with a small set of revised supervision examples ( $D_{few}$ ) to estimate which inputs are affected.

Semantic Discrepancy: It computes the Jensen-Shannon (JS) divergence between the old and revised text-induced probability distributions for an input $x$ .
Sample-Level Localizer: A lightweight network $g_\psi$ is trained on the sparse revised examples to predict a "change target" $r_i$ , indicating whether an input's label has changed relative to the old model.
Rewrite Gate: These signals are combined into a continuous gate $a(x) \in [0, 1]$ , which estimates the probability that input $x$ requires semantic rewriting.

C. Selective Concept Rewriting

The framework employs an input-dependent low-rank adaptation mechanism.

Low-Rank Update: Instead of fine-tuning the entire backbone, it introduces trainable low-rank matrices $U, V$ (LoRA-style) to create a residual transformation.
Modulated Adaptation: The residual is scaled by the gate $a(x)$ $a (x)$ .
- If $a(x) \approx 1$ (affected region), the model applies a strong update to align with the new semantics.
- If $a(x) \approx 0$ (unaffected region), the model remains close to the frozen representation, preserving valid knowledge.

D. Learning Objectives

The training loss ( $L_{total}$ ) is a weighted sum of five components:

$L_{new}$ : Standard cross-entropy on the revised supervision to learn the new rule.
$L_{pres}$ : A preservation loss that distills knowledge only for concepts with unchanged semantics and weights it by $(1-a(x))$ to protect unaffected regions.
$L_{erase}$ : An objective to explicitly suppress the probability of conflicting old decisions on affected samples (preventing "ghost" obsolete associations).
$L_{sem}$ : A semantic consistency loss ensuring adapted prototypes remain close to their text definitions and preserve relational graph structures.
$L_{reg}$ : Regularization to keep the low-rank updates compact.

3. Key Contributions

Problem Formulation: The paper formally defines Evolving Semantic Concept Shift, distinguishing it from covariate shift, domain adaptation, and standard class-incremental learning. It highlights the theoretical impossibility of detecting such shifts using only unlabeled visual data.
Framework (SEMREWRITE): A novel architecture that couples explicit semantic side information with sparse supervision to localize affected regions and perform selective, input-dependent model rewriting.
Benchmark (EVOSHIFT-BENCH): A new evaluation suite spanning ImageNet, iNaturalist, CUB-200-2011, and DomainNet. It covers diverse shift types: class split, merge, boundary revision, insertion, partial redefinition, recurrence, and mixed semantic-appearance shifts.
Evaluation Metrics: The authors introduce metrics specifically for selective revision:
- Rewrite Accuracy (RA): Performance on affected regions.
- Preservation Accuracy (PA): Performance on unaffected regions.
- Obsolete Retention (OR): The probability retained by outdated semantic associations.
- Selective Revision Score (SRS): A harmonic mean of RA and PA to balance rewriting and preservation.

4. Experimental Results

Experiments were conducted using CLIP ViT-B/16 (and validated on SigLIP and DINOv2) against baselines including full fine-tuning, LoRA, continual learning methods (EWC, LwF, Replay), and prompt-based adaptation.

Performance Balance: SEMREWRITE achieved the highest Selective Revision Score (SRS) across all shift types (e.g., 90.7% average SRS vs. 80.1% for Full Fine-Tuning and 81.5% for Replay).
Selective Forgetting: While Full Fine-Tuning achieved high RA (84.7%) but low PA (80.1%), indicating catastrophic forgetting of stable concepts, SEMREWRITE achieved both high RA (88.9%) and high PA (93.4%).
Obsolete Suppression: SEMREWRITE significantly reduced Obsolete Retention (OR) to 7.9%, compared to 13.6% for Full Fine-Tuning and 21.9% for Replay, demonstrating its ability to actively remove outdated rules rather than just overwriting them.
Annotation Efficiency: The method remained effective with extremely sparse supervision (1-shot), leveraging the semantic specification as a strong prior.
Mixed Shifts: In scenarios combining semantic shifts with covariate shifts (e.g., sketch or corruption), SEMREWRITE outperformed domain adaptation methods (like StatA) and prompt-only methods, which failed to handle the simultaneous change in decision rules.
Continual Evolution: Over five sequential revisions, SEMREWRITE maintained PA above 92% and kept OR below 10%, showing stability without accumulating interference.

5. Significance and Claims

The paper claims that successful adaptation under semantic evolution requires a paradigm shift from "preventing all forgetting" to "selectively preserving what remains true while rewriting what has become obsolete."

The authors argue that:

Standard adaptation methods are insufficient because they either fail to update the decision boundary (prompt-only) or destroy valid knowledge (global fine-tuning).
Continual learning methods are insufficient because they treat all previous knowledge as valuable, failing to distinguish between "harmful forgetting" (losing valid concepts) and "desirable forgetting" (removing obsolete associations).
The proposed SEMREWRITE framework demonstrates that by explicitly modeling the semantic change and localizing its visual impact, models can adapt to evolving definitions without requiring massive retraining or suffering from catastrophic interference.

The work concludes that the core challenge in long-lived visual systems is not just learning new data distributions, but dynamically managing the validity of semantic mappings as the world's definitions change.