🤖 machine learning

Unlocking Compositional Generalization in Continual Few-Shot Learning

This paper proposes a novel continual few-shot learning framework that decouples representation learning from compositional inference by leveraging the patch-level geometry of self-supervised Vision Transformers to optimize slot representations for holistic class identity during training and dynamically compose them for novel concepts at inference, thereby achieving state-of-the-art generalization while minimizing catastrophic forgetting.

Original authors: Phu-Quy Nguyen-Lam, Phu-Hoa Pham, Dao Sy Duy Minh, Chi-Nguyen Tran, Huynh Trung Kiet, Long Tran-Thanh

Published 2026-05-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Phu-Quy Nguyen-Lam, Phu-Hoa Pham, Dao Sy Duy Minh, Chi-Nguyen Tran, Huynh Trung Kiet, Long Tran-Thanh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Never-Ending Exam" Problem

Imagine a student taking a series of exams.

The Challenge: Every week, the teacher introduces a brand-new subject (e.g., Week 1: Birds, Week 2: Cars, Week 3: Abstract Art).
The Catch: The student only sees five pictures of each new subject.
The Trap: The student must learn these new subjects without forgetting the old ones, and crucially, they must be able to recognize completely new combinations they've never seen before (e.g., if they learned "Red Cars" and "Blue Birds," can they recognize a "Red Bird" or a "Blue Car" without ever seeing one in class?).

Most current AI models fail at this. They either forget the old subjects (catastrophic forgetting) or they get stuck memorizing the specific five pictures they saw, failing to understand the parts that make up the object.

The Core Idea: Building with LEGO Bricks

The authors argue that to solve this, an AI shouldn't look at a picture as one giant, blurry blob. Instead, it needs to break the image down into individual LEGO bricks (objects).

Old Way: The AI sees a "Red Car" and memorizes the whole shape. If it sees a "Blue Car," it panics because the color is wrong.
New Way (COMPOSE): The AI learns to identify the "Car" brick and the "Red" brick separately. Later, when it sees a "Blue Car," it recognizes the "Car" brick and the "Blue" brick, even though it never saw that specific combination before.

The Two Big Mistakes (and How COMPOSE Fixes Them)

The paper identifies two specific reasons why previous attempts at this "LEGO approach" failed, and proposes a two-step solution called COMPOSE.

Mistake #1: The "Dirty Lens" (Representation Pollution)

The Problem: To find the LEGO bricks, the AI uses a pre-trained "eye" (a Vision Transformer). However, the old methods tried to "clean up" the image using a complex internal memory (called a GRU). The authors found that this internal memory got "dirty" with too much noise, mixing up the bricks. It was like trying to sort LEGO bricks through a foggy, smudged window.

The Fix: Stop cleaning, start reading.
The authors realized that the pre-trained "eye" (specifically one trained without human labels, called a Self-Supervised ViT) already sees the world clearly. It naturally groups pixels that belong to the same object.

The Analogy: Instead of trying to re-sort the bricks with a clumsy hand, they just take a direct photo of the bricks as they naturally sit on the table. They skip the messy internal memory step entirely. This keeps the "LEGO bricks" pure and distinct.

Mistake #2: The "Over-Prepared Student" (The Compositional Trap)

The Problem: When training the AI, previous methods tried to teach it by matching parts directly. "This red patch must match the red patch in the training set."

The Result: The AI became a "parrot." It memorized exactly where the red patch goes on a specific car. If the red patch moved slightly, or if the car was a different model, the AI got confused. It learned specific roles for its parts (e.g., "Slot 1 is always the wheel," "Slot 2 is always the door") rather than learning the general concept of "wheel" or "door."

The Fix: Train Holistically, Test Compositionally.
This is the paper's most clever trick. They split the process into two completely different phases:

Phase 1 (Training): The "Group Photo" Approach.
- The AI is shown a picture and asked to identify the whole object (e.g., "This is a Car").
- It is not allowed to look at individual parts or match them to specific training examples.
- Why? This forces the AI to learn a general, flexible understanding of what a "Car" looks like as a whole, without locking specific parts into rigid roles. It keeps the "LEGO bricks" flexible.
Phase 2 (Testing): The "Puzzle Solver" Approach.
- Now, the AI is shown a new, strange picture (e.g., a "Red Bird").
- It is allowed to break the image into parts and match them piece-by-piece against what it learned.
- Why? Because the parts were trained to be flexible (in Phase 1), the AI can now successfully assemble the "Red" brick and the "Bird" brick, even though it never saw them together before.

The Results: Why It Matters

The authors tested this on standard benchmarks (like recognizing objects in synthetic grids and natural photos).

The Claim: COMPOSE achieved the best results ever for recognizing completely new concepts (94.14% accuracy on the hardest test).
The Efficiency: It did this while using almost no extra computing power. It didn't need to re-train the giant "eye" (the backbone); it just added a tiny, lightweight "router" (about 0.7% of the total size) to manage the parts.
The Comparison: Other methods that tried to fine-tune the whole system took hours and still failed to recognize new combinations. COMPOSE took minutes and succeeded.

Summary in One Sentence

COMPOSE teaches an AI to recognize new things by first learning to see the whole picture clearly without getting stuck on details, and then letting it assemble those clear parts like LEGO bricks to solve puzzles it has never seen before.

What the Paper Does Not Claim

It does not claim this works for medical diagnosis or clinical uses.
It does not claim this solves all AI problems, only this specific type of learning (Continual Few-Shot Learning).
It does not claim the AI is "conscious" or "understands" the world; it simply recognizes patterns better than previous methods.
It explicitly notes that if the underlying "eye" (the pre-trained model) is poor at seeing object shapes, this method will also struggle. The quality of the "LEGO bricks" depends on the quality of the "box" they came in.

Technical Summary: Unlocking Compositional Generalization in Continual Few-Shot Learning

Problem Statement

The paper addresses the challenge of Continual Few-Shot Learning (CFSL), where a model must sequentially acquire new concepts from limited examples without degrading previously learned knowledge. A critical bottleneck in this regime is compositional generalization: the ability to recognize novel concepts formed by unseen combinations of known parts (e.g., a "green shirt" when only "red shirts" and "green grass" were seen).

Existing approaches fail to achieve this due to two intertwined structural flaws:

Representation Pollution: Standard slot attention mechanisms often rely on recurrent GRU states to update slot representations. These states encode routing dynamics (how to suppress competing slots) rather than pure visual content, corrupting the semantic geometry provided by the backbone.
The Compositional Trap: Training with part-level matching objectives (e.g., Chamfer distance) during the learning phase forces slots to over-specialize to specific local patterns of seen concepts. This creates slot-specific gradient directions that prevent the model from generalizing to truly novel concepts (unseen combinations) at inference time.

Methodology: The COMPOSE Framework

The authors propose COMPOSE, a dual-phase framework that strictly decouples representation learning from compositional inference. The core insight is that training and inference require fundamentally different objectives.

1. Structural Foundation: Self-Supervised Geometry

The framework relies on the inherent patch-level semantic geometry of self-supervised Vision Transformers (ViTs) like DINOv2. Unlike supervised ViTs, which learn global class discrimination, self-supervised models learn local consistency where patches from the same object naturally cluster. This geometry is a strict prerequisite for reliable object decomposition.

2. Phase I: Holistic Training (Representation Learning)

Slot Extraction: Instead of using the recurrent GRU hidden state ( $s_k$ ) as the slot representation, COMPOSE computes a direct attention-weighted aggregate ( $\tilde{\phi}_k$ ) from the frozen backbone's patch features. This bypasses the recurrent bottleneck, preserving the backbone's pristine semantic geometry and significantly improving slot purity.
Objective: The model is trained using a holistic prototype cross-entropy loss. Slots are aggregated via a lightweight, trainable MLP router to form a single image-level embedding, which is optimized to match the class prototype.
Constraint: A cross-correlation penalty is applied to the raw slot projections to prevent slot collapse (redundancy) without imposing the conflicting variance constraints found in other regularization methods (like VICReg).
Crucial Distinction: No part-level matching (e.g., Chamfer distance) is performed during training. This prevents the "compositional trap" where gradients force slots to specialize in specific local parts.

3. Phase II: Compositional Inference (Novel Concept Matching)

Frozen Components: All backbone and router weights are frozen. No gradients are computed.
Matching Strategy: Novel scenes are matched to support classes using bidirectional Chamfer distance on the centered, per-slot projections.
Blended Scoring: The final prediction score is a weighted blend of the holistic score (from Phase I embeddings) and the compositional score (from part-level matching). This allows the model to leverage the robust class identity learned during training while utilizing the compositional structure for novel concept recognition.

Key Contributions

Rethinking Slot Representations: The paper identifies self-supervised patch geometry as a prerequisite for object-centric decomposition. It replaces recurrent GRU states with attention-weighted aggregates, demonstrating that this change significantly improves slot purity and preserves the geometric structure necessary for out-of-distribution transfer.
Diagnosing and Escaping the Compositional Trap: The authors analytically prove that part-level matching objectives induce slot-specific gradient directions that drive over-specialization. They propose a strict phase separation: holistic training to preserve generalizable object-level geometries, followed by compositional inference to match novel scenes.
Novel-Concept Transfer Drives Continual Robustness: The framework achieves state-of-the-art performance on the unseen-concept split (noc) of the CFST benchmark, achieving 94.14% accuracy on CGQA. This robustness to novel concepts translates to minimal catastrophic forgetting on standard continual learning benchmarks (MiniImageNet, ImageNet-R, CUB200, CIFAR100).

Experimental Results

CFST Benchmark (CGQA): COMPOSE achieves 94.14% on the noc (unseen concepts) split, outperforming strong baselines like CoFiMA (91.97%) and RanPAC (88.60%). It also achieves a high harmonic mean ( $H_a$ ) of 92.92%.
Ablation Studies:
- Removing the holistic training objective (using Chamfer during training, "COMPOSE-CT") drops noc performance by ~2.5 percentage points, confirming the "compositional trap."
- Replacing the attention-weighted aggregate with the standard GRU state drops noc by ~3.6 percentage points, validating the importance of preserving backbone geometry.
- Removing the cross-correlation penalty causes a significant drop (~5.5 percentage points), highlighting its role in preventing slot redundancy.
Continual Learning Benchmarks: On standard FSCIL and CIL tasks, COMPOSE significantly reduces forgetting (e.g., reducing FF from 27.91% to 3.54% on CIFAR-100 compared to Comp-FSCIL) while maintaining competitive average accuracy.
Efficiency: The method requires only ~5 minutes of training per dataset (training a lightweight router on a frozen backbone), compared to ~4 hours for full backbone fine-tuning methods like CoFiMA.

Significance and Claims

The paper claims that the failure of previous continual learners to generalize to novel concepts is not merely a data or capacity issue, but a structural conflict between how representations are extracted and how they are optimized.

By strictly separating the training phase (focused on holistic class identity to preserve generalizable geometry) from the inference phase (focused on compositional matching), COMPOSE demonstrates that models can learn representations general enough to recognize unseen concepts without forgetting. The authors emphasize that this approach relies on the quality of the underlying self-supervised backbone geometry; when this geometry is weak (e.g., in supervised ViTs), slot decomposition degrades, limiting compositional transfer.

The work suggests that novel-concept robustness is the key to minimizing catastrophic forgetting in continual learning, as representations that are general enough to handle unseen compositions are inherently less prone to over-specialization and drift.