💻 computer science

Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

This paper identifies gradient dilution as the root cause of performance degradation in adapting Detection Transformers to Incremental Object Detection and proposes FAS, a unified framework that focuses, aligns, and sustains gradient flow through prior-injected queries, deterministic anchor distillation, and manifold-support replay to significantly outperform state-of-the-art methods.

Original authors: Aoting Zhang, Dongbao Yang, Chang Liu, Xiaopeng Hong, Yu Zhou

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aoting Zhang, Dongbao Yang, Chang Liu, Xiaopeng Hong, Yu Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly chaotic, robot to recognize animals. You start by showing it pictures of cats and dogs. The robot learns well. Then, you decide to teach it about birds. You show it new pictures, but you don't tell it which ones are cats or dogs anymore; you only label the birds.

In the world of Incremental Object Detection (IOD), this is the standard challenge: how do you teach a robot new things without it forgetting the old things?

This paper argues that when using a specific type of advanced AI called DETR (which is like a robot that looks at an entire image and guesses where objects are all at once), the robot suffers from a problem the authors call "Gradient Dilution."

Here is a simple breakdown of what that means, using everyday analogies, and how the authors fixed it with their new method, FAS.

The Problem: The Robot's "Signal" Gets Drowned Out

The authors say that as the robot learns new things, the "teaching signals" for the old things (cats and dogs) get weaker and weaker until they disappear. They call this Gradient Dilution. They break this down into three specific ways the robot gets confused:

Signal Dispersion (The "Static" Problem):
- The Analogy: Imagine you are trying to hear a friend whispering a secret in a crowded, noisy stadium. Your friend is the "old knowledge" (cats/dogs), and the crowd is "background noise" (empty sky, walls, grass).
- What happens: The robot has thousands of "ears" (queries) listening to the image. Most of them are listening to the empty background. The noise from the crowd is so loud that it completely drowns out your friend's whisper. The robot stops listening to the old animals because the "noise" of the background is mathematically overwhelming.
Assignment Drift (The "Moving Target" Problem):
- The Analogy: Imagine you are trying to teach a student to hit a moving target. In normal learning, the target stays still. But in this robot's brain, the target keeps jumping to a different spot every time the teacher blinks.
- What happens: When the robot tries to learn about a specific cat, one second it thinks the cat is at position A, and the next second it thinks it's at position B. Because the "target" keeps shifting, the robot's learning efforts cancel each other out. It's like trying to push a car that keeps changing direction; you end up going nowhere.
Support Attrition (The "Squashed Balloon" Problem):
- The Analogy: Imagine a balloon filled with air representing all the different ways a "cat" can look (sleeping, running, black, white). When the robot learns new things, it gets pressured to shrink this balloon down to a single, tiny point to save space.
- What happens: The robot forgets the variety of old cats. It only remembers the "average" cat. If it sees a cat that looks slightly different (like a black cat when it only learned white ones), it fails to recognize it. The "shape" of the knowledge collapses.

The Solution: The FAS Method

To fix this, the authors propose a three-step strategy called FAS (Focus, Align, Sustain). Think of it as a new teaching manual for the robot.

1. Focus: Tuning the Microphone

The Fix: Instead of letting the robot listen to the whole noisy stadium, they give it a "smart filter."
How it works: Before the robot even starts guessing, they inject "prior knowledge" (like a mental list of what a cat looks like) to tell the robot: "Ignore the empty sky and the walls. Only listen to the parts of the image that look like animals."
Result: This filters out the background noise, making the "whisper" of the old animals much louder and clearer.

2. Align: Locking the Target

The Fix: They stop the target from jumping around.
How it works: They use a "Teacher" model (a version of the robot that already knows the old animals) to set fixed "anchors." When the "Student" robot learns, it is forced to match its guesses to these fixed anchors, rather than letting them drift randomly.
Result: The robot stops getting confused by shifting targets. It learns in a straight, consistent line, building up knowledge instead of cancelling it out.

3. Sustain: Keeping the Balloon Full

The Fix: They prevent the knowledge balloon from shrinking into a single point.
How it works: Instead of just saving one "average" picture of a cat to remember later, they save a diverse set of pictures that cover the edges and boundaries of what a cat can look like. They call this "Manifold-Support Replay."
Result: The robot remembers the full shape of the knowledge, including the weird and unique cats, so it doesn't forget them when new animals are introduced.

The Result

The authors tested this new method on standard computer vision tests (like recognizing animals in the COCO and VOC datasets).

The Outcome: Their method (FAS) significantly outperformed all previous methods.
The Numbers: In a very difficult test where the robot learned 40 old classes and then 40 new ones, their method improved the accuracy by more than 5.0 points compared to the best existing methods.
The Takeaway: By fixing the "noise," the "shifting targets," and the "shrinking knowledge," they managed to keep the robot from forgetting what it already knew while still teaching it new things.

In short, the paper claims that by carefully managing how the robot pays attention, how it matches objects, and how it remembers variety, we can stop the "forgetting" that usually happens when AI learns new things over time.

Technical Summary: Focus, Align, and Sustain (FAS) for Incremental Object Detection

1. Problem Statement

The paper addresses the critical challenge of Incremental Object Detection (IOD) when applied to Detection Transformers (DETR). While standard deep learning approaches for object detection assume a static set of categories, real-world scenarios require models to learn new classes sequentially without forgetting previously learned ones.

The authors identify that adapting DETR to IOD introduces a systemic instability termed Gradient Dilution. Unlike CNN-based detectors that rely on fixed anchors and dense grids, DETR utilizes set-based optimization via global attention and bipartite matching. The authors argue that directly applying existing IOD strategies (like knowledge distillation or exemplar replay) to this dynamic set-based framework causes a cascading erosion of optimization signals required to preserve old knowledge.

This degradation is driven by three tightly coupled factors:

Signal Dispersion: In DETR, the number of queries ( $N \approx 300\text{--}900$ ) vastly exceeds the number of ground-truth objects ( $M$ ). Consequently, the majority of queries are assigned to the background. During incremental learning, these background-dominated gradients overwhelm the sparse foreground signals, suppressing the learning of old classes.
Assignment Drift: DETR relies on stochastic bipartite matching. As model parameters update, the assignment of a specific query to a ground-truth object fluctuates across iterations. This causes the gradient trajectories for the same old concept to be directionally inconsistent, leading to self-cancellation of updates and preventing coherent knowledge accumulation.
Support Attrition: Standard replay strategies often select samples near class centroids (mono-centric view). This creates a centripetal bias that compresses the feature manifold, causing the geometric support of old classes to shrink. This erosion weakens decision boundaries, particularly in peripheral regions, making them susceptible to interference from new classes.

2. Methodology: The FAS Framework

To counteract Gradient Dilution, the authors propose FAS (Focus–Align–Sustain), a unified framework designed to restore robust gradient dynamics through three specific mechanisms:

A. Focus: Prior-Injected Query Formulation (PIQ)

Goal: Mitigate Signal Dispersion by improving the Signal-to-Noise Ratio (SNR) at the query initialization stage.
Mechanism: Instead of initializing queries with semantic-agnostic embeddings, the authors introduce a bank of learnable class prototypes (semantic anchors). They compute a semantic saliency map by measuring the cosine similarity between encoder features and these prototypes.
Action: A "Top-P" selection strategy identifies the most salient foreground regions, physically filtering out low-confidence background regions before they enter the decoder. Queries are initialized directly from these high-value features.
Effect: This restricts the optimization search space from the entire image ( $H \times W$ ) to high-value candidates ( $N$ ), ensuring that gradient updates focus on foreground signals rather than being drowned out by background noise.

B. Align: Deterministic Anchor Distillation (DAD)

Goal: Mitigate Assignment Drift by stabilizing the direction of gradient updates across learning stages.
Mechanism: Rather than relying on the dynamic bipartite matching of the student model (which causes jitter), DAD establishes invariant spatial anchors derived from a frozen teacher model.
Action: The student model is forced to mimic the teacher's semantic interpretation and output logits at these fixed anchor coordinates. This creates a closed-loop stability constraint where the student's queries are naturally bound to the same instances as the teacher's, regardless of the student's internal matching jitter.
Effect: This enforces consistent gradient accumulation (linear with time steps) rather than sub-linear accumulation caused by random walk behavior, ensuring coherent learning of old concepts.

C. Sustain: Manifold-Support Replay (MSR)

Goal: Counteract Support Attrition by preserving the geometric integrity of old class feature manifolds.
Mechanism: Moving away from centroid-based selection (e.g., Herding), MSR decomposes the feature distribution of each old class into multiple latent sub-distributions using K-Means clustering.
Action: The replay buffer is constructed by selecting the medoids (representative samples) of these local density peaks rather than a single global centroid.
Effect: This approach explicitly preserves diverse local modes and boundary regions of the feature manifold, preventing the "covariance shrinkage" that leads to decision boundary erosion. It sustains the geometric span necessary to resist catastrophic forgetting.

3. Key Contributions

The paper claims the following primary contributions:

Theoretical Formalization: The authors systematically formalize Gradient Dilution in incremental Transformers, deconstructing it into Signal Dispersion, Assignment Drift, and Support Attrition. This provides a novel theoretical perspective on catastrophic forgetting specific to set-based architectures.
Methodological Innovation: The proposal of FAS, a mechanism-aligned framework that addresses the three identified degradation factors through Prior-Injected Query Formulation, Deterministic Anchor Distillation, and Manifold-Support Replay.
Empirical Performance: Extensive experiments demonstrate that FAS restores robust optimization dynamics and achieves state-of-the-art performance.

4. Experimental Results

The authors evaluated FAS on PASCAL VOC 2007 and MS COCO 2017 under various incremental protocols (Two-phase and Multi-phase settings).

Two-Phase Setting (COCO 40+40 & 70+10):
- FAS achieved 44.8 AP in the 40+40 split, outperforming the previous best (SDDGR) by 1.8 points.
- In the challenging 70+10 split, FAS reached 45.5 AP, surpassing the second-best method by 3.1 points.
- The method significantly narrowed the gap to the Joint-training upper bound, reducing the Absolute Gap to just 1.5 in the 70+10 setting.
Multi-Phase Setting (COCO 40+10×4):
- In long-term incremental learning, FAS consistently outperformed baselines across all phases.
- In the final phase of the 40+10×4 setting, FAS achieved 42.9 AP, a substantial improvement over the previous best (DCA) and a gain of over 5.0 AP compared to earlier strong baselines in this specific rigorous setting.
Ablation Studies:
- Removing any of the three components (PIQ, DAD, MSR) resulted in significant performance drops.
- PIQ was shown to be critical for recovering from the initial catastrophic forgetting caused by signal dispersion.
- DAD was proven essential for stabilizing gradient trajectories, reducing the forgetting percentage point (FPP) significantly.
- MSR (with $K=3$ ) was shown to outperform single-centroid replay strategies, validating the importance of preserving manifold boundaries.

5. Significance and Claims

The paper positions FAS as a necessary evolution for applying Transformer-based detectors to dynamic, real-world environments. The authors claim that their work shifts the focus from merely applying standard IOD heuristics to DETR toward addressing the structural fragility inherent in set-based optimization.

By identifying Gradient Dilution as the root cause of performance degradation, the paper argues that future incremental learning for Transformers must explicitly manage signal magnitude, assignment consistency, and feature-space geometry. The authors state that their gradient-centric perspective provides useful insights for developing more stable incremental object detectors, achieving a new state of the art without relying on complex architectural expansions or heavy computational overhead beyond the proposed mechanisms.