🤖 machine learning

TRACER: Persistent Regularization for Robust Multimodal Finetuning

The paper introduces TRACER, a robust multimodal finetuning method that addresses catastrophic forgetting and teacher collapse by employing a Weighted Moving Average (WMA) teacher for persistent regularization, thereby significantly improving out-of-distribution accuracy and calibration.

Original authors: Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah M. Erfani

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hesam Asadollahzadeh, Feng Liu, Christopher Leckie, Sarah M. Erfani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-traveled librarian (the Pretrained Model) who knows everything about the world: art, science, history, and nature. This librarian has read millions of books and can recognize almost anything instantly.

Now, you hire this librarian to work in a very specific, narrow library that only has books about spicy food. You want them to become an expert on spicy food.

The Problem: The "Forgetful" Librarian

If you just tell the librarian, "Forget everything else, only learn about spicy food," they will do a great job at identifying chili peppers. But, because they are so focused on the new task, they start to forget their old knowledge. If you suddenly ask them, "What is a painting?" or "What is a cat?", they might struggle or give a wrong answer. In the paper, this is called Catastrophic Forgetting. The model becomes great at the new job but terrible at being a generalist, which makes it fragile when faced with weird or unexpected situations (Out-of-Distribution).

The Old Solutions: The "Static" and "Slippery" Mentors

Researchers tried to fix this by giving the librarian a mentor to help them remember the old stuff while learning the new stuff.

Static Mentor: Imagine a mentor who is a frozen statue of the librarian from Day 1. The student tries to stay close to this statue.
- The Flaw: The statue is too rigid. If the student needs to move slightly to learn the new "spicy food" task, the statue pulls them back too hard. The student ends up stuck in the middle, not fully mastering the new task nor fully keeping the old one.
Slippery Mentor (EMA): Imagine a mentor who is a "moving average" of the student's recent steps. As the student gets better at the new task, the mentor moves right alongside them.
- The Flaw: By the time the student is an expert, the mentor has moved so close that they are practically the same person. The mentor stops giving any helpful "pull" or guidance. The student is left alone to over-specialize and forget the old knowledge again.

The New Solution: TRACER (The "Trajectory" Mentor)

The authors of this paper created a new method called TRACER. Think of TRACER as a Weighted Moving Average (WMA) Mentor.

Instead of just looking at the student's current position or a frozen past, this mentor looks at the entire journey the student has taken so far.

How it works: The mentor remembers the librarian's original "Day 1" self (the robust, general knowledge) and the recent steps taken to learn the new task. It uses a special mathematical "lens" (called a Beta kernel) that ensures the mentor never forgets the beginning of the journey, even as the student gets further along.
The Result: The mentor provides a constant, gentle "tug" back toward the librarian's original, robust self. This tug is strong enough to prevent the librarian from forgetting the old stuff, but flexible enough to let them learn the new spicy food task perfectly.

The "Surgical" Approach

The paper uses some heavy math to prove that this method works like a surgical operation:

The "Parallel" Part: When the librarian needs to learn something new (like spicy food), the method lets them change their brain in that specific area.
The "Orthogonal" Part: When it comes to things unrelated to spicy food (like recognizing cats or paintings), the method acts like a shield, keeping those parts of the brain exactly as they were.

Why It Matters (In the Paper's Context)

The authors tested this on CLIP, a famous AI model that understands both images and text. They found that:

It doesn't forget: The model keeps its ability to recognize general things (like cats, cars, or art) even after being trained on specific tasks.
It handles surprises: When the model sees weird images (like a cat drawn in a sketch style or a photo with a filter), it still works well. Other methods fail here because they "overfit" to the training data.
It's stable: Unlike other methods that require very careful, tricky timing to work, TRACER works well no matter how often you update the mentor.

Summary Analogy

If training an AI is like teaching a student for a final exam:

Normal Training: The student studies only the exam topics and forgets everything else they learned in school.
Old Methods: The teacher either stands still (too rigid) or walks away with the student (too slippery), leading to a loss of general knowledge.
TRACER: The teacher walks with the student but constantly points back to the student's original notebook of general knowledge, ensuring they learn the exam material without losing their mind.

The paper concludes that this "Trajectory-Robust Anchoring" (TRACER) is a mathematically proven way to make AI models smarter at new tasks without making them dumber at everything else.

Technical Summary: TRACER – Persistent Regularization for Robust Multimodal Finetuning

1. Problem Statement

Pretrained multimodal models, such as CLIP, possess remarkable zero-shot transfer capabilities derived from large-scale pretraining. However, adapting these models to downstream tasks via standard finetuning often leads to catastrophic forgetting, where the model sacrifices its general-purpose, out-of-distribution (OOD) robustness to optimize for task-specific patterns. While various strategies exist to mitigate this trade-off (e.g., weight averaging, L2 regularization, self-distillation), they often suffer from fundamental limitations.

A critical, largely overlooked limitation identified in this work is the behavior of standard Exponential Moving Average (EMA) teachers used in robust finetuning. As training converges and the student model approaches the teacher, the EMA teacher collapses onto the student, causing the regularization signal to vanish precisely when OOD robustness is most vulnerable. Furthermore, static self-distillation methods introduce an "anchor bias" by fixing the teacher at the initial pretrained weights, preventing the model from fully converging to the optimal task solution within the task subspace.

2. Methodology: TRACER

The authors propose TRACER (Trajectory-Robust Anchoring for Contrastive Encoder Regularization), a finetuning framework grounded in a new theoretical analysis of contrastive learning.

2.1 Theoretical Framework

The paper develops a theoretical framework for multimodal contrastive finetuning by linearizing the image and text encoders. Key theoretical insights include:

Contrastive Target Matrix Reformulation: The linearized multimodal contrastive learning (MMCL) objective is reformulated as a matrix least-squares problem driven by a "contrastive target matrix" ( $Y_{FT}$ ). This allows for closed-form solutions for various finetuning strategies.
Geometric Decomposition: The analysis reveals that finetuning involves two distinct geometric behaviors:
1. Orthogonal Preservation: Maintaining pretrained knowledge in directions orthogonal to the finetuning data.
2. Task Subspace Mixing: Adapting or mixing knowledge within the subspace spanned by the finetuning data.
Limitations of Existing Methods:
- Direct Finetuning discards pretrained knowledge in the task subspace.
- L2 Regularization creates a complex, non-surgical blend of old and new weights.
- Static Self-Distillation preserves orthogonal knowledge but introduces a bias in the task subspace, preventing convergence to the optimal task solution.
- EMA Teachers suffer from signal collapse as the teacher and student converge.

2.2 The WMA Teacher

To address the collapse of EMA teachers and the bias of static teachers, TRACER introduces a Weighted Moving Average (WMA) teacher.

Mechanism: Unlike EMA, which weights history based on recency, the WMA teacher weights the student's entire trajectory based on a normalized time kernel (e.g., a Beta distribution).
Persistent Regularization: By using a kernel that places weight on both early (pretrained) and late (converged) states, the WMA teacher maintains a persistent regularizing force over finite training horizons. It prevents the teacher-student gap from vanishing too early.
Bias-Free Convergence: The authors prove that the WMA teacher enables the student to converge to the minimum-norm task solution within the task subspace (removing the static anchor bias) while strictly preserving knowledge in the orthogonal subspace.

2.3 Training Objective

TRACER combines the standard symmetric InfoNCE loss with a dynamic self-distillation loss guided by the WMA teacher:
$L_{TRACER} = L_{MMCL} + \lambda_{SD} L_{SD-WMA}$
The distillation loss ( $L_{SD-WMA}$ ) is a composite of four perspectives to operationalize the geometric principles:

Feature Distillation (FD): Aligns student and teacher embeddings.
Contrastive Relational Distillation (CRD): Matches batch-wise similarity distributions.
Interactive Contrastive Learning (ICL): Encourages student-teacher cross-modal alignment.
Cross Knowledge Distillation (Cross-KD): Aligns cross-modal logits to transfer relational structure.

3. Key Contributions

Theoretical Reformulation: Introduced the "contrastive target matrix" to reformulate linearized contrastive loss as a least-squares problem, enabling closed-form solutions and a geometric decomposition of finetuning strategies.
Identification of Teacher Collapse: Proved that standard EMA teachers suffer from inherent collapse, weakening regularization exactly when OOD robustness is most fragile.
WMA Teacher & Bias-Free Convergence: Demonstrated that a Weighted Moving Average teacher maintains a persistent regularization signal and achieves bias-free convergence in the task subspace while preserving orthogonal knowledge.
TRACER Implementation: Instantiated these principles in a practical method that combines contrastive learning with WMA-guided multi-perspective distillation.

4. Experimental Results

The authors evaluated TRACER on CLIP models (ViT-B/16, ViT-L/14, ResNet50) finetuned on ImageNet-1K and tested on five OOD distribution shifts (IN-V2, IN-R, IN-A, IN-S, ObjectNet).

OOD Performance: TRACER consistently outperformed state-of-the-art baselines (including LP-FT, FLYP, Lipsum-FT, CaRot, and Model Stock) in average OOD accuracy. For example, on ViT-B/16, TRACER achieved 64.07% average OOD accuracy compared to 62.54% for CaRot and 59.91% for LP-FT.
Calibration: TRACER achieved the lowest Expected Calibration Error (ECE) across OOD datasets, indicating improved probabilistic reliability under distribution shifts.
Geometric Preservation: Layer-wise representational similarity analysis (CKA and SVCCA) showed that while Direct Finetuning caused significant drift in deeper layers (forgetting), TRACER maintained similarity > 0.97 with the pretrained model across all layers.
Ablation Studies: Comprehensive ablations confirmed that:
- The multi-perspective distillation components are complementary.
- Moderate distillation strength ( $\lambda_{SD} \approx 1.0-2.0$ ) offers the best ID-OOD trade-off.
- The method is robust to teacher update frequencies (unlike CaRot, which requires brittle scheduling).
- Arcsine-like weighting (Beta(0.5, 0.5)) is most effective for the kernel.
Efficiency: TRACER is computationally more efficient than CaRot, avoiding the $O(d^3)$ spectral regularization cost, with an overhead of only $O(B^2)$ for distillation.

5. Significance and Claims

The paper claims that TRACER bridges the gap between the geometry of finetuning and the practical design of robust methods. By theoretically proving that trajectory regularization with a WMA teacher prevents the collapse of the regularization signal, the method offers a principled solution to catastrophic forgetting in multimodal models.

The authors emphasize that their work is purely defensive, aimed at enhancing the reliability of foundation models in real-world applications where inputs may differ from training data (e.g., medical imaging, autonomous systems). They explicitly state that the method does not accelerate harmful capabilities but rather mitigates overfitting and improves generalization. The theoretical analysis is currently developed in a linearized regime, with future work suggested for extensions to Neural Tangent Kernel (NTK) and random-feature settings for larger nonlinear models.