🤖 AI

InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories

The paper introduces InduceKV, a retrieval-based method for fixed-footprint continual adaptation of multimodal LLMs that stores compact, attention-ready KV memories to achieve superior performance over existing baselines while maintaining a strict, bounded memory budget without modifying the backbone model.

Original authors: Qianyu Chen, Ziteng Feng, Canran Xiao, Runxuan Tang

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qianyu Chen, Ziteng Feng, Canran Xiao, Runxuan Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Forever Student" Who Can't Carry a Backpack

Imagine you have a brilliant, all-knowing student (the Multimodal Large Language Model or MLLM) who can see pictures and answer questions. This student is already trained on a massive amount of data.

Now, imagine this student needs to learn new skills over time: first, how to diagnose medical charts; then, how to solve math puzzles; then, how to understand legal documents.

The Dilemma:

The "Rewrite" Problem: If you try to teach them by rewriting their brain (updating the model's parameters), you might accidentally make them forget how to do the things they already knew. It's like trying to learn a new language by erasing your native one to make room.
The "Backpack" Problem: If you tell them to just "remember" every single example they've ever seen by carrying a giant backpack of flashcards (replay memory), the backpack eventually gets too heavy to carry. You run out of space.

The Goal:
The paper asks: Can we teach this student new things without rewriting their brain and without carrying a giant, growing backpack?

The Solution: InduceKV (The "Smart Index Card" System)

The authors propose InduceKV. Instead of changing the student's brain or carrying a heavy backpack, they give the student a compact, fixed-size index card system that sits outside their brain.

Here is how it works, step-by-step:

1. The Frozen Brain (The Library)

The student's brain (the model) is frozen. We never touch it. It stays exactly as it was. This ensures they don't forget their original skills.

2. The "Index Cards" (KV Memories)

When the student learns a new task (like "Medical Diagnosis"), instead of memorizing the whole textbook, the system extracts the most important "essence" of that lesson.

The Analogy: Think of this as taking a photo of the most important page of a book and turning it into a tiny, compressed index card.
The Tech: These cards contain "Key-Value" (KV) data. In simple terms, a Key is a label (like "Medical Chart"), and the Value is the actual information needed to answer a question about that chart.

3. The Fixed Budget (The Wallet)

You are only allowed to carry a fixed number of index cards (e.g., 256 cards). You cannot add more.

The Challenge: If you learn a new task, you can't just add a new card. You have to decide: Which old card should I throw away to make room for this new one?

4. The "Smart Selector" (Bilevel Optimization)

This is the magic part. The system uses a smart algorithm to pick the best cards to keep. It asks three questions before making a swap:

Current Fit: "Does this new card help me answer the current question right now?"
Retention: "If I throw away this old card, will I forget how to do the old tasks?" (It keeps a few "anchor" cards from the past to check this).
Diversity: "Is this new card just a copy of an old card? If so, don't pick it. We need a variety of different cards, not duplicates."

5. The "Magic Injection" (Retrieval)

When the student gets a new question (e.g., "What is wrong with this X-ray?"):

The system looks at the question and finds the matching Key on the index cards.
It grabs the Value (the answer data) from those cards.
It injects this data directly into the student's attention mechanism.

The Analogy: It's like the student is reading a book, and suddenly, a helpful ghost whispers the exact page they need to look at, right into their ear, without the student having to flip through the whole library.

Why is this better than the old ways?

The paper compares InduceKV to two other common methods:

Method A (PEFT/LoRA): This is like trying to teach the student by adding a small "notebook" to their brain. Over time, you need more notebooks, and they start interfering with each other.
- InduceKV wins: It keeps the brain clean and only uses a fixed-size external memory.
Method B (Replay): This is like making the student re-read old flashcards every time they learn something new. It's slow and requires storing the whole flashcard (the whole image and text).
- InduceKV wins: It stores only the compressed "essence" (the KV data), which takes up much less space and is faster to use.

The Results (What the paper actually found)

The authors tested this on several difficult tasks:

Learning new types of questions (like moving from "What is this?" to "How many are there?").
Learning new domains (like moving from general photos to medical charts or math problems).
Learning forever (a stream of new datasets arriving one after another).

The Findings:

Better Memory: InduceKV remembered old skills much better than the other methods while learning new ones.
Less Forgetting: The student didn't forget how to do the first tasks even after learning 10 new ones.
Efficiency: Even though the system has to "look up" the index cards, it is faster and uses less computer power than reading through a giant pile of old flashcards (replay).
It Works Everywhere: It worked well on different types of AI models (LLaVA, Qwen, DeepSeek), proving it's a general solution, not a trick for just one specific model.

Summary

InduceKV is a way to teach an AI new things without breaking its old knowledge. It does this by keeping the AI's brain frozen and giving it a fixed-size, smartly selected set of "cheat sheets" (KV memories) that it can pull up instantly when needed. It's like having a brilliant librarian who never forgets a book, but instead of carrying the whole library, they just carry a perfectly curated list of the most important pages.

Technical Summary: InduceKV

Problem Statement

Multimodal Large Language Models (MLLMs) face significant challenges in continual adaptation to evolving tasks and domains while operating under strict deployment constraints. Existing approaches typically fall into two categories, both of which struggle with the "fixed-footprint" requirement:

Parameter Updates: Methods using Parameter-Efficient Fine-Tuning (PEFT), Mixture of Experts (MoE), or architecture expansion trade stability and plasticity but inherently rewrite model parameters over time. This complicates budget control (as expert/router states grow) and risks entangling new-task adaptations with the pretrained cross-modal interface, potentially harming generalization.
Replay-Based Mechanisms: Approaches using dynamic data selection, synthetic replay, or distillation reduce forgetting but couple continual improvement to training-time data management. They often face tension between preserving diverse historical behaviors and fitting within a fixed, small memory footprint, especially as the candidate pool grows redundant.

The central challenge addressed by this work is how to continually adapt an MLLM to new tasks while preserving prior competencies, keeping the backbone model frozen, and maintaining a strictly bounded external adaptation state (fixed footprint).

Methodology: InduceKV

The authors propose InduceKV, a retrieval-based method that externalizes task-specific increments into an attention-compatible Key-Value (KV) memory without updating the backbone parameters.

Core Mechanism

Frozen Key and KV Extraction:
- For a given input prefix $x$ , the frozen MLLM extracts a unit-norm retrieval key $r(x)$ by mean-pooling the final-layer hidden states.
- Simultaneously, it extracts compact, layerwise Key-Value (KV) payloads $\{(\bar{K}_\ell(x), \bar{V}_\ell(x))\}_{\ell=1}^L$ for each Transformer layer. These payloads are compressed via segment-mean pooling to a fixed token length $m$ , ensuring a task-independent memory size.
- These components form a memory entry $e(x)$ stored in an external memory.
Retrieval-Weighted KV Induction:
- At inference, a query key $r(x)$ retrieves relevant memory entries based on cosine similarity.
- A lightweight calibration parameter $\phi$ (consisting of a retrieval temperature $\tau$ and per-layer value gates $\lambda_\ell$ ) controls the retrieval strength and the injection of memory values.
- The retrieved KV payloads are concatenated and injected as additional cache tokens into the model's masked self-attention. The attention mechanism is modified to attend to both the standard prefix tokens and the injected memory tokens, allowing the model to leverage external evidence without altering $\theta$ .
Bilevel Inducing-Set Selection:
- To operate under a strict memory budget $B$ $B$ , the system cannot store all extracted entries. It employs a bilevel optimization process to select a compact "inducing set":
  - Inner Level: Fits a minimal retrieval calibration $\phi$ on the current task data to minimize negative log-likelihood (NLL).
  - Outer Level: Selects a subset of memory entries (weights $w$ $w$ ) that jointly optimizes:
    - Current Fit: Likelihood on the current task.
    - Retention: Performance on a small set of historical "anchors" (fixed examples from previous tasks).
    - Coverage: A spectral coverage regularizer (log-determinant of the covariance matrix in a projected retrieval space) to prevent redundancy and ensure diversity in the selected memory.
- The final memory $M_t$ is the Top- $B$ entries selected by the outer optimization.

Key Contributions

Reframing Continual Adaptation: The paper reframes continual MLLM adaptation as a budgeted online inducing-set selection problem for retrieval-based memory. This shifts the stability-plasticity tension from an ever-growing parameter update process to a memory allocation problem.
Retrieval-Driven Adaptation Mechanism: The authors introduce a mechanism that stores task increments as attention-compatible external KV memories. Crucially, it uses a lightweight calibration interface to control retrieval strength and explicitly accounts for prefill and KV-injection overhead, injecting task-specific evidence without repeated backbone updates.
Empirical and Theoretical Validation: The approach achieves state-of-the-art results across diverse settings (task-incremental instruction tuning, continual VQA, domain-incremental adaptation, and lifelong tuning). Theoretically, the paper provides regret bounds showing that the relaxed memory selector competes with the best fixed inducing-set allocation under the same budget, with sublinear regret.

Experimental Results

The authors evaluated InduceKV against strong baselines (including PEFT, MoE, replay, and prompt-retrieval methods) under matched memory budgets and compute constraints.

Task-Incremental Instruction Tuning (UCIT & CoIN): InduceKV consistently outperformed baselines like HiDe-LLaVA, O-LoRA, and MoELoRA. For instance, on the UCIT benchmark with a LLaVA-v1.5-7B backbone, InduceKV improved the average score by 0.88 and the last-task score by 1.12 over HiDe-LLaVA.
Continual VQA: Under both LV and T5 protocols, InduceKV achieved higher Average Precision (AP) and lower forgetting (AF) compared to CL-MoE and QUAD.
Domain-Incremental and Lifelong Tuning: The method demonstrated robustness in domain shifts (Medicine, Chart, Math) and dynamic dataset arrival scenarios (LiIT), outperforming Adapt- $\infty$ and other dynamic selection baselines.
Ablation Studies: Removing components such as retrieval-weighted induction, bilevel coupling, or spectral coverage led to significant performance drops, confirming the necessity of the joint optimization of selection weights and retrieval calibration.
Efficiency: While InduceKV incurs a prefill overhead due to the retrieval key computation, it achieves higher throughput than prompt-based replay methods (which must process retrieved exemplars through the full network) and maintains comparable throughput to PEFT methods while offering superior retention.

Significance and Claims

The paper claims that InduceKV offers a principled alternative to parameter-updating pipelines for continual MLLM adaptation. By freezing the backbone and externalizing increments into a compact, budgeted KV memory, the method:

Decouples Adaptation from Backbones: It avoids the complexity of managing growing parameter states (e.g., expert routers) and prevents the entanglement of new tasks with the pretrained interface.
Ensures Fixed Footprint: The adaptation state is strictly bounded, making it suitable for deployment scenarios with limited storage and privacy constraints.
Leverages Native Attention: Unlike prompt-retrieval methods that treat retrieved data as text, InduceKV injects evidence directly into the attention cache, allowing the model to utilize retrieved knowledge more efficiently and controllably.

The authors emphasize that the gains are not due to a stronger backbone, replay alone, or an unbounded candidate pool, but rather the specific design of the budgeted inducing-set selection and attention-aligned injection. The work suggests that scalable continual adaptation is feasible by treating memory selection as an online optimization problem rather than a static data storage task.