💻 computer science

Continual Image-Text Retrieval via Banzhaf Distillation from the Coalitional Game Perspective

This paper proposes the Banzhaf Distillation Network (BDNet) for Continual Image-Text Retrieval, which addresses challenges like semantic misalignment and knowledge aliasing by modeling high-order token interactions via Banzhaf valuation from a coalitional game perspective to preserve structural cross-modal knowledge across incremental sessions.

Original authors: Lei Zhao, Fuhao Wei, Shaofan Wang, Yanfeng Sun

Published 2026-09-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei Zhao, Fuhao Wei, Shaofan Wang, Yanfeng Sun

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The internet is awash with images and the words that describe them, a vast digital library where a photograph of a sunset might be paired with a poem, a news caption, or a casual social media post. For computers to make sense of this world, they must learn to link these two different languages: the visual language of pixels and the linguistic language of sentences. This task, known as image-text retrieval, is the engine behind finding a specific photo when you type a description, or finding the right caption for an image. However, the real world does not hand us all our data at once. Instead, information arrives in a steady stream, day after day, bringing new subjects, new styles, and new contexts. A computer trained only on pictures of cats from last year might struggle to recognize a new breed of dog or a different way of describing a scene that appears today. This challenge of learning continuously without forgetting what was learned before is the central puzzle researchers are trying to solve.

In a new study, a team of researchers from Beijing University of Technology has proposed a fresh way to help computers handle this continuous stream of visual and textual information. They call their approach the Banzhaf Distillation Network. To understand why this is necessary, imagine a computer trying to match a sentence like "a family sitting in a living room" with a photo. A simple computer might just look for the word "family" and the word "room" and see if they appear in the image. But human understanding is more subtle; it relies on how those words and visual parts work together as a group. The researchers realized that existing methods often treated these connections too simply, looking at words and image patches one by one, missing the complex relationships between them. When a computer learns a new task, like recognizing medical diagrams, it often forgets how to handle the previous task, like identifying animals. This "forgetting" happens because the computer fails to preserve the deep, structural understanding of how images and words relate to each other across different types of data.

The team's solution draws inspiration from a branch of mathematics called cooperative game theory, which studies how groups of players contribute to a shared goal. In their framework, the researchers treat every small piece of an image and every word in a sentence as individual players in a game. They ask a specific question: how much does a specific word and a specific part of an image contribute to the overall meaning when they are paired together, especially when surrounded by other words and image parts? By calculating this contribution, they can measure the true value of the connection between a visual detail and a textual description. This method, known as Banzhaf valuation, allows the computer to see not just isolated matches, but the intricate web of relationships that give a scene its full meaning. It is a way of teaching the machine to understand that the word "sitting" gains its full significance only when seen in the context of the people and the furniture around them.

The researchers built a system that uses this game-theory idea to guide the learning process. As the computer encounters new data, it uses a special module to preserve the structural knowledge it has already learned. Instead of just memorizing which words match which pictures, the system remembers the strength of the relationships between them. It does this by comparing the current understanding of a scene with what it knew before, ensuring that the fine-grained details—the specific way a "train" relates to a "station" or a "boat" to "water"—are not lost when new information arrives. The system also employs a technique called prompt learning, where it adds small, adjustable hints to the data to help the computer adapt to new types of images or writing styles without needing to be completely retrained. This allows the computer to remain flexible, learning new things while keeping its old knowledge intact.

To test their idea, the researchers ran experiments on several different datasets, ranging from collections of everyday photos and captions to specialized sets of medical images and remote sensing pictures. They compared their new system against other leading methods used for this type of continuous learning. The results showed that their approach was significantly better at remembering past tasks while learning new ones. In scenarios where the computer had to learn a sequence of different topics, their system maintained a high level of accuracy, whereas other methods tended to forget the earlier lessons. For example, when asked to retrieve images based on text descriptions across multiple tasks, their system consistently ranked the correct images higher than its competitors. The study suggests that by modeling the complex, group-level interactions between words and image parts, computers can learn more robustly and avoid the common pitfall of forgetting.

The researchers acknowledge that their method is not a perfect solution for every situation. In cases where the difference between old and new data is extremely large, such as jumping from pictures of everyday objects to highly specialized scientific images, the system still faces challenges. They also noted that their current approach requires the computer to keep some examples of old data in memory to help with the learning process, which might not be possible in all real-world settings. Despite these limitations, the work offers a compelling new perspective on how machines can learn continuously. By treating the connection between an image and a text as a cooperative effort among many small parts, the researchers have provided a way for computers to build a more stable and nuanced understanding of the world, one that grows richer with every new piece of information it encounters.

Technical Summary: Continual Image-Text Retrieval via Banzhaf Distillation from the Coalitional Game Perspective

Problem Statement

Continual Image-Text Retrieval (CITR) aims to perform cross-modal retrieval tasks using sequential multimedia data rather than a single static dataset. This setting introduces three primary challenges:

Spatial Disorder and Multi-Granularity: Cross-modal data (images and text) exhibit multi-granular semantics and spatial disorder. Without prior category labels, matching clues follow different spatial distributions, causing performance degradation in pre-trained models.
Heterogeneous Structure and Knowledge Hierarchy: The semantic structures of image tokens and text tokens differ. Incremental sessions often lead to improper balancing of old and new knowledge, jeopardizing the hierarchical relations among tokens.
Knowledge Aliasing: During incremental learning, matching clues shift (e.g., image patches, text words, or scale changes), leading to "knowledge aliasing" where the model fails to distinguish between task-specific and general cross-modal knowledge.

Existing methods, such as those using LSTM-based meta-learners or coarse-to-fine encoders, often suffer from structural semantic ignorance (failing to capture global semantic features beyond simple sequences) and inter-task knowledge aliasing (severe drift in semantic shifts or scale changes). Furthermore, while Transformers have shown success in capturing high-level structural semantics, designing tokenized configurations that alleviate structural ignorance and bias in CITR remains an open question.

Methodology: Banzhaf Distillation Network (BDNet)

The authors propose BDNet, a framework that models image-text token pairs as participants in a cooperative game to preserve fine-grained structural knowledge. The core insight is that the Banzhaf value from coalitional game theory can characterize the multi-grained semantics of image-text pairs by modeling the interaction of token pairs with respect to other tokens.

1. Framework Overview

BDNet consists of three main components:

Image-Text Encoder with Prompts (IETE): Utilizes ViT and BERT backbones. To facilitate fine-grained interaction, the model introduces learnable image and text prompts ( $P^k_I, P^k_T$ ) specific to each task $k$ . A prompt-key matching strategy is employed to select the appropriate prompt based on the input sample's non-prompt class token, ensuring adaptability across different tasks.
Contrastive Learning and Distillation (CLD) Module:
- Intra-task: Performs contrastive learning ( $L_{CL}$ ) to align image and text modalities within the current task.
- Inter-task: Performs contrastive distillation ( $L_{CD}$ ) to maintain the invariance of the image-text similarity matrix across tasks using rehearsal data, thereby mitigating catastrophic forgetting.
Banzhaf Distillation (CBD) Module: The core innovation of the paper.
- Intra-task Banzhaf Supervision: A learnable Banzhaf Interaction (BI) network ( $\Psi$ ) evaluates the Banzhaf value $B(u, v)$ for token pairs $(u, v)$ . This value measures the tendency of a specific image token and text token to contribute to the semantic alignment when interacting with other tokens. The model is supervised by a pre-trained BI network ( $\Psi^*$ ) to inherit prior knowledge of these interactions.
- Inter-task Banzhaf Distillation: To prevent forgetting, the model minimizes the Kullback-Leibler (KL) divergence between the interaction intensities (derived from Banzhaf values) of the old model ( $\Theta_{k-1}$ ) and the new model ( $\Theta_k$ ) on rehearsal data. This preserves fine-grained cross-modal relations across tasks.

2. Banzhaf Value Approximation

Calculating the exact Banzhaf value is computationally expensive ( $O(2^N)$ ). The authors design a lightweight BI network ( $\Psi$ ) consisting of convolutional layers and a single-head self-attention layer to approximate the Banzhaf value. During pre-training, "ground-truth" Banzhaf values are estimated using a sorted similarity matrix approach with reduced complexity $O(L_I L_T (\log L_I + \log L_T))$ , where $L_I$ and $L_T$ are the numbers of image and text tokens.

3. Training Objective

The total loss function combines prompt matching, contrastive learning, contrastive distillation, and Banzhaf supervision/distillation:
$L_{total} = L_{prompt} + L_{CL} + L_{CD} + \alpha L_{Banzhaf} + \beta L_{BD}$

Key Contributions

Coalitional Game Perspective for CITR: The paper introduces a novel perspective where image and text tokens are treated as players in a cooperative game. It utilizes the Banzhaf value to model high-order relations between tokens, capturing fine-grained semantics that traditional token-by-token similarity methods miss.
Banzhaf Distillation Network (BDNet): A framework that integrates Banzhaf interaction modeling with dynamic semantic prompt learning and contrastive distillation. It effectively preserves structural cross-modal knowledge and alleviates knowledge aliasing during incremental sessions.
State-of-the-Art Performance: The proposed method achieves superior performance in both single-set and multi-set continual scenarios compared to existing baselines like LWF, EWC, RATT, and CLCMR.

Experimental Results

The authors evaluated BDNet on six scenarios using datasets including Pascal Sentence, Wikipedia, MS-COCO, Flickr30K, IAPR TC-12, ECommerce-T2I, and RSICD.

Single-Set Scenarios: On Pascal Sentence (Scenario 1) and Wikipedia (Scenario 2), BDNet outperformed all baselines. For instance, in Scenario 1, it achieved an average mAP of 0.8216, 0.8748, and 0.8410 across three tasks, significantly surpassing the next best method. It also showed the best Average Accuracy (ACC), Average Incremental Accuracy (AIA), and Backward Transfer (BWT) scores.
Multi-Set Scenarios: In scenarios involving cross-dataset learning (Scenarios 3–5), BDNet demonstrated strong generalization and anti-forgetting capabilities, particularly in longer task sequences.
Large Domain Gap: In Scenario 6 (combining five diverse datasets), BDNet performed well but did not achieve the top rank in all metrics. The authors attribute this to the large domain gap, noting that the current inter-task Banzhaf distillation (which maintains interaction intensity invariance) may not be sufficient for sparsely distributed token clues in highly heterogeneous domains.
Ablation Studies: Removing the CBD module or the prompt learning scheme resulted in significant performance drops, confirming that both fine-grained Banzhaf modeling and adaptive prompting are critical. Visualizations of token similarity matrices showed that BDNet correctly allocates attention to key semantic regions (e.g., "buildings" in a runway scene) where baseline models failed.

Significance and Limitations

The paper claims that BDNet effectively addresses the structural semantic ignorance and inter-task knowledge aliasing in CITR by leveraging coalitional game theory. By modeling token interactions via Banzhaf values, the method captures fine-grained semantics that are crucial for sequential cross-modal retrieval.

Limitations acknowledged by the authors:

Computational Efficiency: While the BI network approximates Banzhaf values efficiently, high-efficiency evaluation remains a requirement. The current summation over subsets could potentially be optimized with sparse attention.
Rehearsal Dependency: BDNet relies on rehearsal data (memory buffers) for distillation and cannot be applied to rehearsal-free scenarios.
Large Domain Gaps: The method struggles in multi-set scenarios with very large domain gaps (e.g., Scenario 6), suggesting a need for more sophisticated fine-grained knowledge distillation strategies or affinity correction mechanisms in future work.

The work is supported by the National Natural Science Foundation of China and the Beijing Natural Science Foundation. Code is available at the provided GitHub repository.