🤖 AI

Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix

This paper proposes a novel multi-modality knowledge distillation framework that bridges the gap between teacher and student networks by forcing the student to learn the teacher's modality-level Gram Matrix, thereby capturing essential inter-modality relationship information rather than just final outputs.

Original authors: Peng Liu

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Shrinking the Giant

Imagine you have a massive, super-intelligent chef (the Teacher) who can prepare incredible dishes using a huge, fully equipped kitchen. This chef has millions of ingredients and tools (parameters). However, you want to teach a young, small apprentice (the Student) to cook the same dishes, but the apprentice only has a tiny backpack and a small stove. They cannot carry all the tools or remember every single step of every recipe.

In the world of AI, these "chefs" are huge computer models (like UNITER or BERT) that are too large to run on phones or small devices. Knowledge Distillation is the process where the small apprentice learns to imitate the big chef so that the small one can do great work without needing the massive kitchen.

The Problem: Copying Only the Final Dish

For a long time, researchers tried to teach the apprentice by showing them only the final dish that the chef served.

The old way: The chef says, "This is a perfect lasagna." The student tries to make a lasagna that looks exactly like that one.
The flaw: The student achieves the correct final result but does not understand how the chef combined the ingredients. In this specific paper, the "ingredients" are different types of data: Text (words) and Images (photos).

The authors argue that the old method misses the secret ingredient. It does not teach the student how the chef connects an image of a dog with the word "bark." It only teaches the final answer. For this reason, deep down, the student and teacher still think very differently, and the student is not as smart as they could be.

The New Idea: Learning the "Flavor Profile"

The authors propose a new way to teach the apprentice. Instead of just looking at the final dish, they want the student to learn the relationship between the ingredients.

They call this learning the "Modality-Level Gram-Matrix." This sounds complicated, but imagine it this way:

Imagine the chef has a special notebook where they write down how every ingredient relates to every other ingredient.

"When I see an image of a beach, I think of the word 'sand'."
"When I see an image of a storm, I think of the word 'danger'."

This notebook is the Gram-Matrix. It is a map of connections.

The teacher's notebook: The big chef has a perfect map of how images and text relate.
The student's goal: The small student tries to copy this map of connections, not just the final answer.

The paper suggests that by forcing the student to learn this "relationship map" (how the teacher connects text with images), the student becomes much smarter and performs better, even with fewer resources.

How They Tested It

The researchers tested this idea on three different "cooking challenges" (datasets):

Hateful Memes: Determining whether an image with text is malicious or not.
Visual Entailment: Looking at an image and a sentence to see if the sentence fits the image (e.g., Image: A running dog. Sentence: "The dog is sleeping." -> Contradiction).
NLVR: Checking whether a sentence accurately describes a synthetic image.

In all these tests, the "student" model (a smaller version of the big AI) was trained with two things:

The usual method (trying to get the correct answer).
The new method: Trying to copy the teacher's "relationship map" (the Gram-Matrix).

The Results

The paper claims that students who learned the "relationship map" performed better than those who only copied the final answers.

Visual proof: The researchers showed an image of the "maps" at different training stages. At the beginning, the student's map looked chaotic and different from the teacher's. However, as training progressed, the student's map began to look almost identical to the teacher's map.
The conclusion: By teaching the student how the teacher connects different types of information (images and text), the student learns more effectively and achieves better results.

Summary

This paper is about teaching a small AI to be smarter by showing it how a large AI connects images and words, rather than just showing it the final answer. It is like teaching a student not just the answer to a math problem, but the specific way the teacher's brain connects the numbers to get there. This makes the small student much more capable.

Technical Summary: Multi-Modal Distillation via Learning the Teacher's Modality-Level Gram Matrix

1. Problem Statement

The work addresses a specific limitation in existing research on Multi-Modal Knowledge Distillation (KD). While traditional KD methods successfully transfer knowledge from large "teacher" networks to smaller "student" networks, current approaches in the multi-modal domain (e.g., combining text and images) focus predominantly on distilling only the final output of the teacher network.

The authors argue that this approach ignores the rich inter-modal relationship information within the teacher network. Consequently, deep discrepancies exist between teacher and student networks, preventing the student from effectively learning the complex inter-modal dependencies (e.g., how image and text features interact) that the teacher has learned. The work assumes that merely mimicking the final classification output is insufficient for multi-modal tasks where interaction between modalities is critical.

2. Methodology

The proposed approach introduces a paradigm of Modality-Relation Distillation. Instead of merely aligning final logits, the student network is forced to learn the teacher's Modality-Level Gram Matrix.

Core Components:

Input Modalities: The framework processes three input types for both teacher and student networks:
1. Text only ( $x_t$ )
2. Image only ( $x_i$ )
3. Combined Text-Image ( $x_{i+t}$ )
Loss Functions: The overall training objective combines three components:
1. Traditional KD Loss ( $L_{IKD}$ ): Minimizes the difference between the final output logits of the teacher and the student using a standard loss function (e.g., Kullback-Leibler divergence or cross-entropy with soft targets).
2. Cross-Entropy Loss ( $L_{CE}$ ): A data augmentation strategy where the student is trained on ground-truth labels for all three input modalities (text, image, and combined). This is defined as:
  $L_{CE} = \alpha CE(f(x_t), y) + \beta CE(f(x_i), y) + \gamma CE(f(x_{i+t}), y)$
3. Modality-Relation Loss ( $L_{mr}$ ): This is the novel contribution. It forces the student to replicate the teacher's Gram Matrix of modality outputs.
  - Let $A$ be the network's output result for the three modalities (text, image, combined) for a single sample.
  - The Modality-Relation Matrix $G$ is defined as the Gram Matrix: $G = A \cdot A^T$ .
  - The loss minimizes the Mean Squared Error (MSE) between the teacher's relation matrix ( $A_t$ ) and the student's relation matrix ( $A_s$ ):
    $L_{mr} = MSE(A_t, A_s)$

Architecture:

Teacher: A 12-layer pre-trained UNITER network.
Student: A 2-layer UNITER network (same hidden size of 768, but fewer layers).
Process: The student learns to align its internal representation of modality interactions (captured by the Gram Matrix) with that of the teacher, in addition to matching the final classification output.

3. Main Contributions

The authors identify the following primary contributions:

Shift in Distillation Focus: Overcoming the conventional focus on the teacher's final output towards explicitly modeling and transferring inter-modal relationship information.
Modality-Level Gram Matrix: The proposal to use a Gram Matrix to capture relationships between outputs of different modalities (text, image, combined) within a single sample, serving as a new distillation target.
Novel Framework: Development of a modality-relation-driven framework that effectively enables the transfer of "interaction" knowledge from teacher to student, addressing the deep discrepancies that persist with standard KD methods for multi-modal tasks.

4. Experimental Results

The method was evaluated on three multi-modal datasets:

Hateful-Memes: Binary classification for hate speech detection.
SNLI-VE: Visual Entailment (predicting whether an image entails a sentence; classes: Entailment, Neutral, Contradiction).
NLVR: Natural Language Visual Reasoning (semantic analysis of synthetic images).

Performance:
The proposed method ("Ours") outperformed the baseline Conventional KD procedure on all datasets and evaluation splits (Test and Validation).

SNLI-VE (Test): Improvement from 71.22% (KD) to 72.45%.
NLVR (Test): Improvement from 73.62% (KD) to 75.33%.
Hateful-Memes (Test): Improvement from 68.22% (KD) to 69.54%.

Visualization:
The authors visualized the learned relation matrices. They observed that the student's matrix initially deviated significantly from the teacher's due to input perturbations at the start of training. As training progressed, the student's matrix became increasingly similar to the teacher's, and the absolute difference (distance matrix) decreased, indicating successful learning of modality relationships.

5. Significance and Claims

The work claims that its significance lies in addressing the "deep discrepancies" between teacher and student networks in multi-modal settings. By forcing the student to learn modality relationship information (via Gram Matrix) rather than just the final output, the method enables the student to better replicate the teacher's convergence space and understanding of how different modalities interact.

The authors note that this approach effectively utilizes knowledge transfer from teacher to student by modeling relationship information between different modalities, thereby improving multi-modal distillation performance without requiring the student to be as large as the teacher. The work suggests that standard KD is insufficient for multi-modal tasks because it fails to capture the structural relationships between modalities, which are critical for tasks such as Visual Entailment.