🤖 AI

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

The paper proposes GRACE, a novel framework that unifies knowledge distillation and quantization-aware training under the Information Bottleneck principle to enable efficient, high-performance INT4 Vision-Language Models that significantly outperform existing quantization methods while nearly matching full-precision teacher performance.

Original authors: Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yanlong Chen, Amirhossein Habibian, Luca Benini, Yawei Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, world-class professor (the Teacher) who knows everything about the world, from spotting a specific bird in a photo to explaining complex science. This professor is incredibly smart but also huge, heavy, and requires a massive library to store their knowledge. You want to hire a young, eager student (the Student) who is much smaller and lighter, so they can carry the knowledge in a backpack and work on a small, battery-powered device.

The problem? When you try to shrink the professor's massive brain down to fit the student's tiny backpack, you usually lose a lot of the "good stuff." The student ends up confused, making mistakes, or forgetting important details. This is what happens when we try to compress big AI models (called Vision-Language Models) to run on smaller devices.

The paper introduces a new method called GRACE to solve this. Think of GRACE as a super-smart training camp that teaches the student how to keep the most important lessons while throwing away the fluff, all while packing the knowledge into a tiny, efficient suitcase.

Here is how GRACE works, using three simple tricks:

1. The "Trust Meter" (Confidence-Gated Distillation)

Usually, when a student learns from a teacher, they listen to everything the teacher says with equal importance. But sometimes, even a genius professor gets unsure or confused about a specific detail. If the student blindly copies these unsure guesses, they learn bad habits.

GRACE gives the student a Trust Meter.

If the teacher is very confident (low "entropy" or confusion), the student listens closely and learns hard.
If the teacher sounds unsure or hesitant (high entropy), the student puts on noise-canceling headphones and ignores that specific piece of advice.
The Result: The student only learns from the teacher's "best moments," avoiding the confusion that usually ruins the learning process.

2. The "Map of Connections" (Relational Alignment)

Imagine the teacher looks at a picture of a park. They don't just see "a tree" and "a bench" as separate items. They see the relationship: "The bench is under the tree," and "The tree is next to the path."

Standard teaching methods often just ask the student, "What is this?" and check if they get the name right. But GRACE teaches the student to look at the map of connections.

It forces the student to understand how different parts of the image relate to each other, just like the teacher does.
Even if the student is smaller, they learn to group things logically (e.g., "sky tokens" stay together, "ground tokens" stay together) rather than just memorizing isolated facts.
The Result: The student learns to "see" the world with the same structural understanding as the giant teacher, not just the final answer.

3. The "Smart Backpack Manager" (Adaptive Controller)

You have a backpack with a strict weight limit (the Quantization or bit-budget). You can't just throw everything in; you have to be strategic.

GRACE uses a Smart Manager who constantly checks the backpack:

"Are we carrying too much useless junk?"
"Are we forgetting the most important lessons?"
If the student is struggling to remember the teacher's lessons within the weight limit, the manager tightens the rules and forces the student to focus harder on the teacher.
If the student is doing great, the manager loosens the rules so the student can focus on solving the actual problems (like answering questions) rather than just copying the teacher.
The Result: The backpack is packed perfectly—no wasted space, but nothing important is left behind.

The Amazing Outcome

The paper tested this method on two famous AI models (LLaVA and Qwen). The results were surprising:

Better than the original: The tiny, compressed student (running on just 4 bits of memory) actually performed better than the original, full-sized model running on standard computers.
Speed and Size: Because the model is so much smaller and efficient, it runs 3 times faster and uses half the memory.

In short: GRACE is like a master chef teaching an apprentice. Instead of just giving the apprentice a giant recipe book (which is too heavy to carry), the chef teaches them how to taste the food (trust meter), how ingredients relate to each other (map of connections), and how to pack a lunchbox that fits everything they need without spilling a drop (smart manager). The result is a small chef who cooks just as well as the master, but can do it anywhere, anytime.

Technical Summary: GRACE – Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

1. Problem Statement

Vision-Language Models (VLMs) have demonstrated transformative capabilities in multimodal tasks but face significant deployment challenges due to their high computational and memory costs. While Post-Training Quantization (PTQ) offers a simple compression route, it often causes catastrophic accuracy degradation in VLMs, particularly at aggressive bit-widths like INT4, due to the complex multimodal distributions and cross-modal interactions inherent in these models.

Quantization-Aware Training (QAT) is a more principled alternative that allows models to adapt to low-precision constraints during optimization. However, QAT for VLMs remains underexplored. Standard QAT relies solely on task losses, which provide sparse supervision insufficient to guide the model on which information to preserve under strict bit budgets. Furthermore, naive combinations of QAT and Knowledge Distillation (KD) often fail to address the specific challenges of VLMs, such as the varying reliability of teacher supervision and the loss of fine-grained visual reasoning structures.

2. Methodology: The GRACE Framework

The authors propose GRACE (Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs), a unified framework that formulates VLM compression as an Information Bottleneck (IB) problem. Under this principle, quantization acts as a hard capacity constraint, while distillation determines which task-relevant information to retain. GRACE integrates three synergistic components:

2.1 Confidence-Gated Decoupled Knowledge Distillation (GDKD)

Standard KD treats all teacher predictions equally, ignoring that uncertain teacher outputs (high entropy) may introduce noise. GRACE addresses this via:

Decoupled Knowledge Distillation (DKD): The distillation loss is decomposed into Target Class Knowledge Distillation (TCKD) and Non-target Class Knowledge Distillation (NCKD). NCKD transfers "dark knowledge" from the distribution of incorrect classes, which is often more informative than target probabilities alone.
Confidence Gating: The authors observe a strong positive correlation between teacher entropy and prediction error. They introduce a gating mechanism where the distillation weight for each token is defined as $g_i = \exp(-\tilde{h}_i)$ , where $\tilde{h}_i$ is the normalized entropy of the teacher's output. This suppresses supervision from uncertain (high-entropy) tokens and prioritizes confident predictions.
Theoretical Justification: This gating is shown to minimize the covariance between weights and loss, effectively filtering unreliable supervision signals.

2.2 Relational Centered Kernel Alignment (RCKA)

Logit-based distillation fails to transfer the superior visual reasoning capabilities of larger models, which manifest as structured attention patterns rather than just output probabilities.

Visual Token Structure: GRACE extracts visual token representations from the penultimate layer of the LLM backbone (excluding text tokens).
Centered Kernel Alignment (CKA): Instead of aligning point-wise features, GRACE aligns the Gram matrices of visual tokens between the teacher and student. This transfers the relational structure (pairwise similarities) among visual tokens, enabling the student to learn how the teacher organizes visual information into semantically coherent regions (e.g., grouping sky tokens together).
Dimensionality Independence: CKA operates on $n \times n$ Gram matrices, allowing alignment between models with different hidden dimensions without requiring projection layers.

2.3 Adaptive Information Bottleneck Controller

To balance the competing objectives of task performance and knowledge retention under quantization constraints, GRACE employs an adaptive controller:

Constrained Optimization: The problem is formulated as minimizing task loss subject to a constraint on the distillation loss ( $L_{GDKD} \leq \tau$ ), where $\tau$ represents the information preservation budget.
Dual Ascent: The distillation weight $\beta$ is updated dynamically via projected dual ascent with Exponential Moving Average (EMA) smoothing. If the student struggles to retain teacher knowledge ( $L_{GDKD} > \tau$ ), $\beta$ increases to strengthen distillation; otherwise, it decreases to prioritize task performance.

2.4 Quantization Strategy

GRACE employs Group-wise Learned Step-size Quantization (LSQ). Weights are partitioned into small groups (default size 128), each with an independent, learnable scale parameter optimized jointly with model weights during training. This approach captures the heterogeneous weight distributions of VLMs better than per-tensor or per-channel quantization.

3. Key Contributions

Theoretical Unification: The paper establishes a theoretical connection between Knowledge Distillation and QAT through the Information Bottleneck principle, framing quantization as a capacity allocation problem where distillation guides what to preserve.
GRACE Framework: The proposal of a unified framework featuring:
- Confidence-gated decoupled distillation to filter noisy supervision.
- Relational-centered kernel alignment to transfer structural visual reasoning.
- An adaptive IB controller to dynamically balance fidelity and capacity.
Performance Breakthrough: Demonstration that INT4-quantized VLMs trained with GRACE can surpass full-precision (BF16) baselines, a result rarely achieved in aggressive quantization scenarios.

4. Experimental Results

Extensive evaluations were conducted on LLaVA-1.5 (13B teacher $\to$ 7B student) and Qwen2-VL (7B teacher $\to$ 2B student) across multiple benchmarks (SQA, MMBench, ScienceQA, etc.).

Accuracy Recovery and Surpassing:
- LLaVA-1.5-7B (INT4): Achieved 70.1% on ScienceQA, outperforming the BF16 baseline (66.8%) and matching the 13B teacher's performance.
- Qwen2-VL-2B (INT4): Achieved 76.9% on MMBench and 79.1% on ScienceQA, significantly outperforming the BF16 baseline (72.6% and 73.7%, respectively) and existing INT4 methods like RTN, AWQ, and SPEED-Q.
Comparison with Baselines: GRACE consistently outperformed standard QAT, naive KD+QAT, and state-of-the-art PTQ methods (GPTQ, AWQ, MBQ) by significant margins (e.g., +4.5% over baseline on Qwen2-VL-2B INT4).
Efficiency Gains: When deployed with real INT4 kernels on an NVIDIA A100 GPU, the GRACE model achieved:
- 3.03× higher throughput.
- 3.03× lower latency.
- 2.18× reduction in peak GPU memory.
- 3.08× reduction in parameter storage.

5. Significance and Claims

The paper claims that GRACE provides a principled foundation for joint optimization in VLM compression. By treating quantization as an Information Bottleneck problem, the framework enables models to learn representations that are inherently robust to aggressive low-bit compression.

The authors emphasize that their approach allows INT4 models to not only recover but surpass full-precision baselines, validating that principled information allocation (via confidence gating and relational alignment) is superior to standard QAT or naive distillation combinations. The work aims to democratize access to high-quality VLMs by enabling their deployment on resource-constrained devices with significant reductions in memory and compute requirements, while maintaining or improving task performance. The authors acknowledge potential societal implications, including the risk of misuse for generating misleading content, but argue that the benefits of efficient, accessible multimodal AI outweigh these risks when deployed responsibly.