🤖 machine learning

t-gems: text-guided exit modules for decreasing clip image encoder

This paper introduces Text-Guided Exit Modules (T-GEMs) and a rate-based regularizer to enable early exiting in CLIP image encoders, effectively reducing computational costs while preserving cross-modal understanding performance by leveraging textual descriptions to guide exit decisions.

Original authors: Alberto Presta, Grzegorz Stefanski, Michal Byra, Krzysztof Arendt

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alberto Presta, Grzegorz Stefanski, Michal Byra, Krzysztof Arendt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, very large library (the AI model) that can look at a picture and read a description to figure out what they are about. Usually, to answer a question, this library forces every single book to be read from the very first page to the very last page, no matter how simple the question is. This takes a lot of time and energy.

This paper introduces a new way to let the library stop reading early if it already knows the answer. They call this system T-GEMS (Text-Guided Exit Modules).

Here is how it works, broken down into simple concepts:

1. The Problem: Reading the Whole Book When You Don't Need To

Think of the AI's "brain" (the image encoder) as a long hallway with many rooms (layers). To understand an image, the AI usually walks through all 12 rooms.

The Old Way: Even if the AI figures out the answer in Room 3, it keeps walking to Room 12 just to be safe. This is slow and uses a lot of electricity (computing power).
The Goal: We want the AI to say, "I'm confident enough in Room 3, so I'll stop here and give the answer."

2. The Challenge: The "Silent" Image

In many AI systems, the image and the text are processed separately. The text part knows the answer (e.g., "This is a cat"), but the image part is just walking down the hallway blindly. It doesn't know what it is looking for until it finishes the whole walk. Making the image part stop early is hard because it doesn't have the text clues to guide it.

3. The Solution: T-GEMS (The Text Guide)

The authors created a special "guide" called T-GEMS.

The Metaphor: Imagine you are looking at a blurry photo. If someone whispers, "It's a cat," your brain instantly starts looking for cat features (ears, whiskers) instead of scanning the whole image randomly.
How it works: T-GEMS takes the text description (the whisper) and uses it to predict what the image "hallway" should look like at different stages. It tells the image encoder, "Based on the text, you should be seeing these specific patterns by now."

4. The "Surprise" Meter (Class-Rate)

How does the AI know when to stop? The paper introduces a concept called Class-Rate, which acts like a "Surprise Meter."

The Metaphor: Imagine you are guessing a word in a game.
- If you are told the word is "Apple" and you see a picture of a red fruit, you aren't surprised. The "surprise" is low.
- If you are told the word is "Apple" but you see a picture of a car, you are very surprised. The "surprise" is high.
The Application: The system calculates how "surprised" the image data is by the text description at each step. If the surprise is low (meaning the image and text match perfectly), the system says, "We have enough information; let's exit early!"

5. Two Ways to Build the Guide

The paper tested two ways to teach this system:

The "Sample" Method (Naive): They showed the AI thousands of examples of cats and dogs to memorize what they look like at each step. This works, but it requires a huge library of examples and doesn't really connect the text to the image deeply.
The "Learning" Method (T-GEMS): They taught the AI to learn the rules directly from the text. The AI learned to predict what the image should look like just by reading the text, without needing to memorize thousands of specific photos. This is more flexible and efficient.

6. The Results: Smarter and Smaller

The researchers tested this on a standard dataset (CIFAR10) using a popular AI model (CLIP).

Saving Space: By stopping early, they could effectively "cut off" the end of the image encoder. They found that they could remove a huge chunk of the model's size (saving millions of parameters) without losing much accuracy.
Better Accuracy: Surprisingly, using the "Surprise Meter" (Class-Rate) as a training tool actually made the AI better at distinguishing between different things, even when it stopped early.
The Trade-off: If they stopped very early (after just a few rooms), accuracy dropped a bit, but if they stopped in the middle (around the 6th room), they kept almost all the accuracy while saving a massive amount of computing power.

Summary

In short, this paper teaches an AI how to listen to a text description to know exactly when it has seen enough of an image to make a guess. Instead of forcing the AI to process the entire image every time, it uses the text as a map to find the answer faster, saving time and energy while keeping the results accurate.

Technical Summary: T-GEMS – Text-Guided Exit Modules for Decreasing CLIP Image Encoder

Problem Statement
Multimodal (MM) models like CLIP align image and text inputs into a shared latent space using contrastive learning. While effective, their inference is static: every input traverses the entire transformer stack regardless of complexity, leading to high computational costs. Early Exit (EE) methods, which allow models to stop computation at intermediate layers when confident, have been explored in unimodal and fusion-based multimodal models. However, EE remains underexplored in contrastive dual-encoder models like CLIP. The primary challenge lies in the separation of modality encoders; existing dynamic approaches often rely on joint processing or token pruning, failing to utilize text encoder information to guide exits in the visual encoder. Furthermore, developing static EE methods for these models requires a way to estimate semantic content distributions in intermediate layers derived from textual descriptions without relying solely on image data.

Methodology
The authors propose Text-Guided Exit Modules (T-GEMs), a framework designed to perform static early exits in the image encoder of a dual-encoder model using information from the text encoder. The core objective is to estimate the posterior probability distribution of intermediate neuron activations ( $\ell_{ij}$ ) for a specific class $c$ , given an image $x$ and text description $d$ .

Distribution Estimation: The method assumes intermediate activations follow a Gaussian distribution. The goal is to compute the mean ( $\mu$ ) and variance ( $\sigma^2$ ) for each class and layer.
Two Estimation Approaches:
- Sampling-based: A naive approach using a calibration set $S$ to empirically calculate $\mu$ and $\sigma^2$ for each class. This avoids training new parameters but fails to establish a connection between text and image activation maps.
- Learning-based (T-GEMs): A learnable network $\Psi_i$ (the T-GEM) estimates distribution parameters directly from text embeddings ( $d_t$ ), bypassing the need for sample images during inference. This module is optimized to minimize the "rate" (surprise) of the activation signal.
The Jumper Module: To improve discrimination, a projection module $J^i$ (Jumper) maps intermediate image activations to a lower-dimensional space $\tilde{\ell}^i$ . This facilitates better rate estimation and allows for the integration of cross-modal similarity.
Classification Metric (Class-Rate): Instead of standard cosine similarity, the method introduces a class-rate metric ( $R^i_c$ ). This metric calculates the "surprise" of an activation given a class assumption. The predicted class is the one minimizing this rate (Eq. 3), effectively selecting the class that compresses the signal most efficiently.
Training Objective: The modules are trained using a combined loss function $L = L_r + L_s$ $L = L_{r} + L_{s}$ :
- $L_r$ : A rate-based regularizer that minimizes the negative log-likelihood of the Gaussian distribution, enhancing the ability to discriminate semantic content.
- $L_s$ : A cross-modal similarity loss (Cosine Similarity) between the text embedding and the projected image representation.

Key Contributions
The paper claims four primary contributions:

Analysis of Intermediate Representations: An investigation into how intermediate representations in CLIP vary across classes with respect to text embeddings, revealing that these layers contain discriminative information suitable for early exits.
Class-Rate Regularizer: The introduction of a statistic derived from activation maps that correlates with cosine similarity but serves as a trainable regularizer to improve discrimination.
Text-Guided Exit Modules (T-GEMs): The design of a module that predicts intermediate image encoder activations solely from text embeddings, demonstrating a bounded relationship between modalities.
Static Early Exit Framework: The training of static EE modules that leverage intermediate activation maps for tasks like image classification, representing the first study to explore exploiting intermediate layers of both text and visual encoders in contrastive dual-encoder models for static EE.

Experimental Results
Experiments were conducted on CIFAR-10 using CLIP ViT-B-32 and ViT-L-14 architectures.

Performance vs. Full CLIP: The T-GEMs + Jumper approach achieved competitive accuracy compared to the full CLIP model. For instance, with ViT-B-32, the method reached 92% accuracy at the final layer (RB12), comparable to the zero-shot baseline of 91.3%.
Impact of Regularization: Incorporating the rate-based regularizer ( $L_r$ ) significantly improved performance, particularly in earlier layers (RB3–RB5), yielding gains of up to 6% over methods using only cosine similarity.
Parameter Efficiency: The additional parameters introduced by T-GEMs and Jumper are negligible (hundreds of thousands), representing roughly 4% of a single residual block. This allows for significant model compression; for example, exiting at the 5th layer reduces the encoder size by approximately 52 million parameters while maintaining high accuracy.
Class-Rate vs. Cosine: The class-rate metric proved to be a reliable proxy for cosine similarity, often outperforming it when used as the inference metric, especially when combined with the rate regularizer.
Sample Sensitivity: The sampling-based approach required a reasonable number of samples (e.g., $S \ge 100$ ) to be effective, whereas the learning-based approach stabilized with a projection dimension $K=128$ , indicating that large models are not strictly necessary for the T-GEMs.

Significance and Claims
The paper positions this work as a foundational step toward efficient multimodal inference. It claims to be the first study to explore static early exits in contrastive dual-encoder models by leveraging the relationship between text and visual intermediate layers. The authors emphasize that their method reduces the size of the image encoder without compromising image-text similarity performance.

The significance lies in demonstrating that:

Intermediate activations in early encoder layers are sufficiently discriminative for classification tasks.
Rate-based regularization can extract more discriminative representations than standard similarity metrics alone.
Text embeddings can effectively guide the estimation of image activation distributions, enabling a static early exit strategy that does not require dynamic, per-sample decision-making during inference.

The authors remain modest regarding the scope, noting that while they do not currently implement a dynamic exit strategy (where each sample exits at a different layer), their static framework provides a promising basis for future dynamic implementations and lightweight multimodal encoder training.