💬 NLP

Frequency-Modulated Visual Restoration for Matryoshka Large Multimodal Models

This paper introduces FMVR, a simple plug-and-play frequency-modulated strategy that restores visual semantics in Large Multimodal Models by disentangling and modulating low- and high-frequency components, enabling elastic token reduction via Matryoshka Representation Learning while maintaining near-original accuracy with significantly reduced computational costs.

Original authors: Qingtao Pan, Zhihao Dou, Shuo Li

Published 2026-03-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qingtao Pan, Zhihao Dou, Shuo Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Much Information" Traffic Jam

Imagine you have a brilliant detective (the Large Multimodal Model or LMM) who is amazing at solving mysteries by reading clues and looking at pictures. However, this detective has a strict rule: they can only look at a picture if it's broken down into tiny, individual puzzle pieces called tokens.

The Issue: A high-resolution photo might need 576 puzzle pieces to be understood correctly.
The Bottleneck: If you try to feed the detective 576 pieces at once, the table gets crowded, the detective gets overwhelmed, and the process becomes incredibly slow and expensive (like trying to drive a Ferrari in rush hour traffic).
The Old Solution: To fix this, previous methods tried to just throw away the "less important" puzzle pieces to save space. They might keep only 36 or even 1 piece.
The Result: While the detective is now faster, they are also hallucinating. Because they threw away too many pieces, they miss crucial details. They might look at a picture of a kitchen and say, "I see a bed," because they lost the context of the stove and the sink. They lost the "visual semantics" (the true meaning of the image).

The New Solution: FMVR (The "Smart Filter")

The authors of this paper, Qingtao Pan and his team, invented a new trick called FMVR (Frequency-Modulated Visual Restoration). Think of it as a magic noise-canceling headphone for the detective's eyes.

Instead of just throwing away puzzle pieces, FMVR takes the few pieces the detective does have and uses a special filter to restore the missing details.

How it Works: The "High-Frequency" and "Low-Frequency" Analogy

Imagine you are listening to a song on a radio, but the signal is weak and full of static.

The "Saliency" Filter (High Frequency): FMVR uses a tool called AvgPool to find the "loud" parts of the image—the bright spots, the edges, and the main objects (like a red stop sign). It acts like a spotlight, saying, "Hey, look here! This is important!"
The "Anti-Saliency" Filter (Low Frequency): Sometimes, the loud parts are so loud they drown out the quiet details (like a faint shadow or a small text on a shirt). FMVR uses a tool called MaxPool to act as a "quiet finder." It says, "Wait, don't ignore the background. Let's amplify the quiet, weak details so they don't get lost."

By combining these two filters, FMVR takes a tiny, blurry set of puzzle pieces and reconstructs the full picture in the detective's mind. It's like taking a low-resolution JPEG and using AI to magically fill in the missing pixels so it looks like a high-definition photo again.

The "Matryoshka" Trick: The Russian Dolls

The paper also mentions Matryoshka Representation Learning.

The Analogy: Think of a set of Russian nesting dolls. You have a big doll (576 tokens), a medium one (144 tokens), a small one (36 tokens), and a tiny one (1 token).
The Innovation: Usually, if you want to use the small doll, you have to throw away the big one. But with FMVR, the model learns to handle all sizes at once.
The Benefit: You can tell the detective, "Today, I only have time to show you 1 token," or "Today, I can show you 576." The model adapts instantly. If you give it 1 token, FMVR works its magic to make that single token carry the weight of the whole image.

The Results: Fast, Cheap, and Accurate

The researchers tested this on 10 different image tests and 4 video tests. Here is what they found:

Speed: They reduced the computing power needed by 89%. It's like going from a heavy truck to a nimble sports car.
Accuracy: Even with only 36 tokens (instead of 576), the model performed almost exactly as well as the original slow model.
The "One Token" Miracle: Even when they forced the model to look at the image with just 1 single token, it still performed better than other models that were forced to look at 192 tokens!

Summary in a Nutshell

Before: To save time, we threw away picture pieces, and the AI started making mistakes because it couldn't "see" clearly.
Now: We keep the picture pieces but use a smart frequency filter (FMVR) to sharpen the image and restore the missing details.
The Outcome: The AI is now super fast (because it processes fewer pieces) but just as smart (because FMVR fills in the gaps). It's like having a detective who can solve a crime by looking at a single fingerprint and instantly visualizing the entire crime scene.

1. Problem Statement

Large Multimodal Models (LMMs), such as LLaVA, rely on visual encoders to convert images into discrete tokens, which are then processed by a Large Language Model (LLM). While effective, this approach faces two critical challenges:

Computational Overhead: The number of visual tokens is often large (e.g., 576 tokens for a 336×336 image in LLaVA-1.5). Since computational time and memory consumption scale quadratically with sequence length, high token counts significantly hinder inference efficiency and scalability, especially for high-resolution images or video.
Semantic Loss in Compression: Existing methods attempt to reduce token counts via pruning or compression. However, these strategies inevitably lead to the loss of visual semantics, particularly fine-grained details and weak visual signals. The paper demonstrates via Grad-CAM that reducing tokens (e.g., from 576 to 36) causes a noticeable degradation in visual focus, leading to hallucinations and poor reasoning in downstream tasks.

2. Methodology

The authors propose FMVR (Frequency-Modulated Visual Restoration), a plug-and-play strategy designed to recover visual semantics from compressed tokens. The method integrates with Matryoshka Representation Learning (MRL) to enable elastic token adjustment.

A. Matryoshka Visual Token Construction

Instead of fixed token counts, the model constructs nested visual tokens ( $S_i \in \{1, 9, 36, 144, 576\}$ ) by progressively applying $2\times2$ pooling with a stride of 2. This allows the model to dynamically adjust the number of tokens during inference based on computational budgets.

B. Frequency-Modulated Visual Restoration (FMVR)

FMVR is injected into the pooling operations to disentangle and modulate frequency components, restoring lost semantics. It consists of two parallel units:

AvgPool Unit (Saliency Enhancement):
- Mechanism: Applies Average Pooling to extract the low-frequency component ( $X^l_A$ ), representing global semantic context.
- Restoration: Computes the residual ( $X^h_A = X - X^l_A$ ) to obtain the high-frequency component, which acts as a saliency filter.
- Modulation: The high-frequency component is modulated by learnable parameters ( $W^h_A$ ) and used as an attention map to multiplicatively boost the activation of salient visual areas in the original input.
- Goal: Enhance dominant, salient visual features that might be diluted during compression.
MaxPool Unit (Anti-Saliency Enhancement):
- Mechanism: Applies Max Pooling to extract the high-frequency component ( $X^h_M$ ), representing the most significant local features.
- Restoration: Computes the residual ( $X^l_M = X - X^h_M$ ) to obtain the low-frequency component, which acts as an anti-saliency filter.
- Modulation: The low-frequency component is modulated by learnable parameters ( $W^l_M$ ) and used to reactivate suppressed, weak visual semantics.
- Goal: Prevent strong salient features from overshadowing subtle details (e.g., text, small objects) by explicitly strengthening weak signals.

The final restored token set is the sum of the outputs from both units.

3. Key Contributions

Analysis of Token Reduction: The paper identifies and visualizes that reducing visual tokens leads to a degradation in the model's ability to focus on nuanced visual semantics, causing hallucinations.
Visual Restoration Paradigm: It is the first work to formulate "visual restoration" specifically for token-compressed LMMs, addressing the semantic degradation caused by token sparsity.
FMVR Mechanism: A simple, plug-and-play method that disentangles visual representations into low- and high-frequency components. It uses AvgPool to enhance saliency and MaxPool to recover weak semantics, effectively enriching visual representations with minimal parameters.
Integration with MRL: By combining FMVR with Matryoshka Representation Learning, the model achieves elastic visual reasoning, allowing dynamic token adjustment during inference without retraining, while maintaining high performance.

4. Experimental Results

The authors evaluated FMVR-LLaVA on 10 image-based benchmarks (e.g., VQAv2, GQA, MME, MMVet) and 4 video-based benchmarks.

Efficiency vs. Accuracy:
- FMVR-LLaVA reduces the FLOPs of LLaVA-1.5-7B by 89% (using 36 tokens vs. 576) while maintaining ~100% of the original accuracy.
- With only 36 tokens, FMVR-LLaVA outperforms all other token pruning methods (e.g., FastV, PyramidDrop, SparseVLM) and even surpasses the baseline LLaVA-1.5 (576 tokens) by 0.8% on average across 10 benchmarks.
- Even with 1 token, the method outperforms several state-of-the-art models (e.g., InstructBLIP, Qwen-VL).
High-Resolution & Video:
- On LLaVA-NeXT-7B (high-res), FMVR with 720 tokens achieves comparable performance (65.0%) to the full 2880-token baseline (65.2%).
- In video understanding, FMVR-LLaVA outperforms Video-LLaVA by 5.1% on question-answering tasks with significantly fewer tokens.
Ablation Studies: Removing either the AvgPool or MaxPool unit results in performance drops, confirming that both saliency and anti-saliency filtering are necessary for full semantic restoration.

5. Significance

This paper addresses a fundamental bottleneck in the deployment of Large Multimodal Models: the trade-off between computational cost and visual fidelity.

Scalability: It enables LMMs to operate efficiently on resource-constrained devices (e.g., mobile) by dynamically adjusting token counts without sacrificing reasoning capabilities.
Semantic Preservation: Unlike previous compression methods that simply discard tokens, FMVR actively restores lost information through frequency modulation, ensuring that both prominent and subtle visual details are preserved.
Generalizability: The method is model-agnostic and plug-and-play, demonstrated to work effectively across different LLM backbones (Vicuna-7B, 13B) and advanced architectures (Qwen2.5-VL).

In summary, FMVR offers a novel solution to the "token curse" in LMMs, proving that high-level reasoning can be maintained even with extreme visual token reduction through intelligent frequency-based semantic restoration.