💻 computer science

SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions

This paper introduces the IlluChar dataset and the plug-and-play SMSP framework to address the vulnerability of Multimodal Large Language Models to visual illusions caused by high-frequency attention bias, significantly improving their perception accuracy by aligning with human visual strategies.

Original authors: Jinzhe Tu, Ruilei Guo, Zihan Guo, Junxiao Yang, Shiyao Cui, Minlie Huang

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinzhe Tu, Ruilei Guo, Zihan Guo, Junxiao Yang, Shiyao Cui, Minlie Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a picture of a busy, noisy city street. Hidden somewhere in the chaos of neon signs, car headlights, and crowd patterns is a simple message: the number "5."

To a human, if you squint your eyes or step back a few feet, the noise fades away, and the "5" pops out clearly. But for a state-of-the-art AI (a Multimodal Large Language Model, or MLLM), that same picture is a disaster. The AI gets so distracted by the bright, chaotic details of the city that it completely misses the "5." It's like the AI is staring so hard at the individual pixels of a brick wall that it fails to see the shape of the house.

This paper, titled "SMSP: A Plug-and-Play Strategy of Multi-Scale Perception," is about teaching these AI models how to "squint" and "step back" just like humans do.

Here is the breakdown of their discovery and solution, using simple analogies:

1. The Problem: The AI's "Hyper-Focus" Blindness

The researchers found that AI models have a specific weakness called "High-Frequency Attention Bias."

The Analogy: Imagine you are trying to hear a whisper in a room where someone is playing a drum solo. Your brain naturally focuses on the loud, fast, complex beats (the high-frequency noise) and ignores the quiet whisper.
The Reality: In visual illusions, the background is like that drum solo. It is full of complex, high-frequency textures (like static on an old TV or intricate patterns). The hidden message (the whisper) is smoother and simpler. The AI gets "distracted" by the loud background noise and fails to hear the whisper.

2. The New Dataset: "IlluChar"

To study this, the team built a new test called IlluChar.

Previous tests were like hiding a cartoon cat in a picture. Easy for AI to spot.
IlluChar is like hiding a specific Chinese character, a letter, or a number inside a complex, realistic scene or a wall of static noise. It's much harder.
The Result: Even the smartest AI models got less than 15% of these right. Humans, however, got nearly 100% right.

3. The Solution: SMSP (The "Perception Filter")

The team didn't try to retrain the AI (which is like trying to rewire a human brain). Instead, they created a plug-and-play tool called SMSP (Strategy of Multi-Scale Perception).

Think of SMSP as a special pair of glasses you put on the photo before showing it to the AI. These glasses simulate how humans adjust their vision to find hidden things.

The glasses do two main things:

A. The "Squint" Filter (High-Frequency Filtering)

Human Action: When you squint, you blur out the sharp, noisy details.
AI Action: The tool mathematically removes the "static" and sharp edges from the background, leaving only the smooth shapes. Suddenly, the hidden character stands out against a cleaner background.

B. The "Step Back" Filter (Spatial Rescaling)

Human Action: When you step back from a poster, the tiny details blur, but the big picture becomes clear.
AI Action: The tool shrinks the image down and then pads it back out. This compresses the background noise, making the hidden character look denser and easier to read.

C. The "Multi-Scale" Strategy

Since some hidden characters are huge and some are tiny, one setting doesn't fit all.

The Analogy: Imagine you have a toolbox with three different magnifying glasses: one for giant objects, one for medium ones, and one for tiny ones.
The AI Action: SMSP doesn't just show the AI one filtered image. It shows four images at once:
1. The original messy image.
2. A version filtered for big hidden things.
3. A version filtered for medium things.
4. A version filtered for tiny things.

The AI can then look at all four and say, "Ah, in the third image, the noise is gone, and I see the '5'!"

4. The Results: From "Clueless" to "Expert"

The results were dramatic.

Before SMSP: An AI model (Qwen3-VL-8B) got only 13% of the hidden characters right.
After SMSP: That same model got 84% right.

Crucially, this didn't break the AI's ability to do other things. If you showed it a normal picture of a cat, it still recognized the cat perfectly. It didn't need to be retrained; it just needed the right "glasses."

Why This Matters

This paper proves that sometimes, AI isn't "dumb" or lacking knowledge. It just has a different way of "seeing" than humans. It focuses on the wrong details.

By giving the AI a simple, pre-processing step that mimics human intuition (squinting and stepping back), we can fix its blind spots without expensive training. It's a reminder that sometimes, to make a machine smarter, you don't need to teach it more; you just need to show it the world in a way it can understand.

1. Problem Statement

Multimodal Large Language Models (MLLMs) exhibit a critical vulnerability to hidden-pattern visual illusions. In these images, specific content (patterns) is embedded within a background such that it is imperceptible to the model but obvious to humans.

The Gap: Current MLLMs suffer from a "perceptual misalignment" with humans. While humans can instinctively adjust their perception (e.g., squinting or viewing from a distance) to reveal hidden content, MLLMs fail to do so.
The Specific Challenge: Previous datasets focused on simple patterns (animals, clothes). This paper highlights that character-based illusions (digits, letters, Chinese characters) are significantly harder for MLLMs due to fine-grained structural details (e.g., distinguishing '6' from '8').
Security Implication: This vulnerability poses safety risks, as such illusions can be weaponized to bypass automated content moderation or camouflage malicious information.

2. Key Findings & Mechanism Analysis

The authors conducted a systematic investigation to understand why MLLMs fail:

High-Frequency Attention Bias: Through spectral energy analysis (Fast Fourier Transform), the authors found that illusion backgrounds contain strong high-frequency components (textures, noise).
Attention Shift: Analysis of model attention maps (using CLIP-ViT based encoders) revealed that MLLMs disproportionately allocate attention to these high-frequency background textures, causing them to overlook the hidden patterns, which are encoded in relatively lower-frequency components.
Scale Sensitivity: Models struggle significantly as the scale of hidden characters varies, showing a lack of robustness to spatial diversity.

3. Methodology: SMSP (Strategy of Multi-Scale Perception)

To address the high-frequency attention bias, the authors propose SMSP, a plug-and-play, training-free framework that simulates human perceptual strategies. It consists of two main components:

A. Perception Module (High-Frequency Signal Bottleneck)

This module processes the input image to suppress distracting high-frequency noise, mimicking two human strategies:

High-Frequency Filtering (Simulating "Squinting"): The input image is transformed into the frequency domain via FFT. A low-pass filter is applied to discard high-frequency signals above a threshold ( $\lambda$ ), effectively removing background noise.
Spatial Rescaling (Simulating "Viewing from Afar"): The filtered image is downsampled by a factor ( $s$ ) and padded back to the original resolution with a white background. This prioritizes global structure over local textures, making hidden patterns denser and clearer.

B. Multi-Scale Strategy

Since hidden patterns vary in size, a single processing strength is insufficient.

The system generates $K$ variants of the processed image using a geometric progression of parameters ( $\lambda$ and $s$ ), ranging from aggressive filtering (for large patterns) to mild filtering (for small patterns).
The final input to the MLLM is a tuple of $K+1$ images: the original image plus the $K$ processed variants.
This allows the model to dynamically select the most informative view, similar to how humans adjust their focus.

4. Key Contributions

IlluChar Dataset: A comprehensive, challenging benchmark dataset containing 1,000x1,000 resolution images with hidden characters (digits, English, and Chinese). It features diverse background types (Semantic AI-generated scenes and Noise textures) and varying character scales (Large, Medium, Small).
Mechanism Identification: Theoretical characterization of High-Frequency Attention Bias as the root cause of MLLM failure on visual illusions.
SMSP Framework: A novel, training-free, plug-and-play strategy that significantly boosts performance without retraining model parameters.

5. Experimental Results

The authors evaluated SMSP on six state-of-the-art MLLMs (including Qwen3-VL, GLM-4.5V, GPT-5.2, Gemini-2.5-Pro, and Claude-Sonnet-4.5).

Dramatic Performance Gains:
- Qwen3-VL-8B-Instruct: Accuracy on illusion images jumped from 13.0% to 84.0%.
- Generalization: SMSP consistently improved performance across all models, background types (Semantic and Noise), and character scales.
- Comparison: SMSP significantly outperformed baselines like Chain-of-Thought (CoT), Gaussian Blur, and Histogram Equalization.
Preservation of General Capabilities:
- Unlike other methods that degrade performance on clean images, SMSP maintained or slightly improved accuracy on standard Visual Question Answering (VQA) tasks (SimpleVQA, MMStar, RealWorldQA).
- This confirms that the strategy does not interfere with the model's foundational visual reasoning.
Generalization to Other Patterns: SMSP also showed superior performance on non-character illusions (animals, FashionMNIST, harmful patterns).
Efficiency: While SMSP increases input tokens (by a factor of ~3-4), the runtime overhead is minimal (e.g., 1.08s to 1.43s), offering a favorable trade-off.

6. Significance and Impact

Perceptual Alignment: The work demonstrates that many MLLM failures stem from perceptual misalignment rather than a lack of knowledge or model capacity.
Training-Free Solution: It offers a practical, robust solution that does not require costly retraining or fine-tuning, making it immediately deployable.
Safety & Robustness: By mitigating the vulnerability to visual illusions, SMSP enhances the safety of MLLMs in content moderation and reduces the risk of adversarial attacks using hidden patterns.
Future Direction: The paper suggests a new paradigm for "perception-aware" multimodal modeling, where aligning model processing with human visual strategies is key to solving complex visual tasks.

Code Availability: The authors have made their code and the IlluChar dataset publicly available at https://github.com/Tujz2023/SMSP.