Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models

This paper introduces Spatial Colour Mixing as a stress test revealing that Vision-Language Models suffer from systematic perceptual weaknesses under structured color distortions where humans remain robust, suggesting that perception-aware preprocessing can significantly improve model reliability.

The Gist

The Big Idea: Tricking the AI's Eyes

Imagine you have a super-smart robot that can look at a picture and tell you exactly what it sees. It's great at recognizing cats, dogs, and famous landmarks. But, like a human who has never seen a specific optical illusion, this robot has a weird blind spot.

The researchers in this paper decided to stress-test these robots (called Vision-Language Models or VLMs) by showing them pictures that look like a "glitchy" TV screen. They didn't change the actual object in the photo; they just painted over it with a colorful, striped pattern.

The Analogy: Think of a photo of a cat. Now, imagine someone paints thin, vertical stripes of red, green, and blue over the cat's face.

• If you stand close up: The cat looks like a mess of colorful lines.
• If you step back and squint: The colors blend together, and suddenly, you can clearly see the cat again.

Humans can do this "stepping back" trick naturally. The AI, however, gets confused. It sees the stripes and says, "I don't see a cat! I see a dog!" or "I see a painting by Jackson Pollock!" even though the cat is right there.

What Did They Do?

The team created a new "test" called Spatial Colour Mixing. They took real photos and overlaid them with eight different types of colorful patterns (like a grid, stripes, or a checkerboard).

They tested nine different AI models (including popular ones like LLaVA and Qwen) on four types of images:

1. Animals (Cats, dogs, bears)
2. Art (Famous paintings)
3. Landmarks (The Eiffel Tower, Pyramids)
4. General Questions (A mix of everything)

The Shocking Results

1. The AI Crumbled: As soon as they added the colorful stripes, the AI's accuracy dropped like a stone. Even a tiny bit of distortion made the AI guess wildly wrong.
2. Bigger isn't Better: The researchers thought, "Maybe if we make the AI brain bigger, it will be smarter." They tested small, medium, and giant versions of the models. It didn't help. A giant AI was just as confused as a small one when faced with these color tricks.
3. Humans vs. Robots: They asked 61 humans to look at the same tricked-up pictures. Humans did great! They could still identify the animals easily. The AI, however, was completely lost. This proves that human eyes and AI eyes work in totally different ways.

Why Does This Happen?

The paper suggests that human vision is like a smart editor. When we see a messy, striped image, our brain says, "Okay, ignore the messy lines; let's look at the big shape." We naturally filter out the noise.

The AI, on the other hand, is like a very literal accountant. It counts every single pixel. If the pixels are red and green stripes, it gets stuck on those stripes and forgets the shape of the animal underneath. It doesn't know how to "squint" or "step back."

Can We Fix It?

The researchers tried a clever workaround. Since humans fix this by "stepping back" (which blurs the image slightly), they tried blurring the image or shrinking it down and making it bigger again before showing it to the AI.

• The Result: It worked! For some types of color tricks, this simple "blur" step helped the AI get the answer right again.
• The Catch: They also tried giving the AI a "tool" (like a Python code interpreter) to fix the image itself. But the AI didn't know when to use the tool. It kept trying to solve the puzzle with its confused eyes, even when it was failing. It lacked the self-awareness to say, "Hey, this picture looks weird; I should try a different approach."

The Takeaway

This paper isn't saying AI is useless. It's saying that AI is fragile. It can be easily fooled by visual tricks that a toddler would ignore.

To make AI safer and more reliable, we can't just make the models bigger. We need to:

1. Teach them to "squint": Give them tools to preprocess images (blur them) before looking at them.
2. Teach them self-awareness: Help them realize when they are confused so they can ask for help or try a different method.

In short: AI sees the pixels; humans see the picture. Until AI learns to see the picture, it will keep getting tripped up by these colorful illusions.

Technical Summary

1. Problem Statement

Vision-Language Models (VLMs) have achieved strong benchmark results in visual understanding and reasoning. However, they exhibit systematic perceptual weaknesses where structured, large-scale changes to pixel values can trigger confident but nonsensical predictions, even when the underlying semantic content remains easily recognizable to humans.

Existing illusion-based benchmarks often suffer from confounders such as:

• Memorization: Images scraped from the internet may be in the training data.
• Task Bias: Well-known illusions may test recognition of the illusion itself rather than perception.
• Language Grounding: Question phrasing and multiple-choice options can constrain the hypothesis space, masking true perceptual failures.

The authors aim to isolate low-level perception from high-level reasoning by introducing a controlled, programmatic family of distortions that drastically alter pixel values while preserving semantic object identity.

2. Methodology

A. Spatial Colour Mixing Distortions

The core contribution is a framework of eight spatial colour mixing variants applied programmatically to natural images. These distortions overlay structured patterns (stripes or grids) onto images, leveraging the trichromacy of human vision (RGB) and the Ostwald colour system (decomposing color into black, white, and hue).

The eight variants are:

1. RGB System (5 variants):
   • SCMix-1: Single stripe of average color per patch.
   • SCMix-2: Two stripes (R+G and B).
   • SCMix-3A: Three RGB stripes.
   • SCMix-3B: RGB stripes proportional to their original abundance.
   • SCMix-6: Six stripes (RGBCYM) proportional to original abundance.
2. Ostwald System (3 variants):
   • Ostwald RGB: Vertical black, white, and RGB lines summing to original luminance.
   • Ostwald Checker: Grid blocks divided into black, white, and hue.
   • Ostwald Random: Similar to Checker but with random alternation of black/white.

Each variant is parametrized by a distortion degree (stripe width or grid size), allowing for controlled stress testing without altering the scene content.

B. Datasets and Evaluation Protocol

The study evaluates nine VLMs across three model families (Gemma3, LLaVA, Qwen3-VL) with varying parameter scales (from 4B to 34B/30B) on four datasets:

• Animals: 1,140 images of 19 common animals (explicit class options provided).
• Artworks: 1,951 paintings by 8 famous artists (open-ended identification).
• Landmarks: 3,688 images of 15 famous landmarks (open-ended identification).
• MME: A popular benchmark with 1,188 images and 2,376 binary questions.

Human Study: A controlled experiment with 61 participants on the Animals dataset to establish a human baseline for robustness against these distortions.

C. Mitigation Strategies

The authors tested two human-inspired preprocessing steps to simulate "stepping back" or "squinting" (reducing high-frequency noise):

1. Downscale-Upscale (D/U): Reducing image resolution and then bilinearly upscaling.
2. Box Blurring: Applying a blur filter.
   They also evaluated Tool Use (using gpt-5-mini-2025-08-07 with a code interpreter) to see if models could autonomously decide to apply these pre-processing steps.

3. Key Results

A. Performance Degradation

• Sharp Accuracy Drop: VLM accuracy collapses rapidly even at low distortion degrees. For the "Animals" dataset, performance dropped from near-perfect to ~50% after the first distortion level.
• Scale Independence: Increasing the language model scale (e.g., from 4B to 27B) does not reliably mitigate the failure. Performance is driven more by the model family (specifically the vision encoder) than by the size of the language module.
• Qualitative Failures: Models produced nonsensical outputs, such as identifying a dog as an artist, hallucinating foreign text (e.g., "千里江山"), or claiming the image is "highly pixelated" rather than identifying the object.

B. Human vs. Machine Gap

• Humans demonstrated significantly higher robustness than VLMs. While human performance degraded slowly with increased distortion, VLMs failed almost immediately.
• This confirms a fundamental mismatch: VLMs process static images in a single forward pass, lacking the active, temporal integration (saccades, peripheral vision) that allows humans to filter out high-frequency noise.

C. Preprocessing and Tool Use

• Preprocessing Success: Applying D/U or blurring substantially recovered performance (e.g., >30% improvement for Ostwald Checker and SCMix-1) for several distortion types. This suggests that low-pass filtering helps VLMs focus on global shape cues.
• Tool Use Failure: When given access to a code interpreter to apply these filters, the model failed to recognize when its perception was unreliable. The "auto" tool-use mode did not improve performance over the standard mode, indicating a lack of meta-cognitive awareness regarding input quality.

D. Vision Encoder Analysis

• Cosine similarity analysis revealed that training objectives matter more than scale.
• CLIP/SigLIP encoders maintained high similarity to original images even under distortion (suggesting they rely on features easily disrupted by the illusion).
• DINOv3 encoders showed a consistent drop in similarity with increasing distortion, suggesting their self-supervised features better retain mid-level structural integrity. This implies that hybrid designs combining semantic alignment (CLIP) with perceptual robustness (DINO) could improve VLMs.

4. Key Contributions

1. New Benchmark: Introduced a controlled, programmatic family of Spatial Colour Mixing illusions (8 variants) that allow for systematic stress testing of VLM perception without changing scene semantics.
2. Comprehensive Evaluation: Evaluated 9 VLMs across 3 families and 4 datasets, demonstrating that scaling language models does not solve perceptual brittleness.
3. Human Baseline: Conducted a study with 61 participants, quantifying a significant robustness gap between human and machine perception under these specific distortions.
4. Practical Mitigation: Demonstrated that simple, human-inspired low-pass preprocessing (downscaling/blurring) can recover meaningful performance, motivating the use of perception-aware preprocessing pipelines.
5. Tool Use Limitation: Showed that current VLMs cannot autonomously detect when their perception is compromised, even when equipped with code-interpreter tools.

5. Significance and Future Directions

This paper highlights a critical vulnerability in current VLMs: they are brittle to structured pixel-level perturbations that humans easily ignore. The findings suggest that:

• Scaling is not a silver bullet: Simply making models larger does not fix fundamental perceptual misalignments.
• Architecture matters: The choice of vision encoder and its pretraining objective (e.g., DINO vs. CLIP) is more critical for robustness than the language model size.
• Hybrid Approaches: Future VLMs may need hybrid encoders that combine semantic alignment with perceptual robustness.
• Tool Use & Uncertainty: Current models lack the ability to self-diagnose unreliable inputs. Future research should focus on uncertainty-aware tool invocation, where models can recognize when to apply preprocessing or external tools to stabilize perception.

The authors release the distorted datasets and human response data to facilitate further research into VLM robustness and perception-aware AI.