MindSet: Vision. A toolbox for testing DNNs on key psychological experiments

This paper introduces "MindSet: Vision," a versatile toolbox containing systematically manipulated image datasets and code designed to test deep neural networks against 30 specific psychological findings on human visual perception, moving beyond observational benchmarks to rigorously evaluate hypothesis-driven object recognition.

The Gist

Imagine you are trying to teach a robot how to see the world. You show it millions of photos of cats, dogs, and cars, and it gets really good at naming them. You might say, "Wow, this robot sees just like a human!"

But here's the catch: The robot might be cheating.

It might be identifying a cat not by its shape or whiskers, but by the texture of the fur or the specific background in the photo. If you show it a cat on a weird background, or a drawing of a cat, the robot might get confused, while a human would say, "That's obviously a cat!"

This is the problem the paper "MindSet: Vision" is trying to solve.

The Problem: The "Observation" Trap

Most current tests for AI vision are like pop quizzes based on real-life photos. You show the AI a picture of a dog, and it says "dog." If it gets it right, you give it a gold star.

The authors argue this is like testing a student's math skills by only asking them to memorize the answers to specific problems they've seen before. It doesn't tell you if they actually understand math.

In psychology, scientists have spent 100 years doing experiments to figure out how humans actually see. They don't just show pictures; they tweak them in tricky ways to see how our brains react.

• Example: If you put a line inside a box, does it look longer? (The Müller-Lyer illusion).
• Example: If you hide part of a circle behind a square, do we still "see" the whole circle? (Amodal completion).

The paper says: "Let's stop just showing AI photos. Let's give them these tricky psychological puzzles instead."

The Solution: The "MindSet: Vision" Toolbox

The authors built a giant digital toolbox (like a Swiss Army knife for AI researchers) containing 30 different psychological experiments.

Think of this toolbox as a gym for AI eyes. Instead of just lifting heavy weights (recognizing natural photos), the AI has to do specific, weird exercises:

1. The "Crowding" Test: Can the AI spot a letter when it's surrounded by a chaotic mess of other letters? (Humans struggle with this, and so do AIs).
2. The "Illusion" Test: Can the AI be tricked by optical illusions? If an AI sees a circle as bigger because of its neighbors (the Ebbinghaus illusion), it's thinking more like a human. If it doesn't, it's missing something crucial.
3. The "Shape vs. Texture" Test: If you turn a photo of a dog into a line drawing, can the AI still recognize it? Humans can do this instantly; many AIs cannot.

How They Tested It

The authors took 15 of the smartest, most famous AI models (the "Olympic athletes" of computer vision) and put them through 9 of these 30 tests.

The Results?
It was a bit of a disaster for the AIs.

• The Good: Some models could handle simple shapes and silhouettes.
• The Bad: Most models failed the "tricky" tests.
  • They didn't fall for the optical illusions like humans do.
  • They couldn't recognize objects if the picture was just a line drawing or a silhouette.
  • They didn't understand how parts of an object relate to each other (like how a handle is attached to a cup).

The Big Lesson

The paper isn't saying "AI is useless." It's saying, "We are measuring the wrong things."

Currently, we celebrate AI when it gets 99% accuracy on standard photo tests. But this paper shows that even the best AIs are failing the "psychology 101" tests that human babies pass easily.

The Analogy:
Imagine you are testing a driver.

• Old Method: You let them drive on a sunny day on a familiar highway. They drive perfectly. You say, "Great driver!"
• MindSet Method: You put them in a foggy storm, on a road with a fake pothole, and ask them to navigate a detour. They crash.

The paper argues that to build a truly "human-like" AI, we need to stop testing them on the sunny highway and start testing them in the foggy storm.

Why This Matters

By using this toolbox, researchers can:

1. Find the cracks: See exactly where AI vision breaks down compared to human vision.
2. Fix the models: Build better AI that doesn't just memorize textures but actually understands shapes, depth, and relationships.
3. Understand humans: By seeing where the AI fails, we learn more about how our own brains work.

In short, MindSet: Vision is a new set of "trick questions" designed to stop AI from cheating and force it to truly understand how we see the world.

Technical Summary

1. Problem Statement

Current benchmarks for evaluating Deep Neural Networks (DNNs) as models of human vision (e.g., Brain-Score) are predominantly observational. They assess model performance on naturalistic images or standard classification tasks where the stimuli are not systematically manipulated to test specific hypotheses about visual processing.

• The Limitation: DNNs can achieve high scores on these benchmarks by exploiting statistical confounds (e.g., texture vs. shape, background cues) rather than learning human-like representations. Consequently, models may predict brain activity or behavioral errors without actually mimicking the underlying mechanisms of human vision.
• The Gap: There is a disconnect between the rich literature of psychological experiments (which use controlled manipulations of independent variables to isolate specific perceptual phenomena) and the computational modeling community. Researchers often lack access to these datasets in usable formats or the tools to rigorously test DNNs against them.

2. Methodology

The authors introduce MindSet: Vision, a comprehensive toolbox designed to bridge this gap. The methodology involves three core components:

A. The Dataset Collection

The toolbox contains 30 distinct image datasets derived from well-replicated psychological experiments, categorized into three domains:

1. Low and Mid-Level Vision: Includes Weber's Law (perceptual thresholds), Gestalt grouping (emergent features, crowding/uncrowding), amodal completion, and sensitivity to non-accidental properties (NAPs) vs. metric properties (MPs).
2. Visual Illusions: Datasets for size, orientation, and lightness illusions (e.g., Müller-Lyer, Ponzo, Ebbinghaus, Jastrow, Tilt, Thatcher, Adelson Checker Shadow). These include both "illusory" conditions and "scrambled" control conditions.
3. Shape and Object Recognition: Tests on line drawings, silhouettes, texturized shapes, embedded figures, global vs. local feature sensitivity, and invariance to transformations (2D and 3D).

B. Generation and Configurability

• Procedural Generation: Unlike static datasets, these are generated via scripts allowing full parameterization (image size, background color, stroke width, sample count, etc.).
• Reproducibility: Configuration files (TOML format) allow researchers to regenerate datasets with specific parameters, ensuring exact replication or systematic variation of experimental conditions.

C. Three Evaluation Methods

The toolbox provides scripts to test DNNs using three distinct analytical approaches:

1. Out-of-Distribution (OOD) Classification: Testing pre-trained models (e.g., ImageNet-trained) on modified stimuli (e.g., line drawings) to measure robustness and shape bias without retraining.
2. Similarity Judgment Analysis: Computing pairwise similarity (Cosine or Euclidean distance) of activation patterns between image pairs across network layers. This assesses if the network's internal representation aligns with human similarity judgments (e.g., are relational changes perceived as more distinct than coordinate changes?).
3. Decoder Method (Linear Probing): Attaching a small, trainable decoder (linear layer) to frozen DNN layers. The decoder is trained on "non-illusory" or scrambled data to predict a specific feature (e.g., line length, circle size) and then tested on "illusory" configurations. If the decoder's output is biased in the same direction as human perception, the DNN is said to "experience" the illusion.

3. Key Contributions

• Standardized Psychological Benchmarking: The first large-scale, open-source collection of 30 psychological experiments specifically formatted for DNN evaluation, moving beyond observational benchmarks to causal, hypothesis-driven testing.
• Tooling and Accessibility: Provides a modular codebase (GitHub) for dataset generation, configuration management, and the implementation of the three testing methods. It includes pre-generated datasets (~5,000 images per condition) and scripts to scale them.
• Systematic Framework: Enables "severe testing" where models are challenged to fail if they do not possess human-like mechanisms, rather than just measuring overall accuracy.

4. Results

The authors tested 15 state-of-the-art DNNs (including ResNets, ConvNeXts, ViTs, and FocalNets) on 9 of the 30 experiments. Key findings include:

• Partial Successes:

  • Amodal Completion: Some networks showed weak evidence of completing occluded shapes, though the effect was inconsistent across layers.
  • Jastrow Illusion: Many networks estimated the bottom arc as larger than the top arc (matching human perception), though the magnitude of the error was often unhuman-like (e.g., 10x the actual size).
  • NAP Sensitivity: Some networks showed sensitivity to non-accidental properties, but often only in deeper layers, whereas humans process this early.

• Significant Failures:

  • Illusions: Most models failed to replicate the Müller-Lyer and Ebbinghaus illusions. Decoders trained on scrambled data generally failed to generalize to illusory conditions or produced biases in the wrong direction.
  • Relational Reasoning: DNNs were largely insensitive to relational changes between object parts, treating them similarly to coordinate changes, unlike humans who are highly sensitive to relational shifts.
  • Shape Bias: On OOD tasks (e.g., classifying line drawings or silhouettes), DNNs performed significantly worse than humans (often near chance), confirming their reliance on texture rather than shape.
  • Global Structure: Models failed to degrade performance when global structure was distorted (Frankenstein stimuli) while local features were preserved, indicating a lack of global shape weighting.

5. Significance

• Redefining Model Evaluation: The paper argues that high performance on standard benchmarks (like Brain-Score) does not guarantee a model is a valid theory of human vision. True alignment requires passing "severe tests" derived from psychological literature.
• Diagnostic Tool: MindSet: Vision serves as a diagnostic tool to identify specific architectural or training deficiencies in DNNs (e.g., lack of invariance, texture bias, failure to encode 3D structure).
• Future Directions: The authors posit that characterizing these failures is crucial for building better models. The toolkit encourages a shift from "variance explained" metrics to hypothesis-driven testing, potentially guiding the development of DNNs that incorporate mechanisms for Gestalt grouping, amodal completion, and relational reasoning.
• Open Science: By releasing the code and datasets under MIT/CC licenses, the authors lower the barrier for computational neuroscientists and psychologists to collaborate, fostering a more rigorous evaluation of artificial intelligence in the context of biological vision.