Technical Summary: HueManity – Probing Fine-Grained Visual Perception in MLLMs

Problem Statement

While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in high-level visual reasoning, such as visual question answering (VQA) and image captioning, existing benchmarks largely overlook their ability to capture fine-grained perceptual details. Current evaluation paradigms often conflate perception with semantic reasoning, allowing models to rely on language priors rather than pure visual input. As MLLMs are increasingly deployed in safety-critical domains (e.g., autonomous driving, medical imaging), a lack of foundational perceptual acuity—specifically the ability to extract coherent signals from visually cluttered environments—poses a significant risk. The paper argues that the ability to discern patterns from subtle, low-level cues (color, luminance, texture) is a prerequisite for robust visual intelligence, yet this capability remains unprobed by standard benchmarks.

Methodology: The HueManity Benchmark

To address this gap, the authors introduce HueManity, a scalable, automated benchmark designed to isolate and evaluate fine-grained visual perception.

Dataset Construction

• Scale and Content: The benchmark comprises 83,850 images, each featuring a two-character alphanumeric string embedded within an Ishihara-style dot pattern.
• Generation Pipeline:
  1. Text Masking: A 900x900 pixel binary mask is generated for two-character strings using the Pygame library with a bold, italic DejaVu Sans font.
  2. Color Selection: 25 distinct foreground-background color pairs were selected using a multi-stage process. This involved quantitative analysis via the CIEDE2000 ($\Delta E_{2000}$) color-difference formula to ensure balanced perceptual difficulty, followed by manual verification to guarantee human legibility.
  3. Pattern Rendering: An iterative generator populates the image with non-overlapping circles (4–15 pixels). Circles within the character region are assigned a foreground color, while others receive a background color. These colors undergo randomized transformations (gradient shifts, RGB shifts, and lightness scaling) to create dense, challenging patterns.
• Task Design: The benchmark includes two recognition tasks:
  • Numeric Task: Two-digit numbers (e.g., "17"), with a label space of 90 possibilities.
  • Alphanumeric Task: Two-character strings (e.g., "A7", "9b"), with a label space of 3,364 possibilities (excluding visually ambiguous characters like 'l', 'I', 'J', 'O').
• Control Conditions: To disentangle perceptual grouping from character recognition, models are also evaluated on the binary Text Masks (control task) alongside the Ishihara Patterns.

Experimental Setup

• Models Evaluated: Nine state-of-the-art MLLMs were tested, including commercial API-based models (GPT-4.1, GPT-4.1 mini, Claude 3.7 Sonnet, Qwen VL Max, Pixtral) and open-source models (LLaVA-v1.6 variants, Mistral-small3.1).
• Baselines:
  • Human Performance: 16 volunteers with normal color vision evaluated 100 images per task.
  • Traditional CV: A fine-tuned ResNet50 model (trained on 2,000 samples) served as a computer vision baseline.
• Evaluation Protocol: Models were prompted to output the exact text in a strict format. Performance was measured via exact-match accuracy. The study also included ablation studies on image resolution and investigations into in-context learning and fine-tuning capabilities.

Key Results

Performance Gap

The evaluation reveals a stark performance deficit in MLLMs compared to human and traditional computer vision baselines:

• Human Baseline: Achieved near-ceiling performance (99.37% on numeric, 93.25% on alphanumeric).
• ResNet50 Baseline: Achieved high accuracy (96.5% on numeric, 94.5% on alphanumeric), proving the task is solvable for architectures capable of learning fine-grained features.
• MLLM Performance:
  • Numeric Task: The best-performing model (GPT-4.1) achieved only 33.6% accuracy. Other models ranged from 19.0% down to near 0%.
  • Alphanumeric Task: Performance collapsed further. GPT-4.1 achieved only 3.0%, while most other models scored 0% or near-zero.
  • OCR vs. Perception: While models maintained high accuracy (>80%) on the binary Text Masks (demonstrating functional OCR), their performance plummeted on the Ishihara Patterns. This indicates the failure is rooted in perceptual grouping and feature extraction from clutter, not character recognition itself.

Failure Modes

Qualitative analysis identified three distinct failure patterns in MLLMs:

1. Hallucination: Generating unrelated, overly complex text (e.g., phrases like "MUST SEE") when visual processing is overwhelmed.
2. Evasion: Describing the image as a "color test" or explicitly stating an inability to perform the task, rather than attempting a guess.
3. Erratic Outputs: Producing random strings, repetitive patterns, or nonsensical numbers, indicating a lack of stable visual feature extraction.

Additional Findings

• In-Context Learning: Providing few-shot examples did not improve performance; in many cases, it degraded results, suggesting the visual complexity of examples introduces noise.
• Fine-Tuning: Fine-tuning a model (Gemma-3-4B) on the patterned images failed to improve performance and actually destroyed its ability to solve the simple mask task, indicating the models learned to output plausible strings rather than learning the perceptual skill.
• Resolution Ablation: Lowering image resolution (downsampling) unexpectedly improved performance for the best model (GPT-4.1), suggesting downsampling acts as a low-pass filter that smooths high-frequency noise, making character shapes more salient.
• Real-World Correlation: A strong positive correlation ($\rho = 0.82$) was found between HueManity performance and rankings on the Vision Arena benchmark, suggesting that fine-grained perceptual deficits correlate with general performance on real-world multimodal tasks.

Significance and Claims

The paper claims that HueManity exposes a critical weakness in the perceptual grounding of current MLLMs. The authors argue that:

1. Architectural Limitations: The failure is likely due to architectural bottlenecks (e.g., Vision Transformers dividing images into patches, projection layers acting as information bottlenecks) and pre-training objectives optimized for high-level semantics rather than low-level detail.
2. Diagnostic Utility: The benchmark serves as a potent diagnostic tool. The strong correlation with real-world benchmarks implies that improving fine-grained perception is essential for deploying MLLMs in safety-critical applications.
3. Research Direction: The findings suggest that current models cannot simply "learn" these skills through standard fine-tuning or in-context learning. Future work must focus on novel architectures that preserve low-level visual information and training objectives that explicitly foster robust visual acuity independent of semantic reasoning.

The authors release the dataset and generation code to facilitate reproducible research and community-driven extensions in this domain.