Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies

Technical Summary: Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies

Problem Statement
Large-scale Vision-Language Models (LVLMs) are increasingly deployed in real-world settings such as navigation, education, and assistive technologies. A critical, yet underexplored, assumption in these deployments is that users share a uniform visual experience. In reality, visual perception varies significantly due to biological factors, particularly Color Vision Deficiencies (CVDs), which affect approximately 8% of males and 0.5% of females. Current LVLMs are trained on web-scale corpora that implicitly assume typical color vision. While these models possess substantial linguistic knowledge about CVDs and clinical diagnostic tools, it remains unclear whether they can simulate alternative visual percepts. This gap creates a risk of exclusion: an LVLM providing color-based instructions (e.g., "take the green line") may fail to provide actionable information for a user with red-green color blindness, highlighting a disconnect between linguistic knowledge and perceptual grounding.

Methodology
The authors utilize the Ishihara Test, a clinically validated tool for diagnosing human CVDs, as a diagnostic probe for LVLMs. Rather than using the test solely for screening, they repurpose it to evaluate whether models can reproduce the perceptual outcomes experienced by individuals with specific CVDs. The study evaluates six LVLMs (Phi-3.5, Qwen2.5-VL, mPLUG-Owl3, LLaVA-NeXT, Llama-3.2, and GPT-4o) across four simulated vision conditions: Normal, Protanopia, Deuteranopia, and Tritanopia.

The evaluation is conducted through three complementary perspectives:

1. Generation Level: The model is prompted to identify the digit on an Ishihara plate while simulating a specific vision type (e.g., "You are Protanopic"). Accuracy is measured against ground-truth digits associated with that vision condition.
2. Confidence Level: The authors analyze per-token perplexity for forced-decoded answers. This measures whether the model's uncertainty aligns with the perceptual ambiguity experienced by humans with CVDs.
3. Internal Representation Level: Using a layer-wise LogitLens probing method, the authors examine whether internal activations encode distinctions between vision types, even if the final output does not reflect them.

To ensure robustness, the study employs multiple prompt strategies (Base, Linguistic Support, Visual Support) and includes control experiments using a medical-specialized model (MedVLM-R1) and a model fine-tuned on synthetic dot patterns (Qwen-DotPattern) to rule out pattern memorization or domain knowledge as sole explanations for performance.

Key Results

• Linguistic Knowledge vs. Perceptual Simulation: All evaluated models demonstrated strong linguistic knowledge, correctly defining the Ishihara Test and various CVDs. However, they failed to translate this knowledge into perceptual simulation.
• Generation Failure: Under simulated CVD conditions (Protanopia and Deuteranopia), model accuracy collapsed. No model exceeded 24% accuracy in these conditions. Instead of generating the digit perceived by a color-blind individual, models defaulted to the "normative" (Normal vision) answer or produced hallucinated outputs.
• Ineffectiveness of Prompts: Providing linguistic descriptions of the impairment or visual few-shot examples (Visual Support) did not significantly improve performance. Models consistently reverted to normative perception regardless of the prompt framing.
• Confidence Misalignment: Perplexity analysis revealed a pronounced bias toward normal vision. Models exhibited low perplexity (high confidence) for Normal and Tritanopia conditions but significantly higher perplexity for Protanopia and Deuteranopia, indicating fundamental confusion when attempting to simulate altered color vision.
• Internal Representations: Layer-wise probing showed that internal activations did not consistently encode distinctions between vision types. Models often processed images as "Normal" in mid-layers, shifting only at the final layer, suggesting a lack of deep, alternative perceptual states in the latent space.
• Specialized Training: Models trained on medical data or synthetic dot patterns showed improved performance in Normal vision but failed to reproduce CVD perceptual distortions, confirming that domain-specific training does not inherently enable the simulation of alternative perceptual experiences.

Key Contributions

• Diagnostic Framework: The paper establishes a methodology for evaluating LVLMs' ability to simulate human perceptual diversity using the Ishihara Test, moving beyond simple visual recognition to perceptual alignment.
• Empirical Evidence of Limitation: The study provides systematic evidence that current LVLMs lack mechanisms for representing alternative perceptual experiences, despite possessing factual knowledge about them.
• Multi-Level Analysis: By combining generation, confidence, and internal representation analyses, the authors demonstrate that the failure to simulate CVDs is not merely a generation error but a fundamental misalignment in how models process and ground visual information.

Significance and Claims
The authors claim that their findings indicate current LVLMs default to a "standard" visual experience, effectively silencing users with different perceptual characteristics in multimodal interactions. The paper argues that the ability to explain CVDs in language is distinct from the ability to perceive colors as a CVD user would. Consequently, the deployment of these systems in accessibility-critical domains (e.g., navigation for the visually impaired) raises concerns regarding fairness and inclusion. The authors conclude that future multimodal systems require new mechanisms to represent and accommodate perceptual diversity, as current architectures are insufficient for human-centered, inclusive deployment. The study explicitly states it is diagnostic rather than prescriptive, aiming to highlight the need for further research into CVD-aware LVLMs rather than offering immediate mitigation strategies.