Prompt-Driven Color Accessibility Evaluation in Diffusion-based Image Generation Models

This paper introduces "CVDLoss," a novel metric for evaluating color accessibility in diffusion-based image generation, and demonstrates that current models struggle to reliably respond to accessibility-focused prompts despite the metric's sensitivity to structural detail changes.

The Gist

Imagine you have a magical artist named Diffusion. This artist is incredibly talented; you can whisper a description like "a bowl of fruit," and they instantly paint a masterpiece that looks vibrant, detailed, and beautiful to your eyes.

But here's the problem: Not everyone sees the world the same way.

About 1 in 12 men (and fewer women) have Color Vision Deficiency (CVD), commonly known as color blindness. For them, the artist's "vibrant" reds and greens might look like muddy browns or identical shades of gray. A beautiful fruit bowl might turn into a confusing blur where the apple and the leaf are impossible to tell apart.

This paper is a report card for the magical artist, asking a simple question: "If we just ask the artist nicely to paint for color-blind people, will they do it?"

Here is the breakdown of their investigation, explained with some everyday analogies.

1. The Experiment: The "Magic Prompt" Test

The researchers didn't change the artist's brain (the AI model). Instead, they tried to change the instructions (the prompts) they gave the artist. They tested four different ways of asking:

• The Standard Ask: "A bowl of fruit." (The artist paints whatever they want).
• The General Ask: "A bowl of fruit with a colorblind-friendly palette." (The artist tries to be nice).
• The Specific Ask (Red-Blind): "A bowl of fruit for someone who can't see red."
• The Specific Ask (Green-Blind): "A bowl of fruit for someone who can't see green."

They asked the artist to paint 320 different scenes, ranging from candy shops and coral reefs to street views and cartoons.

2. The New Ruler: "CVDLoss"

How do you measure if a painting is "accessible"? You can't just ask a computer to look at it, because computers usually see in "normal" colors.

The researchers invented a new ruler called CVDLoss.

The Analogy: Imagine you are looking at a map with clear, sharp roads (edges and textures). Now, imagine putting on a pair of foggy glasses (simulating color blindness).

• If the roads on the map blur together or disappear behind the fog, the map is hard to read.
• If the roads stay crisp and clear even through the fog, the map is accessible.

CVDLoss measures exactly how much the "roads" (the edges and textures) get blurry or disappear when you put on those foggy glasses.

• High Score: The image looks totally different and confusing to a color-blind person. (Bad!)
• Low Score: The image looks almost the same to a color-blind person as it does to a normal person. (Good!)

3. The Results: The Artist is Confused

The researchers found that simply asking the artist to be inclusive didn't work very well. In fact, it was a mixed bag:

• The "Candy" Effect: For some colorful things (like candy), the artist actually did a better job when asked to be inclusive. The colors shifted in a way that helped.
• The "Flower" Disaster: For other things (like flowers), asking the artist to be inclusive made things worse. The artist got confused, changed the colors in weird ways, and accidentally made the petals and leaves harder to tell apart.
• The "Street View" Rollercoaster: For complex scenes like city streets, the results were all over the place. Sometimes it helped, sometimes it hurt.

The Big Takeaway: The magical artist (the AI) wasn't trained to understand "accessibility." It's like asking a chef who only knows how to cook spicy food to "make it mild." Sometimes they add water and it works; other times, they add sugar and it tastes weird. The artist doesn't truly know what "colorblind-friendly" means; they are just guessing.

4. Why This Matters

The paper concludes that we can't just rely on "magic words" (prompts) to fix accessibility in AI art. If we want AI to create images that everyone can enjoy, we need to:

1. Teach the artist better: Train the AI specifically on how color blindness works, not just hope it guesses right.
2. Use the new ruler: Use CVDLoss to check the work before we show it to the public. It acts like a quality control inspector that says, "Hey, this looks great to you, but it's a mess to someone with red-blindness. Let's fix it."

Summary

Think of this paper as a warning label for the future of AI art. It tells us: "Don't just assume the AI is being inclusive because you asked nicely. It's often just making things up."

The researchers gave us a new tool (CVDLoss) to measure the "blur" caused by color blindness, proving that while AI is amazing at making pretty pictures, it still needs a lot of help to make sure those pictures are clear for everyone.

Technical Summary

Here is a detailed technical summary of the paper "Prompt-Driven Color Accessibility Evaluation in Diffusion-based Image Generation Models" by Zhuang, Echevarria, and Ak¸sit.

1. Problem Statement

While diffusion-based generative models (e.g., Stable Diffusion) excel at creating high-quality, diverse images from text prompts, they lack inherent mechanisms to ensure color accessibility for users with Color Vision Deficiencies (CVD), such as protanopia (red-blindness) and deuteranopia (green-blindness).

• Limitations of Current Approaches: Existing accessibility guidelines rely heavily on luminance contrast, which fails to capture perceptual conflicts arising from hue and saturation differences critical for CVD users.
• The Research Question: Can generative models effectively perform accessibility-aware color transformations solely through prompt engineering (e.g., adding "colorblind-friendly" to the prompt), or do they require explicit post-processing?
• The Gap: There is a lack of systematic evaluation metrics that quantify how CVD simulations alter the perceptual structure (edges, textures, gradients) of generated images, beyond simple color shifts.

2. Methodology

The authors propose a systematic framework involving dataset construction, CVD simulation, and a novel evaluation metric.

A. Dataset Construction & Prompt Design

• Model: Images were generated using Stable Diffusion 3.5-large.
• Categories: Eight semantically distinct categories were selected to cover both color-dominant (e.g., candy, flower, fruit) and structure-dominant (e.g., cartoon, street view) scenes.
• Prompt Strategies: Four types of prompts were tested for each category (10 images each, 320 total):
  1. Standard: Basic visual description (e.g., "A bowl of fruit").
  2. Colorblind-aware: Augmented with "with red-green colorblind palette."
  3. Protanopia-aware: Augmented with "with protanopia-friendly palette."
  4. Deuteranopia-aware: Augmented with "with deuteranopia-friendly palette."

B. CVD Simulation

• Simulation Model: Used the physiologically based model by [VBM99] (via the DaltonLens library) to simulate protanopia and deuteranopia at maximum severity.
• Protocol:
  • For deficiency-specific prompts, only the corresponding simulation was applied.
  • For standard and general "colorblind" prompts, both simulations were computed.

C. The Proposed Metric: CVDLoss

To move beyond luminance-only metrics, the authors introduced CVDLoss, a gradient-based metric designed to quantify the degradation of structural cues (edges, textures) under CVD simulation.

• Mechanism: It compares the Gradient Magnitude Maps (GMM) of the original image ($I$) and its CVD-simulated counterpart ($I_{CVD}$).
• Color Space: Utilizes OKLab color space and HyAB color differences to capture 3D perceptual color shifts (hue/saturation) rather than just luminance.
• Formula:
  $$\text{CVDLoss}(I, I_{CVD}) = \frac{\sum_p (G(I)_p - G(I_{CVD})_p)^2}{N \cdot \max_p G(I)_p^2}$$
  Where $G(\cdot)$ is the gradient magnitude map, $p$ is the pixel index, and $N$ is the total pixel count.
• Interpretation: A lower CVDLoss indicates that the structural perception remains similar between a normal viewer and a CVD viewer. A higher value indicates significant structural distortion or loss of detail for the CVD viewer.

D. Synthetic Verification

To validate CVDLoss, the authors applied daltonization (a post-processing technique to remap colors for CVD) to standard prompt images.

• Hypothesis: A successful daltonization should preserve local gradients, resulting in a lower CVDLoss compared to the original.
• Result: The metric successfully detected reduced gradient loss in protanopia simulations after daltonization, confirming its sensitivity to accessibility-oriented transformations.

3. Key Results

The study evaluated the effectiveness of prompt engineering across the 8 categories and 2 CVD types using normalized CVDLoss values.

• Inconsistency of Prompt Engineering: There is no universal positive trend. Prompt-based interventions are neither reliable nor uniformly successful.
  • Category Dependence:
    • Candy: Generally benefited from accessibility prompts (reduced CVDLoss).
    • Flower: Consistently showed increased CVDLoss, suggesting that prompt-driven color reinterpretation disrupted local structures in this category.
    • Street View: Showed significant improvement with deficiency-specific prompts, returning to distributions comparable to standard prompts but with lower structural loss.
    • Cartoon/Poster: Showed instability under general "colorblind-aware" prompts (widened distribution, higher loss), though deficiency-specific prompts offered partial mitigation.
  • Deficiency Asymmetry: In categories like fruit, prompts for protanopia and deuteranopia produced opposite effects, highlighting that accessibility must be evaluated separately for each deficiency type.
• Model Limitations: Diffusion models are not explicitly trained on accessibility constraints. Consequently, they interpret accessibility language inconsistently, often introducing instability rather than ensuring accessibility.

4. Key Contributions

1. CVDLoss Metric: Introduction of a novel, gradient-based metric that quantifies the loss of perceptual structure (edges and textures) under CVD simulation, addressing the limitations of luminance-only guidelines.
2. Systematic Evaluation: The first large-scale study evaluating the efficacy of prompt-driven accessibility in diffusion models across diverse content categories and specific CVD types.
3. Empirical Evidence: Demonstration that prompt engineering alone is insufficient for reliable color accessibility, as models lack the training to consistently satisfy these constraints without explicit supervision.
4. Validation: Successful validation of CVDLoss against daltonization, proving its utility as a diagnostic tool for measuring structural preservation in accessibility workflows.

5. Significance and Future Work

• Practical Implication: The study establishes that relying solely on text prompts to generate accessible images is risky and unpredictable. It underscores the continued necessity of post-processing or model fine-tuning with explicit accessibility supervision.
• Tooling: CVDLoss provides a quantitative tool for developers to evaluate and improve accessibility in generative pipelines and post-processing algorithms.
• Limitations & Future Directions:
  • The study is limited to a single diffusion model and specific prompt phrasings.
  • CVDLoss focuses on adjacent regions (gradients) and does not yet address semantic conflicts in non-adjacent regions.
  • Future work should involve user studies with actual CVD individuals and the exploration of alternative daltonization pipelines to refine the metric.

In conclusion, while diffusion models offer powerful creative tools, they currently fail to reliably generate color-accessible content via prompts alone. The proposed CVDLoss metric offers a critical step toward objectively measuring and improving structural accessibility in generative AI.