Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision

This paper introduces a new framework of psychophysical tests based on low-level vision principles—specifically contrast sensitivity, masking, and matching—to evaluate and reveal the perceptual strengths and weaknesses of 34 existing image and video quality metrics, demonstrating that standard evaluation protocols often fail to capture these fundamental human visual properties.

The Gist

Imagine you are a judge in a cooking competition. Your job is to taste two dishes and decide which one looks and tastes better. But instead of tasting, you are a computer program trying to judge the quality of images and videos.

For years, we've had these computer judges (called Quality Metrics) like SSIM, LPIPS, and VMAF. They are supposed to act like human eyes, telling us if a video is blurry, if a photo has bad colors, or if a movie looks too grainy. Usually, we test them by asking real humans, "Which one do you like better?" and seeing if the computer agrees.

But here's the problem: Just because a computer gets the "right answer" on a specific test doesn't mean it understands why humans see things the way they do. It might be guessing correctly by accident, or it might be using a completely different (and wrong) logic.

This paper introduces a new way to test these computer judges. Instead of just asking "Did you get the score right?", the authors ask, "Do you see the world like a human eye actually works?"

They use a set of "vision puzzles" based on how our brains and eyes are biologically wired. Think of it as a medical exam for the computer's "eyes."

Here are the four main puzzles they used, explained with simple analogies:

1. The "Whisper vs. Shout" Test (Contrast Detection)

The Human Reality: Our eyes aren't equally good at seeing everything. We are terrible at seeing very faint details in the dark, and we are also bad at seeing tiny, high-pitched details (like fine sand). We are actually best at seeing medium-sized details (like the texture of a brick wall). This is called the Contrast Sensitivity Function (CSF).

The Test: The researchers showed the computer images with patterns of different sizes and brightness levels, asking, "Can you see this?"
The Result:

• Old judges (like PSNR): They act like a robot counting pixels. They think a tiny speck of dust is just as important as a giant blurry smudge. They don't understand that humans ignore tiny specks.
• The "Good" judges (like ColorVideoVDP): These actually mimic the human eye, knowing that medium-sized details are the most important.
• The "Bad" judges (like SSIM): Surprisingly, SSIM is obsessed with tiny, high-frequency details. It's like a judge who cares more about a single grain of salt on a cake than the fact that the cake is burnt.

2. The "Noise vs. Signal" Test (Contrast Masking)

The Human Reality: Imagine you are trying to hear a whisper in a quiet library (easy). Now imagine trying to hear that same whisper in a loud rock concert (impossible). The loud music "masks" the whisper. In vision, if an image is already busy and textured (like a forest), a small distortion (like a scratch) is harder to see than if that scratch was on a plain white wall.

The Test: The researchers showed the computer a busy, noisy image and asked, "How much of a new scratch do I need to add before you notice it?"
The Result:

• Most judges: They didn't care about the background noise. They thought a scratch was equally visible on a plain wall and a busy forest.
• The "Deep Learning" judges (like LPIPS): These were the surprise stars! Even though they were never taught about "masking," they learned from looking at millions of photos that "busy backgrounds hide small errors." They got this right, just like humans do.

3. The "Flicker" Test (Temporal Sensitivity)

The Human Reality: If you wave a flashlight back and forth very fast, it looks like a steady light. If you wave it slowly, you see it flickering. Our eyes have a "sweet spot" for flickering; we are most sensitive to a specific speed (about 8 times a second).

The Test: The researchers showed the computer videos that flickered at different speeds.
The Result:

• Most video judges: They are terrible at this. They usually only look at two or three frames at a time, so they can't tell the difference between a slow flicker and a fast one.
• The "Good" judges: Only a few specialized video metrics could actually tell, "Hey, this is flickering at the speed humans hate the most!"

4. The "Big vs. Small" Test (Contrast Constancy)

The Human Reality: This is the weirdest one. If you look at a faint gray line, it looks very different depending on its size. But if you look at a very bright, high-contrast line, it looks the same size and brightness whether it's tiny or huge. Our brains "flatten" our perception of big, obvious things.

The Test: The researchers asked the computer to match the brightness of a faint line to a bright line.
The Result:

• Everyone failed: None of the computer judges understood this. They kept trying to mathematically calculate the difference, failing to realize that for humans, "big and bright" looks the same regardless of size. This is a major blind spot for current technology.

The Big Takeaway

The authors built a "Vision Gym" to test these computer judges. They found that:

1. Old formulas (like SSIM) are often wrong about what humans care about (they love tiny details too much).
2. AI judges (Deep Learning) are surprisingly good at some things (like masking) because they learned from real photos, even without being taught the rules.
3. Video judges are generally bad at understanding time (flicker).
4. Nobody understands how we perceive "big, bright" things (contrast constancy) yet.

Why does this matter?
If you are streaming a movie on Netflix, you want the video to look good without using too much data. If the computer judge thinks a blurry forest is "bad" (when humans can't even see the blur), the system wastes data trying to fix it. If the judge thinks a flickering light is "fine" (when humans hate it), the movie will look terrible.

By using these "vision puzzles," the authors hope to help engineers build better judges that truly understand how human eyes work, leading to better pictures and videos for everyone.

Technical Summary

Here is a detailed technical summary of the paper "Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision."

1. Problem Statement

Objective image and video quality metrics (e.g., PSNR, SSIM, LPIPS, VMAF) are essential for evaluating compression, processing algorithms, and Quality of Experience (QoE). Currently, these metrics are primarily validated by measuring their correlation with subjective human scores (Mean Opinion Scores - MOS).

However, this correlation-based approach has significant limitations:

• Lack of Insight: High correlation does not explain why a metric performs well or poorly, nor does it reveal the specific visual characteristics the metric captures.
• Data Variability: Results vary across datasets due to observer variability, noise, and differences in content or distortion types.
• Black Box Nature: Modern deep-learning-based metrics have internal representations that are difficult to interpret.
• Missing Low-Level Models: Few metrics explicitly incorporate established models of human low-level vision (e.g., contrast sensitivity, masking), relying instead on hand-crafted formulas or data-driven training.

The paper argues that to truly understand and improve these metrics, they must be evaluated against psychophysical measurements of low-level vision, independent of subjective rating datasets.

2. Methodology

The authors propose a novel evaluation framework that mimics classical psychophysical experiments. Instead of comparing metrics to subjective scores, they compare metric outputs to established psychophysical models and human data regarding low-level vision properties.

Experimental Setup:

• Stimuli: Controlled stimuli (Gabor patches, sinusoidal gratings, noise, flickering disks) generated in linear color space and converted to sRGB.
• Physical Parameters: Assumed display resolution (~60 pixels/degree), peak luminance (100 cd/m²), and mean luminance (21.4 cd/m²).
• Metric Role: The quality metric acts as the "observer." The framework varies stimulus parameters (contrast, spatial frequency, temporal frequency, color direction) and measures the metric's response.

Four Key Test Categories:

1. Contrast Detection:
   • Task: Determine the threshold at which a pattern (Gabor patch) becomes visible against a uniform background.
   • Reference: castleCSF model (Contrast Sensitivity Function).
   • Goal: Check if the metric's sensitivity curve matches the human band-pass sensitivity (peak sensitivity at 2–4 cycles/degree).
2. Contrast Masking:
   • Task: Measure how a "masker" (background texture/noise) affects the detection threshold of a test pattern.
   • Types: Phase-coherent (facilitation/dipper effect) and Phase-incoherent masking.
   • Reference: Human data from [12] and [13].
   • Goal: Assess if the metric correctly predicts that visibility decreases as masker contrast increases (masking) and if it captures the "dipper" effect at low contrasts.
3. Flicker Detection (Temporal):
   • Task: Detect temporal modulation (flicker) in a video.
   • Reference: elaTCSF model (Temporal Contrast Sensitivity Function).
   • Goal: Verify if the metric detects flicker with a band-pass characteristic peaking around 8 Hz.
4. Contrast Matching (Supra-threshold):
   • Task: Determine if the metric perceives the magnitude of contrast as equal across different spatial frequencies or color directions.
   • Reference: Human data from Georgeson & Sullivan [15] (spatial) and Switkes et al. [16] (color).
   • Goal: Test for Contrast Constancy—the phenomenon where perceived contrast differences across frequencies diminish at high (supra-threshold) contrast levels.

Evaluation Metrics:

• Alignment Score (AS): Correlation between metric predictions and psychophysical thresholds (for detection tests).
• RMSE: Root Mean Square Error between predicted and true matching contrasts (for matching tests).

3. Key Contributions

• New Evaluation Framework: Introduced a suite of tests specifically designed to probe low-level vision capabilities (detection, masking, constancy) in quality metrics.
• Comprehensive Benchmarking: Applied these tests to 34 existing full-reference metrics, including traditional (PSNR, SSIM), color difference (CIEDE2000), and deep-learning-based (LPIPS, DISTS, TOPIQ) models.
• Open-Source Tool: Released an open-source framework to allow the community to replicate these tests.
• Diagnostic Insights: Provided a structured way to visualize metric behavior (e.g., contour plots) that reveals properties invisible to standard correlation-based evaluations.

4. Key Results & Findings

A. Contrast Detection (Spatial Frequency)

• Traditional Metrics: PSNR and simple color differences show no frequency dependence. SSIM exhibits a high-pass response (overemphasizing high frequencies), while MS-SSIM improves this to a band-pass response but still deviates from the human CSF.
• Deep Learning: Metrics like LPIPS and DISTS show band-pass characteristics but often with peak sensitivities at incorrect frequencies.
• Best Performer: ColorVideoVDP aligns best with the human CSF because it is explicitly built upon the castleCSF model.

B. Contrast Masking

• Traditional Metrics: Most (PSNR, SSIM, FLIP) show no sensitivity to masking; their discrimination thresholds remain invariant to masker contrast.
• Deep Learning: LPIPS and DISTS (based on VGG/AlexNet) surprisingly capture the "dipper" effect (facilitation) and strong masking at high contrasts, outperforming many traditional metrics. This suggests deep features naturally encode masking effects.
• Transformers: Metrics like TOPIQ and AHIQ fail at low-contrast masking (near threshold) but work at supra-threshold levels, likely due to insufficient sensitivity to subtle variations.
• VMAF: Only models masking at supra-threshold contrasts, ignoring near-threshold effects.

C. Flicker Detection (Temporal)

• Poor Performance: Most video metrics (including VMAF) fail to model temporal sensitivity correctly. They often use small temporal filters that cannot distinguish between different temporal frequencies.
• Success: Only FovVideoVDP and ColorVideoVDP successfully predict the band-pass flicker sensitivity curve.

D. Contrast Matching (Supra-threshold)

• Contrast Constancy Failure: A critical finding is that none of the 34 tested metrics successfully model contrast constancy.
  • Human vision perceives contrast as constant across frequencies at high contrast levels (flat lines).
  • All metrics either maintain frequency dependence (parallel lines) or fail to match the reference entirely.
• Color Matching: Metrics generally struggle to balance achromatic vs. chromatic contrast. CIEDE2000 underestimates chromatic differences at high contrasts, while deep learning metrics often overemphasize them. ColorVideoVDP achieved the closest match.

5. Significance

This paper shifts the paradigm of quality metric evaluation from "correlation with subjective scores" to "alignment with human visual system principles."

• Diagnostic Power: The tests reveal specific failure modes (e.g., SSIM overemphasizing high frequencies, VMAF ignoring near-threshold flicker) that standard MOS correlation hides.
• Guidance for Development: The results suggest that while deep learning metrics have implicitly learned some masking properties, they still lack explicit modeling of supra-threshold contrast constancy and precise temporal sensitivity.
• Future Directions: The authors advocate for incorporating explicit psychophysical models (like CSF and masking models) into the architecture of future metrics, rather than relying solely on data-driven training.

In summary, the paper demonstrates that current state-of-the-art metrics, even deep learning-based ones, often fail to capture fundamental low-level visual phenomena, providing a roadmap for developing more perceptually accurate quality assessment tools.