Position: Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered

This position paper argues that the evaluation of modern visual processing systems must shift from a reliance on single-metric benchmarks toward a human-centered, context-aware paradigm to better align with human perception and foster genuine innovation.

The Gist

Imagine you are a chef trying to create the world's most delicious soup. For years, the only way to judge your soup was by measuring its temperature with a thermometer. If the thermometer said "90°C," you were a genius. If it said "85°C," you were a failure.

But here's the problem: Temperature doesn't taste like flavor.

This is exactly the situation the paper "Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered" is describing for the world of computer vision (specifically, fixing blurry or damaged photos).

Here is the breakdown of their argument using simple analogies:

1. The Old Way: The "Thermometer" of Images

For a long time, researchers improved image restoration (making blurry photos sharp) by trying to beat a computer score called PSNR or SSIM.

• The Analogy: Think of these scores like a ruler measuring how closely a copy matches the original. If you copy a drawing and the lines are 99% identical to the original, you get a high score.
• The Problem: In the real world, we don't just want a perfect copy; we want a beautiful picture. Sometimes, to make a photo look amazing to a human eye, you have to change it slightly (add a bit of texture, fix a face, make colors pop). But the "ruler" hates change. It thinks, "Hey, you changed the pixels! That's a bad job!" So, it gives you a low score, even though the photo looks great to us.

2. The New Problem: The "Fake Sharpness" Trap

Recently, computers got smarter. They started using "Generative AI" (like GANs and Diffusion models) to invent new details that weren't in the original photo.

• The Analogy: Imagine a painter who doesn't just copy a photo but adds realistic fur to a dog or sharp eyes to a person. The picture looks incredible.
• The Glitch: The computer scores (like LPIPS) started to get confused. They saw all the new "sharp" details and thought, "Wow, so much detail! High score!"
• The Danger: This created a trap. Researchers started "cheating" the system. They would make images too sharp, or add weird, noisy textures just to trick the computer into giving them a higher score. It's like a student memorizing the answer key instead of actually learning the subject. The score goes up, but the actual quality (the "taste" of the soup) gets worse.

3. The "One-Size-Fits-All" Mistake

The paper argues that we are trying to judge every type of photo with the same single number.

• The Analogy: Imagine a single judge at a talent show who only cares about how loud the singer is.
  • If a heavy metal band plays, the judge says, "Great! 10/10!"
  • If a soft jazz singer performs, the judge says, "Terrible! 1/10!"
  • But in reality, the jazz singer might be more beautiful and moving to the audience.
• The Reality: A photo of a cartoon needs to look different than a photo of a human face or a forest. A "perfect" score for a cartoon might look terrible on a human face. We need to judge them based on what they are supposed to be, not just a single number.

4. The Proposed Solution: "The Human Taste Test"

The authors are calling for a change in how we evaluate these AI models. Instead of just looking at the computer's score, we need to put the human back in the driver's seat.

• The Analogy: Instead of just checking the thermometer, we need to invite a panel of food critics (humans) to taste the soup.
• What they want:
  • Context Matters: Ask, "Does this face look real?" "Does this building look stable?" "Does this cartoon look fun?"
  • No More Cheating: Stop rewarding models just for making things "sharper" if it looks unnatural.
  • Better Tools: The computer tools (metrics) need to get smarter. They need to learn to understand what is in the picture (semantics), not just count pixels. They need to realize that a blurry background (depth of field) is a good thing in a portrait, not a mistake to be fixed.

The Bottom Line

The paper says: "Stop letting the scoreboard run the game."

Right now, AI researchers are so obsessed with beating the computer score that they are forgetting to make images that humans actually enjoy looking at. They want to shift the focus back to human perception. The goal isn't to get the highest number on a chart; it's to create images that look beautiful, natural, and useful to real people.

In short: Don't let the robot judge the art. Let the human judge the art, and use the robot only to help, not to decide.

Technical Summary

Based on the position paper "Evaluation of Visual Processing Should Be Human-Centered, Not Metric-Centered" by Hu et al., here is a detailed technical summary covering the problem, methodology, key contributions, results, and significance.

1. Problem Statement

The paper identifies a critical misalignment between objective Image Quality Assessment (IQA) metrics and human perception in the era of generative visual processing (e.g., GANs and Diffusion models).

• The Metric Trap: The field has become overly reliant on single-metric benchmarks (PSNR, SSIM, LPIPS) as the primary driver for research, publication, and method comparison.
• The Divergence: As restoration methods evolved from deterministic regression (pixel-level accuracy) to generative synthesis (semantic realism and high-frequency detail), traditional metrics failed to correlate with human preference.
  • PSNR/SSIM: Favor over-smoothed, pixel-perfect outputs that lack realistic texture.
  • LPIPS: Improved for GANs but still struggles with Diffusion models, often penalizing semantically plausible but structurally different outputs.
• The Consequence: Research is incentivized to "game" metrics (e.g., via over-sharpening or artificial high-frequency injection) rather than improving actual visual quality, leading to a stagnation in perceptual innovation and a disconnect between academic leaderboards and real-world user experience.

2. Methodology and Analysis

The authors employ a multi-faceted analytical approach to substantiate their position:

• Longitudinal Metric Adoption Analysis: They analyzed the adoption rates of various IQA metrics (PSNR, SSIM, LPIPS, FID, MUSIQ, MANIQA, CLIP-IQA) across major image restoration models from 2014 to 2025 (Table 1).
• Quantitative Trend Analysis: Using the SRIQA benchmark (100 images with diverse restoration outputs), they tracked the performance of different model categories (PSNR-oriented, GAN-based, Diffusion-based) against standard metrics.
• Vulnerability Testing: They demonstrated how simple image manipulations (e.g., Unsharp Masking, anti-aliasing toggles) could artificially inflate No-Reference (NR) IQA scores without improving visual quality (Figure 4).
• Scenario-Aware User Studies: Instead of aggregated Mean Opinion Scores (MOS), they conducted fine-grained user preference studies comparing models (e.g., SUPIR vs. HAT) across specific semantic categories (e.g., faces, text, architecture, cartoons) to reveal context-dependent performance (Figure 5).
• Scale Gap Comparison: They compared the data scale, resolution, and parameter count of modern restoration models (e.g., SUPIR, HYPIR) against the datasets and architectures used to train current IQA models (Table 2), highlighting a significant disparity.

3. Key Contributions

• Paradigm Shift Proposal: The paper argues for a fundamental shift from metric-centered to human-centered evaluation. Metrics should be tools for reproducibility, not the ultimate optimization objective.
• Evidence of Metric Failure:
  • Demonstrated that Diffusion-based models, which produce the most visually realistic results, often receive lower scores than GANs or PSNR-oriented models on metrics like LPIPS, despite human preference.
  • Showed that newer learning-based NR metrics (MUSIQ, MANIQA, CLIP-IQA) are susceptible to overfitting high-frequency signals, rewarding over-sharpened or hallucinated textures that humans find unnatural.
• Contextual Evaluation Framework: Introduced the concept of scenario-aware evaluation, proving that a single global score is insufficient. A model may excel at restoring faces but fail at text or stylized cartoons, a nuance lost in aggregate metrics.
• Identification of the "Scale Gap": Highlighted that current IQA models are trained on small, low-resolution datasets with limited semantic diversity, whereas modern restoration models are trained on massive, high-resolution datasets. This gap renders current IQA models incapable of evaluating the semantic consistency of generative outputs.
• Call for Multi-Dimensional Feedback: Advocated for structured, explainable feedback signals (e.g., combining Q-series approaches with Vision-Language Models) rather than single scalar scores.

4. Key Results and Findings

• Metric vs. Human Preference: In user studies, Diffusion models (e.g., SUPIR) were preferred over GANs and PSNR-oriented methods in most real-world scenarios, yet metrics like LPIPS often ranked GANs higher.
• Exploitable Metrics: Simple manipulations like applying Unsharp Masking (USM) increased NR-IQA scores significantly, even when the visual result was perceptually degraded (Figure 4 & 6).
• Semantic Blindness: Current metrics failed to distinguish between intentional blur (e.g., depth-of-field) and degradation, often penalizing natural artistic choices or over-sharpening them to boost scores (Figure 7).
• Adoption Trends: While PSNR and SSIM remain ubiquitous (96.2% and 84.6% adoption), newer metrics like MUSIQ and MANIQA show higher adoption in recent generative works but still suffer from the "high-frequency bias."
• Real-World Success vs. Benchmarks: Models like Real-ESRGAN achieved massive community adoption (34.1k GitHub stars) and practical success despite not centering their claims on IQA scores, proving that leaderboard dominance does not equate to real-world utility.

5. Significance and Future Directions

• Reorienting Research: The paper urges the community to stop optimizing for "leaderboard scores" and start optimizing for user preference, semantic correctness, and visual comfort.
• Evolution of IQA: It calls for the development of next-generation IQA models that:
  • Utilize Large Language Models (LLMs) and Vision-Language Models (VLMs) for semantic-level understanding.
  • Are trained on larger, higher-resolution, and more diverse datasets to match the scale of generative restoration models.
  • Provide fine-grained, multi-dimensional feedback (e.g., separating texture realism from structural plausibility) rather than single scores.
• Standardization of Human Evaluation: Proposes that human evaluation should be systematic, reproducible, and scalable (e.g., using standardized protocols like Swiss-style tournaments) rather than treated as an optional supplementary experiment.

Conclusion: The paper concludes that while objective metrics remain necessary for rapid iteration and large-scale comparison, they must be subordinated to human perception. The future of visual processing lies in human-centered, context-aware, and multi-dimensional evaluation to ensure that technological progress translates to genuine improvements in visual experience.