🤖 machine learning

Prominence-Aware Artifact Detection and Dataset for Image Super-Resolution

This paper introduces a novel approach to artifact detection in single-image super-resolution that prioritizes human-perceived prominence over uniform defect classification, supported by a new annotated dataset and a lightweight regressor that effectively guides artifact suppression.

Original authors: Ivan Molodetskikh, Kirill Malyshev, Mark Mirgaleev, Nikita Zagainov, Evgeney Bogatyrev, Dmitriy Vatolin

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ivan Molodetskikh, Kirill Malyshev, Mark Mirgaleev, Nikita Zagainov, Evgeney Bogatyrev, Dmitriy Vatolin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a blurry, low-quality photo of your favorite vacation spot. You use a fancy AI tool to "super-resolve" it, making it look crisp and high-definition. But sometimes, these AI tools get a little too creative. They might turn a smooth patch of grass into a weird, wavy pattern, or give a person's face a slightly melted look. These mistakes are called artifacts.

For a long time, researchers treated all these mistakes the same way: "Is there a mistake? Yes/No." But this paper argues that's like judging a movie by counting every single typo in the script, regardless of whether the typo is in a boring footnote or a crucial plot twist.

Here is the simple breakdown of what this paper does, using some everyday analogies:

1. The Problem: Not All Mistakes Are Created Equal

Imagine you are looking at a painting.

Scenario A: The artist accidentally painted a weird, jagged line right in the middle of a human face. You can't look away; it's disturbing.
Scenario B: The artist painted a slightly weird texture on a patch of grass in the background. You might not even notice it unless you were looking for it.

Previous AI tools treated both of these as "100% bad." This paper says: "Wait a minute! The face mistake is a 10/10 disaster, but the grass mistake is maybe a 2/10 annoyance."

The authors call this "Prominence." It's a measure of how much a mistake bothers a human eye.

2. The New Dataset: The "Crowdsourced Complaint Box"

To fix this, the researchers didn't just ask a computer to find mistakes. They asked humans.

They took 1,302 examples of AI-generated images with mistakes.
They showed them to hundreds of people on a crowdsourcing platform (like a digital town square).
They asked: "Do you see this weird spot? Does it bother you?"
They calculated a "Prominence Score" (0% to 100%) based on how many people noticed and disliked the error.

The Big Surprise: They checked an existing database of "bad AI images" and found that 48% of the "mistakes" listed there were actually invisible to most people. The old tools were crying wolf about things that didn't matter.

3. The Solution: The "Heatmap Detective"

The researchers built a new, lightweight AI detective. Instead of just drawing a box around a mistake, this detective creates a heat map.

Red areas: "Hey, look here! This is a terrible mistake that ruins the picture!" (High Prominence)
Yellow areas: "There's a small glitch, but it's kind of hidden." (Medium Prominence)
Green areas: "Everything looks fine here."

This detective is trained on the human feedback data. It learns that a weird pattern on a face is a big deal, but a weird pattern on a rock or water is usually fine.

4. Why This Matters: The "Fine-Tuning" Chef

The coolest part is how they use this detective. Imagine you are a chef (the AI model) trying to cook a perfect steak (the high-res image).

Old way: The chef gets a list saying, "You messed up 50 times." The chef tries to fix all 50, wasting time on tiny, invisible crumbs on the floor while the burnt steak remains.
New way: The chef gets a heat map saying, "Focus on the burnt part of the steak (Red zone). Ignore the crumbs on the floor (Green zone)."

The researchers showed that by using their "Prominence Heatmap" to guide the AI, the AI learns to fix the annoying mistakes much faster and better than before.

5. The "Fake Reference" Trick

Usually, to check if an AI is good, you need the original, perfect high-resolution photo to compare it against. But in the real world (like restoring old photos), you don't have the original.

The Trick: The researchers used a very fast, simple AI to make a "good enough" version of the photo. They treated this "good enough" version as the "perfect" reference.
The Result: It worked surprisingly well! It's like using a sketch to judge a painting; it's not perfect, but it's good enough to spot the major errors without needing the original masterpiece.

Summary

This paper is about teaching computers to be more human about mistakes.

Old Logic: "Any error is bad."
New Logic: "Only the errors that actually annoy people are bad."

By focusing on what humans actually notice, they built a better system to fix AI images, making them look more natural and less "glitchy" where it counts.

1. Problem Statement

Single-Image Super-Resolution (SISR) has advanced significantly with Generative Adversarial Networks (GANs), Transformers, and Diffusion models. However, these generative approaches often introduce visual artifacts (e.g., unnatural patterns, smeared faces, texture distortions) that degrade perceived quality.

The Core Limitation: Existing artifact detection methods (e.g., LDL, DeSRA) treat all artifacts as binary defects (present or absent). They fail to account for prominence—the degree to which an artifact is noticeable or disturbing to a human observer.

High Prominence: Artifacts on structured objects (faces, buildings) are highly disturbing.
Low Prominence: Artifacts on organic textures (water, grass) are often unnoticeable.
Treating these equally leads to detection methods that may overfit to trivial artifacts while missing critical ones, or fail to guide model fine-tuning effectively.

2. Methodology

The authors propose a Prominence-Aware framework consisting of a new dataset, a crowdsourced annotation pipeline, and a lightweight detection model.

A. Dataset Construction

New Dataset: Created 1,302 artifact examples from 500 source images (Open Images) processed by 11 contemporary SISR methods (including GFPGAN, SwinIR, StableSR, SUPIR).
Existing Data Re-annotation: Annotated 593 artifacts from the existing DeSRA dataset.
Annotation Protocol:
- Used Toloka.ai for crowdsourcing.
- Participants viewed pairs of "Original" vs. "Upscaled" images with highlighted artifact regions.
- Prominence Score: Calculated as the percentage of annotators (30 per image) who confirmed the presence of a distortion.
- Mask Post-processing: To ensure annotators could see the context, tight binary masks were morphologically expanded (dilated) before display, improving detection accuracy by ~1.7%.
Key Finding: Analysis revealed that 48% of artifacts in the DeSRA dataset have a prominence score below 50%, meaning most viewers do not notice them. This validates the need for a prominence-based approach over binary masks.

B. Artifact Prominence Metric (The Model)

The authors developed a lightweight regressor that outputs a spatial prominence heatmap (values 0–1) for an SR image.

Input Features: The model aggregates three features computed block-wise:
- DISTS: A perceptual metric capturing texture distortions.
- $ssm_{jup}$ : A structure similarity map adapted from LDL, modified to use all RGB channels (not just chrominance) to detect luminance-correlated texture distortions.
- $bd_{jup}$ : A weighted sum of LPIPS (perceptual quality) and ERQA (edge restoration quality).
Fusion Module: A shallow Multilayer Perceptron (MLP) (3-128-128-1 architecture) processes the feature heatmaps pixel-wise to predict the final prominence score.
Pseudo-GT Strategy: To enable real-world application where High-Resolution (HR) Ground Truth (GT) is unavailable, the authors use a lightweight, artifact-resistant SR model (RLFN) to generate a "Pseudo-GT." Full-reference metrics are then calculated between the target SR output and this Pseudo-GT.

C. Training Objective

The model is trained to minimize the L2 loss between:

The predicted prominence inside the binary artifact mask (matching the crowdsourced GT score).
The predicted prominence outside the mask (forced to 0).

3. Key Contributions

First Prominence-Annotated Dataset: A novel dataset of 1,302 SISR artifacts with crowdsourced prominence scores, plus re-annotation of the DeSRA dataset.
Prominence-Aware Detection Method: A lightweight model that outperforms existing binary detectors by quantifying how noticeable an artifact is, rather than just if it exists.
Pseudo-GT Framework: Demonstrated that lightweight SR models (like RLFN) can serve as effective pseudo-GT for full-reference metrics, making the approach applicable in real-world scenarios without HR references.
Fine-Tuning Guidance: Showed that using prominence heatmaps to guide SR model fine-tuning (replacing artifact regions with Pseudo-GT) significantly reduces visible artifacts.

4. Experimental Results

A. Objective Evaluation

Performance: The proposed method achieved the best average ranking in PR-AUC (Precision-Recall Area Under Curve) across various reference inputs (Original HR, SPAN, RLFN) and datasets (Proposed and DeSRA).
Comparison: It outperformed state-of-the-art detectors like LDL, DeSRA, DISTS, and PAL4VST, particularly in identifying prominent artifacts.
Generalization: Cross-validation experiments showed the model generalizes well across different SR architectures (CNN, Transformer, Diffusion).

B. Downstream Task: Artifact Suppression

Fine-Tuning: The authors fine-tuned SR models (RealESRGAN, LDL, SwinIR) using the proposed metric to guide artifact removal.
Outcome: The proposed method yielded the highest $\Delta$ IoU (reduction in artifact overlap with GT) and the best Remimg (artifact removal rate) while minimizing Addimg (introduction of new artifacts).
Visual Quality: Fine-tuned models produced images with significantly fewer distracting artifacts compared to baselines.

C. Analysis of SR Models

The study evaluated 11 SR models for artifact propensity:

Best: Transformer-based models (DRCT, HAT-L) showed the lowest mean prominence and fewest confident artifacts.
Worst: Diffusion-based models (SUPIR, SeeSR) and GANs (RealESRGAN) were found to be highly susceptible to generating prominent artifacts, even the latest high-quality models.

5. Significance and Impact

Paradigm Shift: Moves the field from binary artifact detection to perceptual prominence modeling, aligning evaluation with human visual perception.
Practical Utility: The ability to use Pseudo-GT allows for artifact detection and model fine-tuning in real-world applications where HR ground truth is unavailable.
Broader Applicability: The authors demonstrated transferability to JPEG AI compression artifacts, suggesting the prominence modeling approach is applicable to various image restoration and generation tasks.
Future Direction: Highlights the need to address semantic artifacts (e.g., object hallucination) in next-generation models, which current texture-based metrics may miss.

In summary, this paper provides the data, the metric, and the methodology to move SISR evaluation beyond pixel fidelity and binary defect detection, focusing instead on the human-perceived severity of visual artifacts.