💻 computer science

PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media

This paper introduces the PROVE framework, comprising the perception-aligned RC-S and RC-T metrics and the PROVE-Bench dataset, to address the limitations of existing evaluation methods by providing a more accurate assessment of object removal coherence in visual media that better aligns with human judgment.

Original authors: Fuhao Li, Shaofeng You, Jiagao Hu, Yu Liu, Yuxuan Chen, Zepeng Wang, Fei Wang, Daiguo Zhou, Jian Luan

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fuhao Li, Shaofeng You, Jiagao Hu, Yu Liu, Yuxuan Chen, Zepeng Wang, Fei Wang, Daiguo Zhou, Jian Luan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a photo editor hired to remove a photobomber from a group picture. You do a great job: the person is gone, and the background looks natural. But how do you know if your edit is actually good?

This is the problem the paper PROVE tackles. The authors argue that the current tools we use to grade these edits are like using a ruler to measure the taste of soup—they are looking at the wrong things.

Here is a breakdown of their solution using simple analogies.

1. The Problem: The "Copy-Paste" and "Blurry" Traps

The paper explains that existing grading tools (metrics) make two major mistakes:

The "Copy-Paste" Trap (Full-Reference Metrics): Imagine a teacher grading a student's essay by comparing it word-for-word to a sample answer. If the student copies the sample perfectly, they get an A, even if the essay is boring.
- In image editing, old tools reward models that just "copy-paste" the background from the original photo rather than actually "erasing" the object and inventing a new, realistic background. They prefer safe, boring answers over creative, realistic ones.
The "Blurry is Clean" Trap (No-Reference Metrics): Imagine a judge who thinks a blurry photo is better than a sharp one because the blur hides mistakes.
- Current tools often give high scores to blurry results because blur makes things look "smooth" and uniform. They fail to notice that the image is actually low-quality or that the object removal created weird artifacts.

The Video Problem: When editing a video, existing tools look at the whole screen. If the background is stable but the spot where the object was removed is flickering wildly, the tool might still give a high score because the rest of the video is fine. It's like a teacher grading a 10-page essay but only reading the first page and ignoring the rest.

2. The Solution: The "Spotlight" Approach (RC Metrics)

The authors propose a new way to grade edits called RC (Removal Coherence). Instead of looking at the whole picture or comparing it to a perfect reference, they use a "sliding spotlight" to zoom in on the specific area where the object was removed.

They have two main tools:

RC-S (Spatial Coherence): Think of this as a texture detective. It zooms in on the hole where the object was and asks: "Does the new background here look like the surrounding neighborhood?"
- It uses a sliding window (like a magnifying glass) to check if the textures, lighting, and patterns match the area around the edit. If the new background looks like a different texture or is too blurry, the score drops.
RC-T (Temporal Coherence): This is the video stability inspector. It looks at the same spot across two consecutive frames.
- It asks: "Does this spot jump around or flicker as the video plays?" If the background in the removed area is shaking or changing weirdly from frame to frame, this metric catches it, even if the rest of the video is smooth.

The Secret Sauce: They use a special "brain" (a feature extractor called DINOv2) that is very sensitive to tiny details and frequency changes, much like a human eye is sensitive to subtle visual glitches.

3. The New Playground: PROVE-Bench

To prove their new grading system works, they built a new testing ground called PROVE-Bench. They realized that old test sets were either:

Too fake: Computer-generated videos that don't look like real life.
Too hard to grade: Real videos where we don't know what the "perfect" background should look like.

Their new benchmark has two parts:

PROVE-M (The Controlled Lab): They filmed real scenes, removed an object, and filmed the scene again without the object. This gives them a "perfect" ground truth to compare against, but they added simulated camera shakes to make it feel like a real handheld video.
PROVE-H (The Stress Test): A collection of 100 difficult, real-world videos (crowds, fast motion, reflections) where they don't have a perfect reference. This tests if the models can handle chaos.

4. The Results

When they tested their new "Spotlight" grading system against the old tools:

Old tools often gave high scores to blurry, copy-paste results or missed flickering in videos.
PROVE (RC-S and RC-T) aligned much better with what actual humans thought looked good. If a human said, "That looks fake," the new metric agreed. If a human said, "That looks great," the new metric agreed.

Summary

The paper argues that to judge object removal, we need to stop looking at the whole picture or comparing it to a perfect reference. Instead, we need to zoom in on the edited area to check if it blends in with its neighbors (Spatial) and stays steady over time (Temporal). They built a new set of tools and a new testing ground to prove that this "zoom-in" approach is the only way to get a grade that matches human perception.

Technical Summary: PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media

1. Problem Statement

Object removal in images and videos is an inherently ill-posed, one-to-many problem where multiple perceptually plausible restorations can exist for a single erased region. Consequently, defining a unique ground truth (GT) is often impossible. This characteristic undermines traditional evaluation protocols:

Full-Reference (FR) Metrics: Metrics like PSNR, SSIM, and LPIPS assume a strict point-to-point correspondence to a single reference. They frequently reward "copy-paste" behaviors (conservative outputs that preserve the background) over genuine, perceptually realistic erasure. Furthermore, they suffer from "regression to the mean" biases, where reducing diffusion inference steps (degrading visual quality) paradoxically improves scores due to reduced pixel variance.
No-Reference (NR) Metrics: Methods like ReMOVE and CFD, while avoiding the need for GT, exhibit systematic biases. They often assign inflated scores to blurry outputs ("Blur is Clean" bias) and incorrectly penalize structurally sound restorations in complex occlusion scenarios.
Temporal Metrics: Existing video metrics (e.g., Temporal Consistency, Temporal Flickering) operate on global frame-level features. Because removed regions typically occupy only a small fraction of the frame, these global aggregations dilute or entirely mask localized temporal artifacts within the edited regions.

Additionally, existing benchmarks lack a combination of real-world authenticity and paired reference videos. Synthetic datasets (e.g., ROSE-Bench) offer precise masks but lack real-world complexity, while real-world datasets (e.g., DAVIS) lack paired target-free videos for controlled quantitative comparison.

2. Methodology

2.1 The RC Metrics (Removal Coherence)

The authors propose RC, a unified local distribution matching framework in deep semantic feature space, instantiated as two complementary metrics:

RC-S (Spatial Coherence): Measures the local consistency between the restored content and its surrounding background within a single frame.
- Process: The target region is cropped with expanded context. Deep features are extracted using DINOv2. A sliding window is applied over the feature map.
- Mechanism: Within each window, the feature distribution of the masked (restored) region is compared against the unmasked (background) region using the squared Maximum Mean Discrepancy (MMD).
- Rationale: This local comparison detects spatial incoherence that global aggregation misses. MMD is chosen over cosine similarity for its sensitivity to distributional shifts (e.g., blur).
RC-T (Temporal Consistency): Measures the stability of restored regions across adjacent frames.
- Process: For adjacent frames, a shared evaluation region is defined using the union of their masks. Both frames are cropped synchronously.
- Mechanism: The metric focuses on the intersection of the restored regions across the two frames. It compares the feature distributions of these intersected regions using MMD.
- Rationale: By restricting evaluation to the shared restored area, RC-T avoids the dilution effect caused by unchanged background regions, enabling sensitive detection of localized temporal flickering.

2.2 PROVE-Bench

To support rigorous evaluation, the authors introduce PROVE-Bench, a two-tier benchmark suite:

PROVE-M (Motion-Augmented Paired): An 80-video dataset comprising real-world paired recordings (input video, mask, and target-free ground truth). To bridge the gap between static lab captures and real user videos, these pairs undergo motion augmentation (Ken Burns-style geometric transformations like panning, zooming, and shaking) to simulate handheld camera instability.
PROVE-H (Hard Unconstrained): A 100-video dataset without ground truth, designed to stress-test model generalization. It targets extreme scenarios including crowds, fast motion, complex reflections, and highly textured backgrounds. Masks are generated automatically (SAM3) to reflect practical imperfections.

3. Key Contributions

Systematic Diagnosis: The paper provides a rigorous empirical analysis of failure modes in existing metrics, identifying specific biases such as "copy-paste" preference in FR metrics, "blur-favoring" in NR metrics, and regional insensitivity in global temporal metrics.
RC Metrics: The proposal of RC-S and RC-T, which utilize a sliding-window distribution matching framework (DINOv2 + MMD) to quantify local spatial coherence and temporal consistency. These metrics achieve substantially stronger alignment with human judgments than existing protocols.
PROVE-Bench: The construction of a unique benchmark suite that combines a real-world paired dataset (PROVE-M) with motion augmentation and a challenging ground-truth-free subset (PROVE-H), addressing the realism-evaluability dilemma in current datasets.

4. Experimental Results

Extensive experiments were conducted across diverse image (RORD-Val, OBER-Wild) and video (DAVIS, ROSE-Bench, PROVE-Bench) benchmarks.

Human Correlation: RC-S achieved the highest correlation with human rankings (Kendall's $\tau$ and Spearman's $\rho$ ) across six benchmarks, outperforming FR metrics (PSNR, SSIM, LPIPS), NR metrics (ReMOVE, CFD), and global temporal metrics. Notably, RC-S ranked first on five of the six benchmarks.
Sensitivity Analysis:
- Blur Sensitivity: Unlike ReMOVE and CFD, which often increased scores as blur intensity increased, RC-S correctly decreased scores monotonically, demonstrating resistance to the "Blur is Clean" bias.
- Temporal Sensitivity: In sensitivity tests involving random frame drops and replacements, RC-T showed a monotonic degradation in scores as corruption increased. In contrast, TC and TF remained largely insensitive or even improved, confirming their inability to detect localized temporal artifacts.
Ablation Studies: The authors validated that the combination of DINOv2 (for perceptually sensitive features), sliding windows (for locality), and MMD (for distributional discrepancy) is critical. Removing any component (e.g., using cosine similarity or global aggregation) significantly reduced correlation with human judgment.
Benchmark Difficulty: Experiments on PROVE-M revealed that introducing simulated camera motion causes consistent and significant performance degradation across all state-of-the-art models, highlighting the necessity of motion-aware evaluation.

5. Significance and Claims

The paper claims that PROVE provides a more reliable foundation for assessing the efficacy and failure modes of object removal methods. By shifting the evaluation paradigm from "fidelity to a single reference" to "perceptual coherence with context and time," the framework addresses the fundamental mismatch between current metrics and the one-to-many nature of the task.

The authors state that their work serves as a practical foundation for standardized benchmarking, offering a unified framework where RC metrics and PROVE-Bench complement each other to support both rigorous quantitative comparison (via PROVE-M) and stress-testing under unconstrained real-world conditions (via PROVE-H). The code and datasets are made publicly available to facilitate community adoption.