🤖 AI

MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

This paper introduces MatReplace, a reference-free benchmark designed to evaluate material replacement in interior scenes across four verifiable dimensions and three conditioning tracks, revealing that while top closed-source models excel at rendering named materials, grounding materials from visual references remains a significant challenge.

Original authors: Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mingzhe Du, Thong Thanh Nguyen, Nguyen Tran Cong Duy, See-Kiong Ng, Luu Anh Tuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine walking into a furniture store and asking to see a sofa upholstered in a specific navy blue velvet, or a kitchen island topped with a particular kind of green marble. In the real world, this requires sourcing materials, shipping them, and physically rebuilding the object. In the digital world, designers have long hoped to simply type a request or show a picture of a material and have a computer instantly repaint a room with that new texture, while keeping the shape of the furniture, the lighting, and the rest of the room exactly the same. This is the promise of generative image editing: tools that can alter a photograph as easily as a painter changes a brushstroke. However, while these tools have become common, no one had a reliable way to test if they were actually doing the job correctly. The challenge is that a perfect edit is not just about making the new material look right; it is about ensuring the old parts of the image do not change, the shadows fall naturally on the new surface, and the object does not warp or melt into something else. Without a clear standard, it is impossible to know if a computer is truly understanding the request or just guessing.

A team of researchers from the National University of Singapore, Nanyang Technological University, and VinUniversity has created a new way to measure this ability, called MatReplace. Instead of asking a computer to compare its new image to a single "correct" answer stored in a database, they designed a test that checks four specific things about every edit. First, does the painted area actually look like the requested material? Second, did the rest of the room stay exactly the same? Third, did the shape of the object remain intact, or did the computer accidentally reshape the furniture? And fourth, does the new surface look like it belongs in the room, with the correct lighting and shadows? By checking these four points without needing a pre-existing perfect photo to compare against, the researchers built a benchmark that can fairly judge different computer programs, even if those programs were built by different companies.

The researchers tested this system on over 1,400 different interior design tasks, ranging from changing a headboard to navy suede to swapping a floor for herringbone oak. They organized the tests into three different ways of giving instructions to the computer. In the first setup, the computer only received a text description. In the second, it received the text plus a digital outline showing exactly which part of the image to change. In the third, it received the outline plus a reference photograph of the material to be used. The results revealed a clear split between the most advanced, closed-source systems and the open-source ones. The strongest commercial tools, which are not available to the public, performed exceptionally well when given just a text description. They could change the material perfectly while keeping the rest of the scene untouched, proving that for the best systems, simply naming the material is enough.

However, the study found that adding more information did not always help. When the researchers gave the computer a digital outline of the area to change, it only improved the results for systems that were already struggling to keep the rest of the room stable. For the strongest systems, the outline made no difference, and for some weaker systems, it actually made the results worse. The most surprising finding came from the third setup, where the computer was shown a picture of the material instead of a text description. In this case, every single system performed worse. Instead of using the picture to understand the texture, many of the computers simply copied the picture onto the screen without adjusting the lighting or shape, or they failed to apply the material at all. One system even painted the reference picture itself instead of the room. This suggests that while computers have become very good at understanding words like "navy suede," they still struggle to understand what a picture of a material means when it needs to be applied to a new object.

The researchers also compared their new scoring method against older ways of judging these edits, which often relied on comparing the result to a stored reference image or using text-matching tools. Their new method matched the judgments of human experts much more closely than the old methods did. Human raters agreed with the new system's ranking of the tools, confirming that the four-part check was a reliable way to measure quality. The study concludes that the problem of changing a material based on a text description is largely solved for the most advanced tools, but the problem of using a picture to guide that change remains unsolved. Until computers can better translate a visual reference into a realistic, three-dimensional change, designers will likely get better results by naming the material in words rather than showing a picture. This work provides a clear, fair way to track progress in the field, ensuring that future tools are judged on whether they truly preserve the scene and lighting, rather than just on how closely they mimic a specific style.

Technical Summary: MatReplace

Problem Statement
Generative image editors are increasingly integrated into interior-design workflows to perform material replacement: repainting a marked surface with a new material while strictly preserving the object's geometry, the surrounding scene, and the illumination. Despite the commercial routine nature of this task, no public benchmark isolates it. Existing instruction-editing benchmarks fold material replacement into broad categories and rely on Ground Truth (GT) referenced metrics (comparing outputs to a single stored reference image). The authors argue this approach is fundamentally flawed for material replacement because:

One-to-Many Nature: Many valid outputs exist for a single prompt (e.g., many correct "navy suede" headboards). Penalizing distance to a single stored reference punishes valid diversity.
Generator Bias: If all reference images come from a single generator, metrics reward imitating that specific generator's style, biasing leaderboards toward the data source rather than general capability.
Conditioning Confounds: Mixing models that receive different guidance (text only, text+mask, text+mask+reference) on a single leaderboard conflates model quality with input signal quality.

Methodology
The paper introduces MatReplace, a reference-free benchmark and evaluation protocol designed to isolate material replacement across three distinct conditioning tracks.

Benchmark Construction:
- Dataset: 1,421 task specifications derived from 2,017 candidate sessions generated by a disclosed model (GPT-Image-2).
- Task Tuple: $(x, m, r, T, c)$ , where $x$ is the scene, $m$ is a binary mask, $r$ is a reference image of the target material, $T$ is a text instruction, and $c$ is the material category.
- Curation: Tasks undergo algorithmic checks (resolution, mask binarity) and a semantic audit by a vision-language agent to ensure mask alignment, edit correctness, and scene preservation.
- Tracks: The benchmark defines three tracks to vary conditioning signals while keeping model families aligned:
  - Track A: Instruction only ( $x + T$ ).
  - Track B: Instruction + Region Mask ( $x + m + T$ ).
  - Track C: Mask + Reference Image ( $x + m + r$ ), where the material is specified only by pixels, not text.
Reference-Free Evaluation Protocol:
Instead of comparing against a stored output, the protocol scores edits on four independently verifiable dimensions based solely on the task inputs:
1. Material Correctness ( $S_{mat}$ ): Combines SigLIP2 classification of the masked region (at the material family level) with DINOv2 cosine similarity to the reference image $r$ .
2. Outside Preservation ( $S_{pres}$ ): Measures LPIPS distance between the input and output restricted to the region outside the mask.
3. Inside Structure ( $S_{struct}$ ): Uses monocular depth estimation to ensure the geometry inside the mask remains unchanged, aligned via scale-and-shift on the outside region.
4. Lighting Harmony ( $S_{harm}$ ): Correlates low-pass luminance fields inside the mask to ensure the new surface respects the scene's illumination (albedo changes should not restructure lighting).
The primary aggregate score is a "gate" metric ( $Overall_G$ ) that penalizes failures in preservation, structure, or harmony more heavily than raw averaging, validated against human rankings.

Key Contributions

MatReplace Benchmark: The first benchmark to isolate material replacement with a frozen split, taxonomy labels, and released inputs across three conditioning tracks.
Reference-Free Protocol: A scoring system that decomposes the task into four verifiable properties, avoiding the biases of GT-referenced metrics.
Conditioning-Aligned Leaderboard: A framework that evaluates the same model family across different input modalities (text, mask, reference) to measure the specific value or cost of each signal.

Results

Track A (Instruction Only): Strong closed-source models (e.g., Nano-Banana-2-Lite, GPT-Image-2) achieve exemplar-quality results, clearing the "exemplar anchor" (a soft ceiling derived from the data generator's own outputs). This suggests rendering a named material is largely solved for top-tier closed editors without spatial guidance.
Track B (Instruction + Mask): Masks provide a benefit only to models that are weak at scene preservation and capable of "consuming" the mask (e.g., FLUX-Kontext gains +0.137). For strong closed models, masks add no value or slight friction.
Track C (Mask + Reference): Introducing a reference image degrades performance across all model families (drops ranging from -0.031 to -0.508).
- Failure Modes: Models fail to ground the pixel-based specification, leading to three distinct failure modes: (1) Appearance-grounding loss (less precise rendering), (2) Material-identification collapse (inability to translate pixels to semantic targets), and (3) Role-binding failure (repainting the reference image itself instead of the scene).
- Conclusion: While naming a material is solved, grounding a material shown as pixels remains an open problem.

Significance and Claims
The paper claims that MatReplace provides a more rigorous and fair evaluation of generative editing by:

Decoupling Model Quality from Input Signals: By aligning families across tracks, it reveals that masks are not universally beneficial and that reference images currently degrade performance.
Validating Reference-Free Scoring: Expert human ratings correlate strongly with the proposed aggregates ( $\tau = 0.68$ ), outperforming established GT-referenced and CLIP-based baselines.
Identifying the Current Frontier: The work establishes that while closed editors can render named materials effectively, the ability to interpret and render a material specified solely by a reference image (without text description) is a significant bottleneck in current generative editing capabilities.

The authors maintain a modest stance, noting the benchmark is currently fully synthetic and single-source, and that the transfer to real-world photographs remains unverified. The primary contribution is the isolation of the task and the demonstration that current "reference-guided" approaches often fail to outperform "instruction-only" approaches in this specific domain.

Technical Summary: MatReplace

More like this