💻 computer science

TexEditor: Structure-Preserving Text-Driven Texture Editing

TexEditor is a structure-preserving text-driven texture editing model that leverages a high-quality synthetic dataset (TexBlender) and a reinforcement learning-based method (StructureNFT) to overcome structural consistency issues in existing models, validated by a new real-world benchmark (TexBench) where it outperforms strong baselines.

Original authors: Bo Zhao, Yihang Liu, Chenfeng Zhang, Huan Yang, Kun Gai, Wei Ji

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bo Zhao, Yihang Liu, Chenfeng Zhang, Huan Yang, Kun Gai, Wei Ji

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a favorite wooden dining table. You want to change its finish from rough oak to smooth, polished walnut. You ask a digital artist (an AI) to do this.

In the past, if you asked a smart AI to "make this table look like polished walnut," it might get the color right, but it would accidentally redraw the table's legs, change its shape, or even make the table disappear and reappear as a slightly different table. It was like hiring a painter who, when asked to repaint a wall, decided to rebuild the whole house instead.

This paper introduces TexEditor, a new AI tool designed to solve exactly that problem. Think of TexEditor as a master interior designer who is obsessed with preserving the blueprint of your room.

Here is how they built it, explained simply:

1. The Problem: The "Over-enthusiastic" Painter

The authors noticed that even the best AI tools today struggle with texture editing. When you ask them to change the material (like turning a plastic chair into a leather one), they often lose the structure (the chair's shape, the curve of the back, the position of the legs). They tend to "hallucinate" a new object rather than just repainting the old one.

2. The Solution: A Two-Step Training Camp

To fix this, the team built TexEditor using a clever two-step training process, like teaching a student before letting them work in the real world.

Step A: The "Perfect Classroom" (TexBlender)

First, they couldn't just use real photos because it's hard to find pairs of photos where only the texture changed and the shape stayed exactly the same.

The Analogy: Imagine trying to learn how to paint a car without ever seeing a car that hasn't been crashed.
The Fix: They built a virtual classroom using 3D software (Blender). They created thousands of perfect "before and after" scenes. In this virtual world, they could swap a table's texture from wood to metal while the computer guaranteed the table's legs and shape remained 100% identical.
The Result: The AI learned the golden rule: "Change the skin, but never touch the skeleton."

Step B: The "Real World Internship" (StructureNFT)

Once the AI learned the rules in the virtual classroom, it needed to practice on messy, real-world photos (like a photo of a boat in a harbor).

The Analogy: A student who aced math in a quiet classroom might panic when asked to solve a problem in a noisy cafeteria.
The Fix: They used a technique called Reinforcement Learning (like training a dog with treats).
- If the AI changed the texture but kept the shape, it got a "treat" (a high score).
- If the AI accidentally warped the shape, it got a "time-out" (a penalty).
- They added a special "structure detector" (a digital ruler) that constantly checked: "Did the edges move? Did the shape bend?" If yes, the AI had to try again.

3. The New Ruler: TexBench and TexEval

The authors realized that the old ways of testing AI were broken.

The Old Way: They used other AIs to grade the results. But those "grading AIs" were often too polite or too confused to notice that a chair's leg had been subtly twisted.
The New Way: They built TexBench, a new test bank of real-world photos, and TexEval, a new grading system.
The Analogy: Instead of asking a poet to grade a math test, they built a system that checks both the poetry (did you follow the instructions?) and the math (is the geometry correct?). They found that a mix of 60% poetry and 40% math gave the most accurate results.

4. The Result: The "Magic Paintbrush"

When they tested TexEditor against the current champions (like "Nano Banana Pro"), TexEditor won.

Nano Banana Pro might turn a boat into a "rustic wooden boat," but it might accidentally change the boat's hull shape or make the sails disappear.
TexEditor turns the boat into a rustic wooden boat, but the hull, the sails, and the cracks in the wood stay exactly where they were. It respects the original object's identity.

Summary

TexEditor is like a digital artist who has been trained to understand that texture is just the paint, and structure is the canvas. They learned this by practicing in a perfect virtual studio and then being strictly graded on real-world photos. The result is a tool that can change the look of your world without accidentally breaking the world itself.

1. Problem Statement

Text-guided texture editing aims to modify the appearance of objects in an image (e.g., changing material, roughness, or color) based on natural language instructions while strictly preserving the underlying geometric structure, spatial layout, and semantic identity of the object.

Key Challenges Identified:

Structural Collapse: Even State-of-the-Art (SOTA) models (e.g., Nano Banana Pro, Qwen-Image-Edit) frequently fail to maintain structural consistency. Instead of merely changing the texture, they often "regenerate" the object, altering its shape, boundaries, or internal details.
Data Scarcity: There is a lack of high-quality, paired training data that explicitly decouples texture changes from geometric variations, especially in complex, real-world scenes.
Evaluation Limitations: Existing benchmarks rely heavily on synthetic, single-object scenes (Blender) which lack realism. Furthermore, current evaluation metrics often rely on Large Multimodal Models (LMMs) that fail to detect subtle, unintended structural distortions.

2. Methodology: TexEditor

The authors propose TexEditor, a dedicated texture editing model based on the Qwen-Image-Edit-2509 backbone. The approach integrates data construction and a two-stage training strategy to enforce structure preservation without modifying the model architecture.

A. Data Construction: TexBlender

To address the lack of structural priors, the authors constructed TexBlender, a high-quality, scene-level simulated dataset.

Source: Built using Blender and the 3D-Front dataset (diverse indoor layouts).
Design: Unlike previous datasets focusing on isolated objects, TexBlender uses grouped 3D objects (e.g., a table with legs) to simulate complex scene-level edits.
Process:
1. Scene Prep: Filters 3D-Front scenes for geometric plausibility.
2. Editing: Applies texture changes (via Principled BSDF shaders) to grouped objects while keeping the 3D geometry and camera viewpoint identical.
3. Instruction Generation: Uses Qwen3-VL and SAM3 (Segment Anything Model) to generate precise, grounded natural language instructions that align with the visual changes, reducing hallucinations.
Outcome: Provides "clean" supervision where the only difference between input and output is the texture, forcing the model to learn structure-invariant editing.

B. Training Strategy

The training follows a two-stage pipeline:

Supervised Fine-Tuning (SFT): The model is cold-started on TexBlender to learn basic texture editing skills and the concept of preserving geometry.
Reinforcement Learning (RL) with StructureNFT: To generalize to real-world scenes, the model undergoes RL on real-world data (derived from COCO).
- Reward Function: A hybrid reward system balances instruction adherence and structural preservation.
  - $Score_{ins}$ : Instruction adherence score computed by Gemini 3 Flash.
  - $Score_{struct}$ : A structure-preserving loss derived from SAUGE (a wireframe extractor) and SSIM (Structural Similarity Index).
- StructureNFT: A novel RL approach that introduces a regularization term to the loss function. It penalizes deviations in low-level structural cues (edges/wireframes) extracted from the image, ensuring the model does not "regenerate" the object.
- Normalization: Empirical normalization is applied to the structural score to handle the sensitivity of SSIM to minor pixel perturbations.

3. Key Contributions

A. TexEditor Model

A structure-aware editing model that consistently outperforms baselines in preserving object geometry while following complex texture instructions.

B. TexBlender Dataset

A new, large-scale synthetic dataset featuring complex indoor scenes and grouped object editing, providing strong structural priors for cold-start training.

C. TexBench Benchmark

A new, real-world benchmark constructed on COCO images to evaluate texture editing in realistic scenarios.

Features: Includes diverse, human-verified instructions for both texture replacement and attribute modification.
Scale: Contains 825 high-quality samples (453 replacement, 372 attribute).

D. TexEval Metric

A composite evaluation metric designed to overcome the limitations of LMM-only scoring.

Formula: $TexEval = \alpha \cdot Score_{Ins} + (1-\alpha) \cdot Score_{struct}$
Significance: It jointly evaluates instruction following and structural consistency (using wireframe/SSIM cues), showing higher alignment with human preference than existing metrics.

4. Experimental Results

Quantitative Performance

TexBench: TexEditor achieves the highest scores in both Instruction Adherence and Structure Preservation compared to strong baselines like Nano Banana Pro, Qwen-2509, and Alchemist.
- Example: On TexBench texture tasks, TexEditor scored 0.929 on Structure (vs. 0.801 for Nano Banana Pro) and 0.886 on TexEval (vs. 0.824).
Blender Benchmark: TexEditor also dominates on synthetic benchmarks, achieving near-perfect scores (0.972 TexEval) on texture replacement tasks.

Qualitative Analysis

Visual comparisons show that while baselines often distort object shapes (e.g., changing the shape of a boat or a clock when asked to change material), TexEditor successfully modifies only the surface texture while retaining the original contours and internal details.

Ablation Studies

Data: Training on TexBlender (multi-granularity, scene-level) significantly outperforms training on single-object datasets (like Alchemist's).
RL: The combination of SFT and RL with the StructureNFT loss is crucial. Removing the structural loss leads to a drop in structure preservation; removing instruction loss leads to poor adherence.
Generalization: When tested on the general-purpose ImgEdit benchmark, TexEditor showed improved performance on texture-related sub-tasks (Background, Adjust, Compose), indicating its structure-preserving capabilities generalize beyond specific texture tasks.

5. Significance and Impact

Bridging the Gap: The paper addresses a critical failure mode in current generative editing models: the tendency to hallucinate geometry when modifying textures.
Methodological Innovation: The introduction of StructureNFT demonstrates that integrating low-level structural cues (wireframes/SSIM) into RL reward signals is an effective, architecture-agnostic way to enforce geometric consistency.
Community Resources: By releasing TexBlender, TexBench, and TexEval, the authors provide the community with the necessary tools to benchmark and improve structure-preserving editing, moving the field beyond simple single-object synthetic tests toward complex, real-world applications.

In conclusion, TexEditor sets a new standard for text-driven texture editing by proving that with the right data priors and structural constraints, models can edit surface appearance without compromising the underlying 3D reality of the scene.