💻 computer science

Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Models

This paper proposes a novel Diffusion Transformer framework for high-fidelity texture tiling that achieves precise control over pattern frequency, orientation, and scale while preserving structural integrity and scene consistency through a Coordinate-Transformed Rotary Embedding mechanism and a Disjoint Attention Mask.

Original authors: Junrong Huang, Zhiyuan Zhang, Rui Tang, Hongbo Fu, Jnig Liao

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junrong Huang, Zhiyuan Zhang, Rui Tang, Hongbo Fu, Jnig Liao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, high-resolution photo of a room, and you want to change the wallpaper on one specific wall. You have a picture of the new wallpaper pattern you love, but you want it to look like it was actually painted on that wall, not just stuck on top like a sticker.

The problem with current AI tools is that they often struggle with two things:

The "Sticker" Effect: They might paste the pattern on, but it looks flat and doesn't bend around the corners or follow the curve of the wall.
The "Blurry Copy" Effect: To make the pattern fit, they often squish or stretch the original image, which makes the fine details (like the tiny weave of fabric or the grain of wood) turn into a blurry mess.

This paper introduces a new AI method called ControlTile that solves these problems. Here is how it works, explained with simple analogies:

1. The Magic of "Moving the Map, Not the Paint"

Most AI tools try to physically warp the image of the wallpaper (like stretching a rubber sheet) to make it fit the wall. This is like trying to stretch a printed map to fit a globe; the lines get distorted, and the details get blurry.

The authors' secret sauce is called Transformed RoPE.

The Analogy: Imagine you are playing a game of "Hide and Seek" in a giant warehouse. The AI is the seeker, and the wallpaper pattern is the person hiding.
How it works: Instead of moving the person (the pixels of the wallpaper) to a new spot, the AI simply changes the map it uses to find them. It tells the AI, "When you look for the pattern, pretend the coordinates are rotated and scaled."
The Result: The AI grabs the original, perfect, high-definition wallpaper pattern and places it exactly where you want it, without ever stretching or blurring the original image. It's like teleporting the perfect pattern to the right spot rather than dragging it there.

2. The "Soundproof Room" (Disjoint Attention Mask)

When an AI tries to learn from a reference image, it sometimes gets confused. It might accidentally mix the texture of the new wallpaper with the color of the old furniture in the background, creating a muddy, messy result.

The authors use a Disjoint Attention Mask.

The Analogy: Think of the AI as a chef trying to copy a recipe. The "Reference Texture" is the master chef, and the "Background" is a noisy kitchen.
How it works: The authors put a soundproof glass wall between the master chef and the noisy kitchen. The AI can look at the master chef to learn the exact recipe (the texture), but the noise from the kitchen (the background furniture) cannot leak through to mess up the recipe.
The Result: The new wallpaper looks crisp and true to the original reference, while still blending perfectly with the lighting and shadows of the room.

3. The "3D GPS" (Latent Fusion)

Even if the pattern is perfect, it needs to look like it belongs on a 3D object, not a flat piece of paper.

The system uses Depth and Lighting Maps (like a 3D GPS and a sun tracker).

The Analogy: Imagine you are wrapping a gift. You don't just tape the paper on; you fold it around the box.
How it works: The AI looks at the 3D shape of the wall (is it curved? is it flat?) and the lighting (where is the sun shining?). It then "folds" the texture around the object so the shadows and highlights match the real world.
The Result: The wallpaper looks like it physically exists on the wall, with realistic shadows and curves.

What Did They Test?

The team built a massive library of 15,000 practice scenes (a mix of computer-generated and realistic photos) to teach the AI. They tested it against the current best tools (like FLUX and MatSwap).

The Score: In tests, their method was chosen by users as the best 42% of the time for how realistic it looked, and 91% of the time for how well it followed the user's instructions (like rotating the pattern 45 degrees).
The Comparison: Other tools either made the pattern blurry or failed to make it follow the 3D shape of the wall. This new method kept the details sharp and the shape perfect.

In Summary

This paper presents a new way to change textures in images that is like having a perfect, high-definition stamp that can be rotated, scaled, and bent to fit any 3D shape without ever losing a single pixel of detail. It does this by tricking the AI's internal "GPS" rather than physically squishing the image, and by using a "soundproof wall" to keep the design pure.

Technical Summary: Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Models

1. Problem Statement

The integration of user-specified textures into scene images is a fundamental task in computer graphics and image editing. While existing material transfer and reference-guided inpainting methods (e.g., FLUX.1-dev-Fill, ZeST, MatSwap) can edit surface appearances, they struggle with the specific requirements of texture tiling. Current approaches face three primary limitations:

Lack of Precise Control: They often fail to strictly adhere to user-defined spatial parameters such as frequency, orientation, and scale, frequently "hallucinating" patterns based on learned distributions rather than the specific reference.
Destructive Resampling: Methods that rely on explicit pixel warping (e.g., MatSwap) or cross-attention mechanisms suffer from destructive downsampling, leading to the loss of high-frequency details and structural fidelity.
Semantic Leakage: Existing models often fail to preserve the coherent lighting and geometry of the original scene or allow reference features to bleed into unedited background regions.

The core challenge is to achieve controllable, high-fidelity texture tiling that decouples spatial manipulation from content generation, preserving both the reference texture's identity and the scene's physical consistency.

2. Methodology

The authors propose a novel framework built upon the Diffusion Transformer (DiT) architecture, specifically leveraging the FLUX.1-Kontext-dev model. The approach reformulates texture mapping as a specialized visual-conditioned inpainting task using two key technical innovations:

A. Coordinate-Transformed Rotary Embeddings (Transformed RoPE)

Instead of physically warping the input texture pixels (which causes resolution loss), the method injects spatial transformations directly into the attention mechanism.

Mechanism: The framework utilizes 2D Rotary Positional Embeddings (RoPE), which encode spatial awareness via relative distances between tokens.
Implementation: User-defined affine transformations ( $\mathcal{T}$ ), including rotation, scaling, and translation, are applied directly to the positional embeddings of the reference keys and values during Self-Attention.
Effect: This allows the model to treat the reference texture as a single unit and perform "Implicit Generative Tiling." The attention mechanism retrieves high-fidelity features from the reference unit based on the transformed coordinates without explicit pixel resampling, preserving the full spectral fidelity of the latent features.

B. Disjoint Attention Masking

To prevent semantic leakage and ensure structural integrity, the authors introduce a specialized attention mask that partitions the token sequence into three disjoint subsets:

Reference Tokens ( $\mathcal{S}_{ref}$ ): Restricted to attending only to themselves. This shields the reference texture from background interference and prevents color bleeding.
Background Tokens ( $\mathcal{S}_{bg}$ ): Allowed to attend to the scene but strictly blocked from attending to the reference tokens to prevent the reference texture from altering the unedited context.
Inpainting Target Tokens ( $\mathcal{S}_{mask}$ ): Granted full visibility to attend to both the reference (for texture features) and the background (for global illumination and geometry).

C. System Pipeline

The system accepts a source image ( $I_{src}$ ), a target mask ( $M$ ), a reference texture ( $I_{ref}$ ), and transformation parameters ( $\mathcal{T}$ ).

Geometric Priors: Depth maps ( $D$ ) and irradiance maps ( $L$ ) are extracted using upstream estimators and encoded into the latent space to modulate the tiled pattern according to surface normals and shading.
Latent Fusion: The target latent is constructed by concatenating the noisy latent, the unedited background, the mask, and the geometric priors. The reference texture is zero-padded to match channel dimensions and concatenated along the sequence dimension.
Training: The model is trained in two stages: first on a synthetic Blender Subset (14,000 samples) to learn fundamental tiling mechanics, and then fine-tuned on an Adaptation Subset (1,300 samples) using a physically-based rendering engine to bridge the reality gap.

3. Key Contributions

High-Fidelity Tiling via Transformed RoPE: The paper introduces a mechanism to inject 2D affine transformations directly into Rotary Positional Embeddings. This enables precise spatial control and preserves high-frequency details by avoiding destructive pixel warping.
Disjoint Attention Control: A dedicated masking strategy is proposed to eliminate feature leakage between the reference condition and the background, ensuring artifact-free non-edited regions and preserving the structural integrity of the exemplar.
Benchmark Dataset: The authors release a large-scale dataset of 15,000 diverse, high-quality scene-texture pairs with ground-truth transformation matrices and dense semantic masks to establish a standardized evaluation protocol for texture editing controllability.

4. Experimental Results

The method was evaluated against state-of-the-art baselines (FLUX.1-Fill-dev, FLUX.2-dev, ZeST, MaterialFusion, MatSwap) on both synthetic and real-world datasets.

Texture Fidelity: On synthetic data, the proposed method achieved state-of-the-art performance across all metrics (LPIPS, CLIP-Score, and Gram Matrix Loss), outperforming baselines by significant margins. On real-world data, user studies showed a 42.0% Top-1 preference rate, significantly higher than the next best method (MatSwap at 9.8%).
Spatial Controllability: In ablation studies and comparisons where baselines were forced to use pre-warped inputs, the proposed method demonstrated superior geometric precision. It successfully handled complex rotations, scaling, and even non-affine injections, whereas baselines suffered from aliasing, structural degradation, or "copy-paste" biases that ignored 3D surface topology.
Ablation Analysis: Removing the Disjoint Attention Mask led to semantic contamination and loss of structural clarity. Replacing the implicit coordinate injection with explicit pixel-space warping (Global or Local Explicit) resulted in geometric misalignment and irreversible loss of high-frequency details due to downsampling.

5. Significance and Claims

The paper claims to bridge the gap between precise geometric editing and generative photorealism. By decoupling spatial manipulation from visual content generation, the framework offers a solution that is:

Lossless: It preserves the high-frequency details of the reference texture by avoiding explicit pixel resampling.
Physically Consistent: It integrates the synthesized texture with the scene's original lighting and geometry through latent fusion with depth and irradiance priors.
Controllable: It allows for user-defined affine transformations that are strictly adhered to, rather than relying on the model's probabilistic interpretation of text or image prompts.

The authors position this work as a new state-of-the-art for material transfer, offering an optimal balance of structural control, texture fidelity, and visual realism. They acknowledge limitations regarding resolution bottlenecks in the VAE latent space, the lack of explicit material maps (e.g., roughness), and the inability of a single affine transform to handle sharp transitions on complex geometries, noting these as avenues for future work.