HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images

The paper proposes HiFi-Inpaint, a novel framework that utilizes Shared Enhancement Attention and a Detail-Aware Loss to overcome data and supervision limitations, achieving state-of-the-art, high-fidelity generation of detail-preserving human-product images.

The Gist

🎨 The Big Idea: The "Perfect Fit" Tailor

Imagine you are a fashion designer. You have a photo of a model wearing a plain white shirt, but you want to replace that shirt with a specific, expensive designer jacket that has a very intricate logo, a specific texture, and tiny text on the tag.

If you just ask a regular AI to "put the jacket on the model," it might get the shape right, but the logo might look like a blurry smudge, the text might be gibberish, and the fabric might look like plastic. It's like a tailor who knows how to sew a jacket but doesn't know how to stitch the brand name correctly.

HiFi-Inpaint is a new, super-smart AI tailor. Its superpower is High-Fidelity Reference-Based Inpainting. In plain English: it takes a photo of a product (the "Reference") and a photo of a person with a hole cut out where the product should go (the "Mask"), and it fills that hole with the product so perfectly that it looks like the person was wearing it all along.

The paper's main goal? To make sure the AI doesn't just guess what the product looks like, but faithfully copies every tiny detail, from the texture of the leather to the tiny letters on a soda can.

---

🚧 The Problem: Why Current AI Struggles

The authors point out three main reasons why current AI tools fail at this specific task:

1. Not Enough Practice: The AI hasn't seen enough examples of "people holding specific products" to learn the rules.
2. The "Blurry Memory" Issue: When AI tries to copy a reference image, it tends to "average out" the details. It remembers the idea of a logo but forgets the sharp edges. It's like trying to draw a friend's face from memory; you get the nose and eyes, but the specific mole on the cheek might disappear.
3. Too Vague: Existing tools are told to "fix this area," but they aren't forced to look at the high-frequency details (the sharp, crisp edges) of the original product.

---

🛠️ The Solution: HiFi-Inpaint's Secret Weapons

To fix these problems, the researchers built a new system with three special tools:

1. The "Self-Taught" Library (HP-Image-40K)

• The Analogy: Imagine you want to learn how to bake the perfect cake, but you don't have a cookbook. So, you hire a robot to bake 40,000 cakes for you, and then you use a robot inspector to throw away the burnt ones. Now you have a perfect library of 40,000 good cakes to study.
• What they did: Since real photos of people holding specific products are hard to find, they used an AI to generate 40,000 fake but realistic examples. They then used another AI to filter out the bad ones, creating a massive, high-quality training dataset called HP-Image-40K.

2. The "Sharpness Filter" (Shared Enhancement Attention - SEA)

• The Analogy: Imagine you are trying to copy a drawing. You have the original drawing (the product) and a blank space (the mask).
  • Old AI: Looks at the original, gets a general idea, and draws.
  • HiFi-Inpaint: It has a special "X-Ray glasses" mode. It looks at the original product, strips away the colors and shapes, and keeps only the sharp, high-frequency lines (the edges, the text, the texture). It then "shouts" these sharp details into the blank space while it's drawing.
• How it works: They use a technique called Shared Enhancement Attention. It takes the "high-frequency map" (the sharp edges) of the product and injects it directly into the AI's brain while it's generating the image. This forces the AI to pay attention to the tiny details, not just the big shapes.

3. The "Pixel-Perfect Inspector" (Detail-Aware Loss - DAL)

• The Analogy: When a student turns in homework, a teacher usually checks if the answer is "mostly right." But for this project, the teacher is a microscope.
• What they did: They created a new rule for the AI's training called Detail-Aware Loss. Instead of just checking if the picture looks good overall, this rule zooms in on the tiny, sharp parts (like text and logos). If the AI blurs a letter or smudges a pattern, it gets a "penalty." This forces the AI to learn that sharpness matters.

---

📊 The Results: Why It Matters

The team tested HiFi-Inpaint against other top AI tools. Here is what happened:

• Text & Logos: Other AIs turned the text on a bottle into gibberish. HiFi-Inpaint kept the text perfectly readable.
• Texture: Other AIs made the fabric look smooth and fake. HiFi-Inpaint kept the rough texture of the material.
• Consistency: Even when the "hole" to fill was very small, HiFi-Inpaint didn't get confused. It stayed true to the original product.

The Verdict: HiFi-Inpaint is like a master photographer who can take a product photo, paste it onto a model, and make it look like a real photo taken in a studio, down to the reflection on a button.

🚀 Why Should You Care?

This isn't just about making pretty pictures. This technology is a game-changer for:

• E-commerce: Imagine buying a shirt online and seeing it on a model that looks exactly like you, wearing the exact shirt you ordered, with the correct logo.
• Advertising: Brands can create thousands of ads instantly without hiring models or photographers, while ensuring their product looks 100% accurate.
• Trust: When the product looks real, people trust the brand more.

In short, HiFi-Inpaint teaches AI to stop guessing and start copying with precision, ensuring that the digital world looks as crisp and real as the physical one.

Technical Summary

1. Problem Statement

The paper addresses the challenge of generating human-product images (images where a human seamlessly holds or interacts with a specific product) for advertising and e-commerce. While diffusion models have advanced text-to-image generation, existing methods struggle with high-fidelity reference-based inpainting.

Key Challenges:

• Detail Preservation: Current models often fail to preserve fine-grained product details (textures, logos, specific text, patterns) when integrating a product into a masked human image.
• Spatial & Appearance Alignment: Standard diffusion denoising processes tend to "average" or hallucinate content, leading to inconsistencies in shape and branding.
• Data Scarcity: There is a lack of large-scale, diverse training data specifically curated for high-fidelity human-product interactions.
• Coarse Supervision: Existing training objectives (like standard MSE in latent space) are insufficient for guiding the reconstruction of high-frequency details.

2. Methodology: HiFi-Inpaint

The authors propose HiFi-Inpaint, a novel framework built upon the FLUX.1-Dev diffusion transformer (DiT) architecture. The method consists of three core components:

A. Dataset Construction: HP-Image-40K

To overcome data scarcity, the authors constructed a new dataset of 40,000+ samples using a self-synthesis pipeline with automatic filtering:

1. Diptych Synthesis: Using FLUX.1-Dev to generate paired images (Left: Product only; Right: Product + Human) based on specific prompts to ensure semantic alignment.
2. Segmentation: Using edge detection (Sobel filter) to split the diptychs.
3. Semantic Filtering: Using YOLOv8 and CLIP similarity to ensure the product in the "Human-Product" image matches the reference product.
4. Textual Filtering: Using InternVL to extract and compare text on products, ensuring brand labels and instructions are preserved across the pair.

B. High-Frequency Map-Guided DiT Framework

The core architecture introduces two mechanisms to enhance detail retention:

1. Token Merging: The model concatenates encoded tokens from the masked human image ($I_h$), the product reference ($I_p$), and the noisy ground truth ($I_{gt}$) to create joint visual tokens.
2. Shared Enhancement Attention (SEA):
   • Concept: A dual-stream mechanism where a second branch processes high-frequency map tokens derived from the product reference image (extracted via Discrete Fourier Transform and high-pass filtering).
   • Mechanism: The high-frequency tokens are processed through the same DiT blocks as the main branch (parameter sharing). The output is masked to the inpainting region and added to the main branch's output via a learnable weighting factor ($\alpha_i$).
   • Goal: This explicitly injects high-frequency details (edges, textures, text) into the generation process, refining the visual tokens in the masked region.

C. Detail-Aware Training Strategy (DAL)

To complement latent-space supervision, the authors introduce a Detail-Aware Loss (DAL):

• Mechanism: Instead of minimizing error only in the latent space, DAL applies pixel-level supervision on the high-frequency components of the generated image.
• Formula: $L_{DA} = || H(\hat{I}_{gt}) \odot M - H(I_{gt}) \odot M ||_2^2$, where $H(\cdot)$ is the high-frequency extraction operator and $M$ is the mask.
• Goal: This forces the model to accurately reconstruct fine details (like text and logos) that are often lost in standard latent diffusion training.

3. Key Contributions

1. HiFi-Inpaint Framework: A new reference-based inpainting architecture featuring Shared Enhancement Attention (SEA) to refine fine-grained features and Detail-Aware Loss (DAL) for precise pixel-level supervision.
2. HP-Image-40K Dataset: A large-scale, high-quality dataset of 40k+ human-product images generated via a rigorous self-synthesis and filtering pipeline, addressing the lack of training data.
3. State-of-the-Art Performance: Demonstrated superior ability to preserve product details (text, logos, textures) compared to existing methods like Paint-by-Example, ACE++, and Insert Anything.

4. Experimental Results

The paper evaluates HiFi-Inpaint on both synthetic and real-world datasets against four baselines.

Quantitative Metrics:

• Visual Consistency: HiFi-Inpaint achieved the highest CLIP-I (95.0%) and DINO (91.9%) scores, indicating superior alignment with the reference product.
• Detail Preservation: It outperformed all baselines in SSIM-HF (42.9%), a metric specifically designed to measure high-frequency structural similarity.
• Aesthetics: Achieved top scores in Q-Align-IQ (4.36) and competitive results in LAION-Aes.

Qualitative Results:

• Visual comparisons show that while other methods often generate blurry textures, distorted logos, or fail to integrate the product correctly, HiFi-Inpaint produces seamless integrations with faithful preservation of text, patterns, and branding.
• User Study: In a blind user study, HiFi-Inpaint was preferred by 36.4% of users for text alignment and 41.5% for visual consistency, significantly outperforming the next best method.

Ablation Studies:

• Removing the Synthetic Data (HP-Image-40K) significantly dropped performance in text alignment and visual consistency.
• Removing SEA resulted in the loss of intricate patterns and logos.
• Removing DAL led to blurry reconstructions and a failure to preserve high-frequency details like text.

5. Significance

• Commercial Impact: This work directly addresses a critical bottleneck in e-commerce and digital marketing: the ability to automatically generate high-quality, brand-consistent product images without manual design effort.
• Technical Advancement: It moves beyond global semantic manipulation by introducing specific mechanisms (high-frequency token injection and pixel-level high-frequency loss) to solve the "detail preservation" problem in diffusion-based inpainting.
• Generalizability: The method demonstrates robustness even in challenging real-world scenarios (varying lighting, occlusion, and small mask regions), suggesting it can be deployed in diverse practical applications beyond the synthetic training data.

In summary, HiFi-Inpaint sets a new standard for reference-based inpainting by successfully balancing global semantic coherence with the rigorous preservation of fine-grained product details, a capability essential for real-world commercial image generation.