💻 computer science

DuET: Dual Expert Trajectories for Diffusion Image Editing

DuET is a training-free inference method that enhances diffusion-based image editing by temporarily transitioning through a text-to-image phase to better align with target instructions while preserving structural integrity, with a selective variant further optimizing the balance between edit fidelity and source preservation.

Original authors: Lidia Troeshestova, Alexander Ustyuzhanin, Sergey Kastryulin

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lidia Troeshestova, Alexander Ustyuzhanin, Sergey Kastryulin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where you can talk to a computer and ask it to draw anything you can imagine. For years, scientists have built "text-to-image" models that act like magical sketchpads: you type "a cat wearing a hat," and they paint it. But what if you want to change a picture that already exists? Maybe you want to swap the cat for a dog, or turn a sunny day into a stormy one. This is where "image editing" models come in. They take your original photo and your new instructions, then try to rewrite the picture while keeping the parts you didn't ask to change exactly the same.

The tricky part is that these editing models are like very cautious artists. Because they are glued to the original photo the entire time they are working, they sometimes get too scared to make big changes. If you ask them to turn a small house into a giant castle, they might just make the house slightly bigger, leaving the rest of the scene looking weird or stuck in the past. They are so focused on not messing up the original that they fail to follow your new instructions fully. This paper tackles that specific problem: how do we get these digital artists to be bold enough to make big changes without losing the soul of the original picture?

The authors introduce a clever new trick called DuET (Dual Expert Trajectories). Think of an image editing process like a long journey from a blank canvas to a finished masterpiece. Usually, the computer takes this journey in "Edit Mode," where it constantly looks at the original photo to make sure it doesn't stray too far. The DuET method suggests a different route: for a short, specific part of the journey, the computer should stop looking at the original photo entirely. Instead, it switches to "Text-to-Image Mode," where it only listens to your description of the new scene. It's like a musician who usually plays along with a backing track (the original photo) but, for a few bars of the song, puts on noise-canceling headphones and improvises a solo based only on the sheet music (your instructions). This allows the computer to completely reorganize the scene to match your vision.

After this brief "solo," the computer takes off the headphones and switches back to "Edit Mode" for the rest of the journey. Now that the big, structural changes have been made, it can look back at the original photo to fill in the fine details and make sure the lighting and textures still look natural. The paper shows that this "Edit → Text-to-Image → Edit" dance works much better than just sticking to one mode the whole time. It makes the final result look more like what you asked for and feel more natural, especially for big changes like replacing a whole object or changing the setting.

However, the researchers found a catch. When the computer stops looking at the original photo, even for a moment, it sometimes changes parts of the picture you wanted to keep exactly the same. It's a trade-off: you get a better, more accurate edit, but you might lose a tiny bit of the original image's exactness. The paper measures this using a score called SSIM, which checks how similar the pixels are. They found that while the new method improves the "fun" parts of the image (like how well it follows instructions and how natural it looks), the "preservation" score drops a little bit.

But here is the really cool part: the authors suggest this trade-off isn't a hard rule. They created a smarter version called Selective DuET. This version acts like a traffic cop. Before the journey starts, it takes a quick peek at the image to see how much the computer is already "holding on" to the original. If the image needs a huge change (like turning a house into a tree), the traffic cop says, "Go ahead, take the solo!" But if the image only needs a tiny tweak, the cop says, "No need to switch modes, just keep editing." By only using the risky "solo" method when it's absolutely necessary, they found a way to get the high-quality edits without making the preservation score drop in a way that humans can actually notice.

In short, the paper proves that you don't need to train a new, super-complex robot to edit images better. You just need to teach the existing robots when to stop looking at the original photo and when to start listening only to your words. It's a simple, free upgrade that makes digital art editing more flexible and creative, showing that sometimes, to get the perfect result, you have to be willing to let go of the past for just a moment.

Technical Summary: DuET (Dual Expert Trajectories) for Diffusion Image Editing

1. Problem Statement

Recent instruction-based image editing models, such as FLUX2-Klein and unified multimodal models (UMMs), condition the denoising process on a source image at every step to preserve structural integrity. While this "persistent source-image anchoring" ensures that unedited regions remain stable, the paper argues it creates a fundamental limitation: when the target scene diverges substantially from the input (e.g., global rewrites, object replacements), the model struggles to fully execute the instruction. This often results in outputs that are less natural than pure text-to-image (T2I) generation, exhibiting compositing artifacts, texture clashes, and lighting inconsistencies.

Current pipelines typically operate in a single mode for the entire trajectory—either strictly edit mode (source + instruction) or T2I mode (caption only)—failing to leverage the model's dual capabilities to modulate source anchoring dynamically during a single edit.

2. Methodology: DuET

The authors propose DuET (Dual Expert Trajectories), a training-free inference framework that dynamically switches conditioning modes within a single denoising trajectory. The core mechanism involves a temporary transition from edit mode to caption-only T2I mode and back to edit mode (E → T2I → E).

2.1 The Switching Mechanism

Standard Edit (E): Conditions on the source image ( $x_{src}$ ) and edit instruction ( $p_{edit}$ ) at all steps $k$ .
Standard T2I: Conditions only on a target caption ( $c_{tgt}$ ) at all steps.
DuET Schedule: The sampler runs in Edit mode for the initial steps, switches to T2I mode (using a target caption generated by an external Vision-Language Model) for a specific interval $[k_1, k_2)$ $[k_{1}, k_{2})$ , and then reverts to Edit mode for the remaining steps.
- The "experts" in this context are not separate network weights but distinct conditioning modalities.
- This requires no weight updates, no additional forward passes, and no model finetuning.

2.2 Target Caption Generation

When the trajectory enters the T2I phase, the model requires a target caption ( $c_{tgt}$ ) describing the intended outcome. DuET obtains this using an external VLM (e.g., Gemini-2.5-pro) conditioned on the source image and the original edit instruction.

2.3 Selective DuET (Adaptive Routing)

Recognizing that fixed intervals may unnecessarily degrade source preservation for edits that do not require global changes, the authors introduce Selective DuET. This variant uses lightweight "attention-probe" signals read from the early stages of a pure-editing pass to decide whether to switch modes.

Signals: Two metrics are computed at an early probe step ( $k=9$ $k = 9$ ):
1. $\pi$ (Preservation Pressure): A high-quantile measure of how strongly the model's attention is anchored to the source image.
2. $cv$ (Coefficient of Variation): A measure of the spatial locality of the reliance on the source.
Decision Rule: If the edit shows high preservation pressure ( $\pi > 0.22$ ) or high spatial locality ($cv > 0.50$), the system stays in pure Edit mode. If the edit is spatially uniform and has low preservation pressure (indicating a global rewrite), the system triggers the T2I excursion.

3. Key Contributions

Training-Free Task Switching: A framework that modulates source anchoring by alternating between edit and T2I modes along a single denoising path, improving instruction relevance and fidelity without finetuning or extra compute.
Comprehensive Evaluation: An extensive evaluation on FLUX2-Klein (4B/9B) and BAGEL models across multiple benchmarks (GEdit, ImgEdit, MagicBrush, GIE-Bench), utilizing matched controls to isolate the effects of dropping the source image versus switching the caption.
Selective DuET: A per-edit routing strategy demonstrating that the trade-off between edit fidelity and source preservation is not fundamental. By routing based on attention signals, it improves fidelity and naturalness while keeping source preservation perceptually indistinguishable from the baseline.

4. Experimental Results

The authors evaluated DuET on three base models using 50-step sampling.

Fixed Interval Performance: Fixed DuET schedules (e.g., switching to T2I for intervals like $[10, 16)$ , $[25, 35)$ , or $[35, 45)$ ) consistently improved semantic consistency (G_SC), perceptual quality (G_PQ), and functional correctness across all models compared to the pure editing baseline.
The Trade-off: These gains came with a measurable cost in source preservation (lower pixel SSIM and Unmasked-CLIP scores). The paper notes that this trade-off is not fundamental but depends on the switching schedule.
Ablation Studies:
- Conditioning Variants: The gains are driven primarily by dropping the source image conditioning during the T2I interval, rather than just switching the text prompt. Variants that kept the source image (I2IC) or kept the original prompt (T2IEP) yielded significantly lower improvements in fidelity and naturalness.
- Timing: Narrower windows and specific placements (e.g., $[10, 16)$ ) offered the best balance of fidelity and preservation for FLUX2-Klein 4B/9B, while later windows favored semantic consistency on BAGEL.
Selective DuET Performance: By applying the T2I excursion only to edits that benefit from it (global rewrites), Selective DuET achieved high functional correctness while maintaining source preservation scores (SSIM $\approx 0.786$ ) that were perceptually indistinguishable from the baseline, effectively breaking the fidelity-preservation trade-off observed in fixed schedules.

5. Significance and Claims

The paper claims that DuET demonstrates a previously underutilized capability of unified multimodal models: the ability to switch conditioning modalities within a single inference trajectory.

Modesty of Claims: The authors explicitly state that Selective DuET is a "proof of existence" rather than a universal solution. They acknowledge limitations, including the reliance on early trajectory signals for routing (which limits decision depth), the use of VLM judges rather than human studies for perceptual thresholds, and the dependency on external VLMs for caption generation, which can introduce semantic mismatches if the caption diverges from the instruction's implicit intent.
Core Insight: The paper concludes that persistent source anchoring limits the naturalness of edits for divergent scenes. By temporarily relaxing this anchoring via a T2I phase, DuET allows the denoising trajectory to move toward the target distribution while retaining the structural benefits of image-conditioned editing for the final steps. This suggests that "how, when, and how often" to switch conditioning modes is a critical area for future systematic study in diffusion-based editing.