Technical Summary: DARS – Dual-Level Credit Assignment RL with Structured Reasoning for Instruction-Based Image Editing

Problem Statement

Instruction-based image editing systems increasingly adopt a planner-renderer pipeline: a Vision-Language Model (VLM) interprets a natural language instruction to generate an intermediate edit plan, which a diffusion model then executes to produce the final image. While this decomposition improves controllability, it introduces a dual-level credit assignment problem when optimizing via Reinforcement Learning (RL) with final-image rewards:

1. Cross-Module Ambiguity: A poor final image does not reveal whether the failure originated from a flawed plan (planner error) or a failure to execute a correct plan (renderer error). Standard joint RL strategies either update modules uniformly (diluting the learning signal) or optimize them separately (missing coordinated improvements).
2. Within-Planner Opacity: Even when the planner is the bottleneck, free-form reasoning traces make it difficult to localize which specific part of the plan (e.g., the modification intent, preservation constraints, or execution hints) is incorrect, preventing targeted corrections.

Existing methods struggle to distinguish between "plan-dominant" and "render-dominant" failure modes, leading to inefficient optimization where low-scoring edits do not indicate where additional optimization effort should be focused.

Methodology: DARS Framework

DARS (Dual-Level Credit Assignment RL with Structured Reasoning) addresses these issues through a joint RL framework that decomposes credit assignment across two levels: cross-module routing and within-planner diagnosis.

1. Structured Reasoning and Slot-Wise Credit Assignment

To enable fine-grained diagnosis within the planner, DARS replaces opaque, free-form reasoning traces with a four-field structured output:

• Modify: Specifies the requested visible change.
• Preserve: States what must remain unchanged.
• Overall: Captures scene-level coherence and global consistency.
• Tips: Provides execution hints for the renderer.

Prefix-Gated Reward: The framework employs a causal ordering where later slots are only rewarded if preceding slots are correct. A prefix-gated reward mechanism ensures that if the Modify slot fails, downstream slots (Preserve, Overall, Tips) receive zero reward, preventing the model from being rewarded for "good" downstream text that compensates for a fundamental planning error.

Token-Level Advantage Reweighting: For policy optimization, the advantage signal is reweighted at the token level based on slot scores. Tokens in a specific slot receive stronger negative updates if that slot is the primary cause of a low overall plan score, while positive updates are reinforced only if the entire causal chain of slots is sound.

2. Rollout Variance Decomposition for Cross-Module Routing

To determine whether to emphasize the planner or the renderer for a given sample, DARS utilizes multi-plan, multi-render rollouts ($M$ plans, $K$ renderings per plan). It decomposes the variance of the shared reward ($R_{share}$) into two components using the law of total variance:

• Across-Plan Variability ($U_{plan}$): Measures how much the expected reward changes when switching plans. High values indicate that the current plan is the bottleneck (plan-dominant).
• Within-Plan Variability ($U_{rend}$): Measures reward fluctuation under a fixed plan. High values indicate that the renderer is unstable or failing to execute the plan (render-dominant).

These statistics drive soft module routing, assigning extra update weight to the module responsible for the dominant variability. This allows the system to dynamically route learning signals: samples with high $U_{plan}$ receive stronger planner updates, while those with high $U_{rend}$ receive stronger renderer updates.

3. Adaptive Curriculum Scheduling

DARS estimates sample hardness based on the mean reward across rollouts. An adaptive curriculum schedules training by prioritizing easier samples early in training and progressively introducing harder samples as the model improves. This hardness estimate works in tandem with the module routing: curriculum determines when a sample contributes strongly, while variance decomposition determines where (which module) the emphasis should be placed.

4. Optimization Objectives

The framework optimizes the planner and renderer jointly but with distinct objectives:

• Planner: Optimized using text-GRPO with token-level, slot-weighted advantages derived from the prefix-gated reward.
• Renderer: Optimized using flow-GRPO with per-render advantages.
• Weighting: Both objectives are scaled by a curriculum weight ($w_{cur}$) and a residual module-routing weight ($\alpha$ for planner, $\omega$ for renderer) derived from the variance decomposition.

Key Contributions

1. Dual-Level Credit Assignment Formulation: The paper formulates joint RL for planner-renderer editing as a dual-level problem, explicitly addressing both cross-module update allocation and within-planner reasoning diagnosis.
2. Rollout Variance Decomposition: A novel method to separate across-plan from within-plan variability, providing data-driven signals for soft module routing without requiring external failure labels.
3. Structured Planner with Prefix Gating: The introduction of a four-field structured reasoning schema combined with prefix-gated rewards and token-level advantage reweighting, enabling localized planner optimization from post-render feedback.
4. Adaptive Curriculum Integration: A curriculum scheduling mechanism driven by rollout-derived hardness estimates that complements the module routing strategy.

Experimental Results

DARS was evaluated on five benchmarks: KRIS-Bench (reasoning-intensive), RISE-Bench (reasoning-informed), ImgEdit-Bench (general editing), GEdit-Bench-EN (semantic consistency), and PICA-Bench (physical realism).

• Performance: DARS achieved the best overall scores across all five benchmarks.
• Reasoning Gains: The largest improvements were observed on reasoning-intensive tasks (KRIS-Bench: +8.57 over the controlled baseline; RISE-Bench: +1.80), validating the efficacy of the structured reasoning and credit assignment mechanisms for complex edits.
• Ablation Studies:
  • Removing the adaptive curriculum or soft routing led to significant performance drops.
  • Replacing the structured planner with free-form reasoning caused divergence in some settings and lower performance in others, highlighting the stabilizing and diagnostic value of the four-field schema.
  • The prefix-gated reward outperformed flat averaging and weighted summation, confirming the importance of causal dependency in credit assignment.
• Validation: The routing signals (plan-dominant vs. render-dominant) showed high agreement (>86% accuracy, >0.93 AUROC) with pseudo-labels generated by GPT-5, confirming that rollout statistics effectively capture failure modes.

Significance and Claims

The paper claims that DARS provides a principled solution to the under-specified nature of credit assignment in two-stage image editing. By distinguishing between plan-dominant and render-dominant failures, DARS avoids the inefficiency of uniform joint optimization. The structured reasoning output transforms outcome-level feedback into localized supervision, allowing the planner to diagnose and correct specific semantic or compositional errors that free-form traces obscure.

The authors emphasize that their approach does not rely on engineering complex cold-start formats or expert-designed prompting recipes but rather on a compact, semantically aligned schema that integrates naturally with existing VLM backbones. The results suggest that explicit credit assignment mechanisms are critical for scaling reasoning capabilities in instruction-based image editing, particularly for tasks requiring temporal, spatial, and physical reasoning.