Technical Summary: Taming I2V models for Image HOI Editing

1. Problem Statement

Current instruction-based image editing methods excel at modifying static attributes (e.g., color, style) but struggle significantly with Human-Object Interaction (HOI) editing. HOI editing requires a rational, verb-driven pairwise modification where the model must synthesize a contextually logical interaction involving pre-existing subjects and objects without regenerating them or hallucinating new entities.

The field faces two critical gaps:

1. Lack of Dedicated Benchmarks: Existing benchmarks conflate HOI with static attributes, relying on global metrics (e.g., CLIP-score) that cannot simultaneously assess dynamic interaction validity and the preservation of entangled human-object pairs.
2. Insufficient Evaluation Protocols: Standard metrics fail to discern specific targets or focus on pairwise context, lacking the granularity to evaluate the consistency and rationality of interacting pairs.

Furthermore, while Image-to-Video (I2V) models possess inherent temporal generation capabilities suitable for dynamic remodeling, their application to HOI editing has been underexplored. Static models often fail to model the temporal process of interactions, leading to "editing inertia" (failing to change) or "hallucination" (generating new objects instead of modifying existing ones).

2. Methodology

The paper proposes a comprehensive framework consisting of a new benchmark, a novel evaluation metric, and an agentic self-correcting system.

A. HOI-Edit Benchmark

The authors introduce HOI-Edit, the first benchmark structured around three progressive cognitive levels to assess editing capabilities:

• L1: Foundational Edits: Evaluates the shift from static appearances to dynamic interactions (e.g., changing "holding" to "riding"). It tests relation transformation and strict entity preservation.
• L2: Context Spatial Understanding: Assesses the model's grasp of spatial relationships for initial entity selection (e.g., "topmost apple") and terminal state placement (e.g., "into the vase").
• L3: Causal and Physical Reasoning: Requires simulating event causal chains with logical and physical plausibility. This includes inferring unstated procedural sub-actions (e.g., removing a lid before stirring) and adhering to physical laws (e.g., illumination consistency, non-rigid deformation).

B. HOI-Eval Metric

To address the limitations of global metrics, the paper proposes HOI-Eval, a grounded evaluation protocol based on the "Thinking with Pair-wise Regions" paradigm.

• Mechanism: It explicitly annotates paired bounding boxes for interacting subjects and objects.
• Process:
  1. Entity Localization: Tracks annotated ground-truth boxes from the original image to the edited image (using tracking or prompt-based detection).
  2. Identity Verification: Crops and compares the original and edited entities to compute Human (H) and Object (O) consistency scores.
  3. Interaction & Rationality Evaluation: Uses a Vision-Language Model (VLM) to answer context-aware questions regarding interaction status, spatial grounding, and physical/logical reasoning.
• Scoring: Introduces the I+Q&A metric, which penalizes interaction scores to zero if the context-aware reasoning questions are answered incorrectly, ensuring the model adheres to constraints rather than just achieving perceptual similarity.

C. SCPE (Self-Correcting Process Editing)

Recognizing that I2V models provide a temporal "replay of the failure process" (revealing why an error occurred, e.g., a hand steering toward the wrong target), the authors propose SCPE, an agentic, self-correcting framework.

• Architecture: A closed-loop system powered by specialized agents (Generator, Analyzer, Reflector, Curator) and a Dynamic Playbook.
• Workflow:
  1. Generator: Integrates user instructions with the current Playbook to create enhanced prompts for the I2V model.
  2. Analyzer: Samples frames from the generated video to diagnose errors (e.g., "person grabbed the wrong tool").
  3. Reflector: Distills "General Insights" from the failure, abstracting specific errors into universal principles (e.g., "Model prioritizes nearest objects; strong negation required").
  4. Curator: Updates the Playbook with new strategies, pitfalls, and templates based on insights.
• Outcome: The refined prompts guide the I2V model to re-generate the video, iteratively correcting errors until the target HOI is achieved. The final result is a selected frame from the corrected video.

3. Key Contributions

1. HOI-Edit Benchmark: A multi-level benchmark designed to evaluate the shift from static attribute editing to dynamic HOI editing, covering foundational, spatial, and causal cognitive levels.
2. HOI-Eval Metric: A region-sensitive evaluation protocol that leverages spatial anchors to verify entangled human-object pairs, offering robust identity preservation and interaction reasoning checks.
3. SCPE Framework: An agentic, playbook-driven framework that "tames" I2V models. It utilizes the temporal nature of video generation to diagnose and correct interaction failures iteratively, achieving performance comparable to or exceeding commercial state-of-the-art (SOTA) models.

4. Experimental Results

• Performance: On the HOI-Edit benchmark, SCPE (applied to the open-source Wan 2.2 I2V model) achieves SOTA performance across nearly all metrics. It outperforms the commercial SOTA model Nano Banana in interaction scores (I) and I+Q&A scores across all three cognitive levels.
• Identity Preservation: While SCPE shows slightly lower identity scores than Nano Banana (attributed to the drastic spatial reconfigurations inherent in I2V generation), it significantly outperforms other open-source baselines.
• Ablation Studies:
  • The Playbook is crucial; removing it leads to marginal gains and severe identity drift.
  • Iterative Refinement: SCPE converges rapidly, with the first iteration (R1) already surpassing Nano Banana.
  • Diagnosability: The framework's success is attributed to leveraging video priors. Unlike static models where failures are opaque spatial artifacts, I2V models expose the process of failure, allowing SCPE to intercept errors at their source.
• Generalization: SCPE demonstrates robustness when applied to other backbones (e.g., TurboDiffusion) and other benchmarks (ByteMorph), confirming its scalability and agnostic nature to specific VLM reasoning engines.

5. Significance and Claims

The paper argues that robust HOI editing necessitates a paradigm shift from static pixel in-painting to dynamic process simulation.

• World Modeling Prerequisite: The work provides a rigorous testbed and a viable pathway for approximating physical and causal processes at the interaction level. This is presented as a fundamental prerequisite for building learned World Models, rather than serving as a full physical simulator.
• Diagnosability: The paper highlights the unique value of I2V models in providing a "replay of the failure process," transforming editing from blind trial-and-error into logical process correction.
• Practical Impact: The framework offers a cost-effective method to enhance existing I2V models without retraining, enabling high-quality, instruction-following dynamic editing that rivals commercial closed-source systems.

The authors acknowledge limitations, including inference latency compared to static methods, potential background drift inherited from video generation models, and the current scale of the benchmark, suggesting future work in camera control and dataset expansion.