Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos

1. Problem Definition: Egocentric Instructed Visual State Transition (EIVST)

The paper addresses a gap in current generative AI: the inability to model complex physical transformations from an egocentric (first-person) perspective.

• Task: Given an initial frame ($I_1$), a target frame ($I_N$), and a brief textual action instruction, the model must generate a coherent sequence of intermediate frames that depict the object's state transition.
• Challenges:
  1. Reasoning: The model must understand the visual scenes of the start and end states and logically infer the intermediate steps required to bridge them, rather than just interpolating pixels.
  2. Consistency: The generated video must follow the specific action instruction while maintaining the consistent appearance of objects across the transition (avoiding morphing artifacts).
  3. Limitations of Existing Methods: Current Image-to-Video (I2V) models often fail to reach the specific target state or hallucinate incorrect intermediate steps. Video frame interpolation methods lack semantic understanding of how an action occurs, and standard video prediction models lack the target state as a visual anchor.

2. Methodology: The EgoIn Framework

The authors propose EgoIn, a two-stage framework that combines a fine-tuned Vision-Language Model (VLM) for reasoning with a diffusion-based video generation model for synthesis.

Stage 1: Transition Process Modeling (TransitionVLM)

Instead of relying on a pre-trained VLM which often hallucinates, the authors introduce TransitionVLM, a model fine-tuned on a curated dataset to infer the logical steps between states.

• Data Curation: They use GPT-4o to generate training data. To ensure accuracy, they provide GPT-4o with:
  • State-aware inputs: Initial and target frames with object bounding boxes (detected by Qwen2.5-VL) to focus on manipulated objects.
  • Transition-aware inputs: The full video sequence (for training data generation) to learn temporal dynamics.
• Architecture: Based on Qwen2.5-VL, the model is fine-tuned using LoRA (Low-Rank Adaptation) with specific adapters (S-Adapter for state, T-Adapter for transition).
• Output: The VLM outputs:
  1. Descriptions of the initial and target object states.
  2. A sequence of $K$ intermediate transition steps.
  3. Temporal ranges ($R_j$) indicating which frames in the sequence correspond to each step.

Stage 2: Transition Conditioned Video Generation

The inferred transition steps are used to guide a diffusion-based video generation model (built upon DynamiCrafter).

• Transition Conditioning (TC) Module: This module integrates the VLM's outputs into the video generator.
  • Multi-modal Semantic State Encoding: Extracts features from the initial/target frames and VLM state descriptions using SQ-Formers and Self-Attention to create a state-aware representation ($\tilde{F}^S$).
  • Range-Aware Transition Encoding: Uses the predicted temporal ranges ($R_j$) to create frame-wise transition features. It employs a softened weighting mechanism (Gaussian resampling) to assign weights to specific transition steps for each frame, ensuring smooth temporal alignment.
  • These features are injected into the U-Net's cross-attention layers via a frame-wise mechanism.

Object-Aware Auxiliary Supervision (OAS)

To prevent object appearance drift (e.g., a bowl changing shape or color during the video), the authors introduce an auxiliary loss.

• Mechanism: A localization head is added to the U-Net to predict object masks.
• Training: The model is trained to minimize the cross-entropy loss between predicted masks and ground-truth masks (generated by SAM2 using the action instruction).
• Inference: The localization head is not used during inference, adding no computational overhead.

3. Key Contributions

1. EIVST Task Definition: Formalized a new task requiring generative models to bridge initial and target states with semantically meaningful, instruction-guided intermediate frames.
2. EgoIn Framework: A novel architecture integrating:
   • TransitionVLM: A domain-specific fine-tuned VLM that reduces hallucinations and infers multi-step transitions with temporal grounding.
   • Transition Conditioning (TC): A module that explicitly guides video generation with frame-wise, range-aware conditions.
   • Object-Aware Auxiliary Supervision (OAS): A training strategy to enforce object consistency and smooth motion.
3. Comprehensive Evaluation: Extensive experiments on human-object (Epic100, Ego4D) and robot-object (DualArm, Bridge) interaction datasets.

4. Experimental Results

The authors evaluated EgoIn against state-of-the-art methods including video interpolation (FILM, TRF, GI), video transition (SEINE), and text-guided interpolation (DC-Interp).

• Quantitative Performance: EgoIn significantly outperformed all baselines across four datasets on:
  • FVD (Fréchet Video Distance): Lower is better (e.g., 215.27 vs. 296.67 for Baseline on Epic100).
  • VTQ (Video Temporal Quality): Higher is better.
  • VTC/VIC (Text/Image Consistency): Higher is better.
• User Study: In a preference test with 40 participants, EgoIn received >70% of votes for "Reasonability" and "Motion Quality," significantly outperforming other methods. Users noted that EgoIn generated logical intermediate steps (e.g., opening a door before placing an item inside), whereas baselines often skipped steps or created unrealistic overlaps.
• Ablation Studies:
  • Removing TransitionVLM resulted in unrealistic state changes (e.g., objects appearing in impossible locations).
  • Removing TC led to poor temporal alignment of steps.
  • Removing OAS caused object appearance inconsistencies.
  • Optimal performance was found with a variable number of transition steps ($K \in [1, 4]$), rather than a fixed number.

5. Significance and Future Work

• Significance: This work bridges the gap between high-level human cognition (understanding cause-and-effect in physical transformations) and machine execution. It is crucial for embodied AI, robotic manipulation planning, and interactive educational tools where visualizing the "how" of an action is as important as the result.
• Limitations: The current framework struggles with long-horizon transitions involving significant scene or viewpoint changes.
• Future Work: The authors plan to improve multi-view consistency and model long-range dependencies to handle dynamic environments and large viewpoint shifts.