🤖 machine learning

Do-Undo Bench: Reversibility for Action Understanding in Image Generation

This paper introduces the Do-Undo benchmark, a novel framework that evaluates vision-language models' ability to understand real-world action dynamics by requiring them to generate plausible scene transformations and subsequently reverse them to their original states.

Original authors: Shweta Mahajan, Shreya Kadambi, Hoang Le, Rajeev Yasarla, Apratim Bhattacharyya, Munawar Hayat, Fatih Porikli

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shweta Mahajan, Shreya Kadambi, Hoang Le, Rajeev Yasarla, Apratim Bhattacharyya, Munawar Hayat, Fatih Porikli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magic photo album. You can tell the album, "Open the refrigerator," and it shows you a picture of an open fridge. But here's the catch: most of today's smartest AI photo albums are like magicians who only know the trick, not the physics. They might show you an open fridge, but if you ask them to "close it again," they might accidentally delete the fridge, change the color of the wall, or leave the door floating in mid-air. They don't truly understand that opening a door and closing it are two sides of the same coin.

This paper introduces a new test called Do-Undo Bench to see if AI can actually understand how the real world works, rather than just guessing what a picture should look like.

Here is the breakdown of their idea using simple analogies:

1. The Problem: The "One-Way Street" AI

Current AI models are great at following instructions like "add a cat" or "remove the tree." But they struggle with actions that change the state of an object.

The Analogy: Think of an AI that knows how to build a sandcastle. If you ask it to "build a sandcastle," it does a great job. But if you then ask it to "undo that and put the sand back in the bucket," it might just delete the picture entirely or turn the sand into water. It doesn't understand that the sand moved from the bucket to the castle and can move back.

2. The Solution: The "Do-Undo" Challenge

The researchers created a new game for AI. To pass the test, the AI has to do two things in a row:

Do: Look at a picture (e.g., a closed drawer) and a command ("Open the drawer"), then generate a new picture showing the drawer open.
Undo: Look at that new picture of the open drawer and a reverse command ("Close the drawer"), then generate a picture that looks exactly like the original closed drawer.

If the AI can do both perfectly, it proves it understands cause and effect. It knows that "opening" pushes the drawer out, and "closing" pulls it back in, without changing the rest of the kitchen.

3. The Dataset: Real-Life Kitchen Videos

To teach the AI this lesson, the researchers didn't just make up random pictures. They used thousands of real videos from the Epic-Kitchens dataset (people cooking in their own homes).

The Process: They took video clips of real actions, like "picking up a spoon" or "turning on a tap."
The Filter: They only kept actions that could be reversed. You can open and close a drawer, but you can't really "un-cut" a piece of paper. So, they threw away the irreversible ones.
The Expansion: The original video descriptions were very short (like "open drawer"). The researchers used a smart AI to expand these into long, detailed stories (e.g., "Use your right hand to pull the wooden drawer backward until it is fully open, revealing the silverware inside"). This gives the AI a much clearer map of what to do.

4. The Results: AI is Still Learning

The researchers tested the world's best AI models on this new "Do-Undo" game.

The Score: Even the smartest models failed. They were good at making the picture look pretty (high visual quality), but they were terrible at getting the physics right.
- Example: When asked to "turn off the tap," some models kept the water flowing or made the tap disappear. When asked to "put the knife back," they might leave the knife floating in the air.
The Fix: The researchers took one model (called BAGEL) and trained it specifically on their "Do-Undo" dataset.
- The Result: This trained model got much better at the game. It learned that if you open a drawer, you must be able to close it and return to the exact starting point. It started understanding that actions change the state of objects, not just the style of the image.

5. Why This Matters

The paper argues that for AI to be truly useful in the real world (like helping a robot arm in a kitchen), it needs to understand dynamics, not just static images.

The Metaphor: Right now, AI is like a child who has memorized a dictionary of words but hasn't learned how to speak in sentences. They know what "open" and "close" mean, but they don't know how those words interact with the physical world.
The Goal: This "Do-Undo" test is a new report card. It forces AI to prove it understands that the world is reversible and logical, paving the way for smarter robots and virtual assistants that don't just generate pretty pictures, but actually understand how things work.

In short: The paper says, "We built a test where AI has to open a door and then close it perfectly to prove it understands physics. Current AI fails this test, but if we train it on real-life reversible actions, it starts to get it right."

Technical Summary: Do-Undo Bench: Reversibility for Action Understanding in Image Generation

Problem Statement
Current Vision-Language Models (VLMs) have achieved significant progress in text-driven image synthesis and editing. However, they remain fundamentally limited in their ability to understand and simulate the physical dynamics of real-world scenes. Existing approaches primarily focus on object-level manipulations (e.g., adding or removing objects) or maintaining stylistic properties like lighting, often neglecting the underlying cause-and-effect relationships that govern physical interactions. Consequently, models struggle to generate images that plausibly reflect the state changes resulting from specific physical actions, such as picking up an object or opening a drawer, without observing a continuous video sequence. This gap hinders the deployment of VLMs as reliable synthetic data generators for robotics and embodied AI agents, which require an understanding of how actions transform environments.

Methodology
To address this limitation, the authors introduce the Do-Undo task, a novel benchmark and training paradigm that requires models to simulate the outcome of a real-world action and then reverse it to the original state.

Task Formulation: The task involves an input image ( $I_O$ ) containing an object and an agent (e.g., a hand). Given a forward action prompt ( $P_F$ ), the model must generate the "Do" image ( $I_F$ ) representing the state after the action. Crucially, given a reverse prompt ( $P_R$ ) and the generated $I_F$ , the model must generate the "Undo" image ( $I_R$ ) that reconstructs the original input state $I_O$ . This forward-reverse requirement tests genuine cause-and-effect understanding rather than superficial semantic editing.
Dataset Curation (Do-Undo Dataset): The authors curated a large-scale dataset from the Epic-Kitchens video dataset, which features first-person perspectives of humans performing daily tasks.
- Selection: They filtered for high-quality frames with reversible actions (e.g., "open/close," "pick-up/put-down," "turn-on/turn-off") and consistent backgrounds.
- Prompt Expansion: Since original annotations are short, they employed a VLM (Qwen3-VL) to expand action narrations into detailed prompts. These prompts include visual context, object attributes (color, material), agent details (hand posture), and spatial relationships.
- Structure: The dataset consists of tuples $(I_O, P_F, I_F, P_R)$ , totaling 22,529 training samples and 451 test samples across 10 action classes.
Model Training: The authors fine-tuned BAGEL, a unified vision-language model capable of interleaved text and image generation. The training objective minimizes the mean-squared error (via rectified flow matching) for generating the forward image $I_F$ conditioned on $I_O$ and $P_F$ , and subsequently generating the reverse image $I_R$ conditioned on $I_F$ and $P_R$ , with the constraint that $I_R$ should match $I_O$ .

Key Contributions

Do-Undo Task Formulation: A new task requiring models to generate the visual outcome of an action and accurately invert it to reconstruct the original scene. This explicitly tests the capture of cause-and-effect dynamics.
Do-Undo Benchmark and Dataset: A high-quality dataset of reversible actions extracted from real-world videos, accompanied by a specialized prompting strategy to ensure physically consistent visual generation.
Empirical Evaluation: A comprehensive evaluation demonstrating that current state-of-the-art models (including Qwen-Image and Flux-Kontext) struggle with action reversibility, revealing a fundamental gap in action reasoning despite high semantic fidelity.
Baseline Improvement: The development of BAGEL-DoUndo, a model fine-tuned on the Do-Undo dataset, which shows improved fidelity and consistency in action-aware generation compared to the baseline.

Results

Zero-Shot Performance: State-of-the-art models like Qwen-Image and BAGEL achieved high semantic similarity scores (DINO, CLIP) but performed poorly on action understanding metrics (Action Accuracy). For instance, Qwen-Image achieved ~52% accuracy on forward actions and ~29% on reverse actions, often hallucinating objects or failing to reverse the state correctly.
Fine-Tuning Impact: The BAGEL-DoUndo model significantly outperformed the baseline BAGEL. It achieved higher action accuracy (58.77% forward, 36.26% reverse) and better semantic consistency (DINO scores >0.83).
Ablation Studies: Training only on forward actions (even with expanded prompts) failed to induce robust action understanding. The inclusion of reverse image pairs was identified as critical for performance gains.
Multi-Turn Generation: In multi-turn scenarios where models generate a sequence of actions, BAGEL-DoUndo maintained better background consistency and object state accuracy compared to the baseline, as evidenced by optical flow error (EPE) analysis.
User Study: Human evaluators preferred BAGEL-DoUndo over the baseline 66.7% of the time, citing better adherence to action semantics and object interactions.

Significance and Claims
The paper claims that the Do-Undo benchmark serves as a critical testbed for evaluating and advancing action-aware generation in multimodal systems. The authors argue that the inability of current models to reverse actions highlights a lack of genuine physical reasoning, which is essential for reliable embodied AI. By demonstrating that explicit supervision on reversible actions improves generation fidelity, the work suggests a pathway toward developing VLMs that can reason about real-world dynamics and generate physically plausible synthetic data for robotics and simulation. The authors remain modest, noting that their work is currently limited to kitchen environments and general-purpose VLMs, with future work needed to extend these benchmarks to specialized embodied environments and more complex causal relationships.