🤖 AI

Thinking in Text and Images: Interleaved Vision--Language Reasoning Traces for Long-Horizon Robot Manipulation

This paper introduces Interleaved Vision-Language Reasoning (IVLR), a policy framework that generates explicit traces alternating between textual subgoals and visual keyframes to enable robust, long-horizon robot manipulation by combining logical coherence with geometric grounding, achieving state-of-the-art success rates on challenging benchmarks.

Original authors: Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang, Long Chen, Hangjun Ye, Xiaoshuai Hao, Wenbo Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to clean a messy kitchen. The task isn't just "pick up the spoon." It's a long story: "First, move the dirty plate to the sink, then grab the sponge, scrub the counter, and finally put the sponge back."

Current robot brains (called Vision-Language-Action policies) are like actors who are great at the next line of dialogue but terrible at remembering the whole script. They look at the sink, see a plate, and grab it. But if the task requires them to grab a cup before the plate, they might get confused, grab the wrong thing, or forget why they are there in the first place. They are "myopic"—they only see the immediate next step.

This paper introduces a new way for robots to think, called IVLR (Interleaved Vision–Language Reasoning). Here is how it works, using simple analogies:

1. The Problem: The "Amnesiac Actor"

Think of a standard robot as an actor who is given a script but has to memorize it line-by-line while performing. If the play is short, they are fine. But if the play is long (a "long-horizon" task), they forget the plot.

Text-only plans are like a script that says "Pick up the cup." It tells the robot what to do, but not where exactly or what the scene should look like. It's like a recipe without pictures.
Visual-only plans are like a movie reel of the future. It shows the robot what the scene looks like, but without words, the robot might not understand why the cup is there or in what order things happened.

2. The Solution: The "Storyboard Director"

The authors give the robot a Storyboard. Before the robot even moves a muscle, it pauses and creates a complete "movie" of the entire task in its head. This storyboard is called an IVLR-Trace.

This trace is special because it mixes two things together, alternating like a comic book:

Text Panels: "Pick up the white mug." (The logic/causal order).
Image Panels: A picture of the white mug sitting exactly where it needs to go. (The visual goal).

The robot generates this entire storyboard once at the very beginning. It's like a director saying, "Okay, here is the whole movie. Scene 1: Pick up mug. Scene 2: Place on plate. Scene 3: Pick up yellow mug..."

3. How the Robot Uses It: The "GPS with a Map"

Once the storyboard is made, the robot doesn't just blindly follow the images like a video game character. Instead, it keeps the storyboard "cached" (saved in its short-term memory) while it works.

The Loop: At every single moment, the robot looks at the real world (what it sees right now) and checks it against the storyboard (what it planned to see).
The Analogy: Imagine you are driving to a party. You have a map (the storyboard) that shows the route and the destination. But you also have eyes (the live camera). If you take a wrong turn, your eyes tell you, "Hey, that's not the street on the map!" You then correct your steering.
The robot does this constantly. It uses the storyboard to remember the order of events and the visual goal, but it uses its live eyes to make sure it doesn't crash into a chair that wasn't there when it made the plan.

4. How They Taught the Robot (The "Pseudo-Supervision")

Real robots don't come with storyboards; they just come with videos of people doing tasks. The authors had to teach the robot how to make its own storyboards.

They took videos of robots doing tasks and used a smart AI to chop the video into "scenes" (stages).
Then, they used another AI to write a caption for each scene (e.g., "The gripper moves to the mug") and pick a "keyframe" (a representative photo of that moment).
They fed these made-up storyboards to the robot as training data. It's like giving a student a textbook with answers filled in, so they can learn how to solve the problems themselves later.

5. The Results: Why It Matters

The paper tested this on computer simulations where robots had to do long, multi-step tasks (like stacking blocks or moving mugs).

Without the storyboard: The robot failed often, especially on long tasks (only about 37% success). It got lost in the middle of the story.
With just text or just images: It did better, but not great (around 60-68%).
With the full "Storyboard" (Text + Images): The robot succeeded 92.4% of the time on the hardest tasks.

The key finding is that you need both. You need the text to remember the order (causality) and the images to remember the location (geometry). One without the other isn't enough.

6. The Catch (Limitations)

The paper is honest about the downsides:

Thinking Time: The robot has to "think" for about 10 seconds before it starts moving to create the storyboard. This is fine for a slow, careful task, but it's too slow for a robot that needs to dodge a fast-moving ball.
Stale Plans: If the world changes after the robot makes the plan (e.g., someone moves the cup while the robot is thinking), the storyboard becomes wrong. The robot might try to follow a plan for a world that no longer exists.
Simulation Only: They tested this in a computer simulation, not on a real robot in a real kitchen.

Summary

This paper proposes that instead of forcing a robot to guess the next step based only on what it sees right now, we should let it write a "comic book" of the whole task first. By mixing words (the plan) and pictures (the goal) into a single "trace," the robot can remember the big picture while still reacting to the immediate world. It's a shift from "reacting" to "planning and reacting."

1. Problem Statement

Long-horizon robotic manipulation requires policies that satisfy two conflicting requirements: causal coherence (maintaining the logical order of subgoals) and geometric grounding (precise spatial positioning).

Limitations of Current Methods:
- Standard VLA (Vision-Language-Action) Policies: Typically hide planning in latent states. They often suffer from "temporal myopia," failing to maintain state over long sequences or executing steps out of order.
- Text-Only Chain-of-Thought (CoT): Encodes causal order well but lacks the specific spatial pose, contact geometry, and target placement details required for low-level control.
- Visual Prediction/World Models: Provide geometric cues but often remain local and semantically under-constrained, leading to plausible but task-incorrect visual futures.
The Gap: There is a lack of an explicit intermediate representation that simultaneously captures the semantic logic (text) and the geometric state (vision) across the entire task horizon.

2. Methodology: Interleaved Vision–Language Reasoning (IVLR)

The authors propose IVLR, a framework centered on the IVLR-Trace, an explicit intermediate representation that alternates textual subgoals with visual keyframes.

A. Core Representation: IVLR-Trace

The trace is a sequence of stages $z = [(c^{(1)}, k^{(1)}), \dots, (c^{(N)}, k^{(N)})]$ , where:

$c^{(i)}$ : A textual subgoal (caption) describing the causal role of the stage.
$k^{(i)}$ : A visual keyframe (RGB image) anchoring the intended spatial state for that stage.
This creates a "full-horizon storyboard" that makes the robot's reasoning visible and ablatable.

B. Architecture

The system utilizes a single native multimodal transformer (initialized from Show-o2 1.5B) with two distinct phases:

Reasoning Phase (t=0):
- Input: Initial observation ( $o_0$ ) and task instruction ( $I$ ).
- Process: The model generates the complete IVLR-Trace.
- Mechanism: Text is decoded autoregressively via a language head; visual keyframes are generated via a flow-matching head (predicting vector fields to transport noise to target latents).
Execution Phase (t>0):
- Input: Current observation ( $o_t$ ), original instruction ( $I$ ), and the cached trace ( $z$ ).
- Process: A closed-loop action decoder predicts continuous actions ( $a_t$ ).
- Mechanism: The action token attends to the full cached trace and the live observation, allowing the model to dynamically select relevant parts of the plan based on the current state without a hard-coded stage pointer.

C. Pseudo-Supervision Pipeline

Since standard robot datasets lack interleaved traces, the authors constructed a pipeline to generate pseudo-traces for training:

Segmentation: Use the Universal Visual Decomposer (UVD) to split demonstration trajectories into stages based on monotonicity breaks in visual embeddings.
Keyframe Selection: The final frame of each segment becomes the visual keyframe ( $k^{(i)}$ ).
Captioning: A Vision-Language Model (Qwen3-VL) generates a structured text caption ( $c^{(i)}$ ) describing the gripper-object interaction and spatial relations for each stage.

D. Training Objective

The total loss combines three components:

$L_{text}$ : Next-token cross-entropy for captions.
$L_{flow}$ : Flow-matching loss for visual keyframes.
$L_{action}$ : L1 loss for continuous action prediction.
Robustness: Random trace noise and dynamic context masking are applied during training to prevent over-reliance on perfect traces.

3. Key Contributions

Explicit IVLR-Trace: Formulation of a full-horizon, interleaved sequence of text and RGB keyframes that serves as a semantic-geometric interface for VLA control.
Unified Architecture: Implementation in a single native multimodal transformer that generates the plan once and conditions closed-loop execution on the cached trace.
Scalable Pseudo-Trace Construction: A pipeline using UVD and VLMs to retrofit unstructured robot demonstrations with staged multimodal supervision.
Empirical Validation: Demonstration that interleaved traces significantly outperform unimodal baselines in long-horizon tasks and are robust to moderate execution drift and trace corruption.

4. Experimental Results

The method was evaluated on LIBERO (simulated manipulation) and SimplerEnv-WidowX (visual distribution shift).

Main Performance (LIBERO):
- IVLR: Achieved 95.5% average success.
- LIBERO-Long (Long-horizon): Achieved 92.4%, significantly outperforming CoT-VLA (69.0%) and VLA-0 (87.6%).
- SimplerEnv-WidowX: Achieved 59.4% overall success, outperforming SpatialVLA (42.7%).
Ablation Studies (The Necessity of Interleaving):
- No Trace: Success drops to 37.7% on LIBERO-Long.
- Text-Only Trace: 62.0% (Good for order, lacks spatial anchors).
- Vision-Only Trace: 68.4% (Good for geometry, lacks semantic sequence).
- Full IVLR: 92.4% (Proves text and vision are complementary, not interchangeable).
Robustness Stress Tests:
- Execution Perturbation: With a random 2cm end-effector displacement, success dropped only moderately (95.5% $\to$ 92.1%), showing the closed-loop nature can recover from small slips.
- Trace Corruption: Masking 30% of text or visual tokens resulted in moderate degradation (to ~90-92%), indicating the system tolerates local missing content.

5. Significance and Limitations

Significance:

Explicit Reasoning: Shifts the paradigm from implicit latent planning to explicit, inspectable, and controllable multimodal reasoning.
Scalability: Demonstrates that pseudo-supervised traces can effectively train policies for complex, multi-stage tasks without human-labeled symbolic plans.
Complementarity: Validates that combining semantic logic (text) with geometric grounding (vision) is critical for long-horizon success.

Limitations:

Latency: Generating the full trace takes ~10 seconds (on an H20 GPU) before execution begins, which is a bottleneck for dynamic environments.
Static Assumption: The method assumes a fully observed, static workspace at $t=0$ . If the scene changes drastically (e.g., object removal, occlusion) after planning, the cached trace becomes stale.
Simulation Only: Results are currently limited to simulated benchmarks; real-robot deployment requires further validation.

Future Directions:
The authors suggest exploring asynchronous plan-while-acting (generating future trace stages while executing current ones) and self-triggered replanning to handle dynamic scene changes.