💻 computer science

Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents

This paper proposes a closed-loop VLM agent framework that leverages iterative reasoning and three specific inference-time techniques to significantly improve text-guided 6D object pose rearrangement and robot manipulation performance without requiring additional fine-tuning.

Original authors: Sangwon Baik, Gunhee Kim, Mingi Choi, Hanbyul Joo

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sangwon Baik, Gunhee Kim, Mingi Choi, Hanbyul Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot how to rearrange a messy table. You give it a simple instruction: "Put the marker in the mug with the cap facing up."

If you ask a standard AI to do this, it might look at a single photo of the table, guess where the marker should go, and then tell the robot to move it there. But because the AI only saw one picture, it might get the depth wrong (thinking the mug is closer than it is) or the orientation wrong (putting the cap facing down). The robot tries, fails, and the marker ends up on the floor.

This paper introduces a new way to solve this problem. Instead of a "one-shot guess," the authors created a closed-loop agent that acts like a perfectionist art director working with a 3D printer.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Shot" Blind Spot

Standard AI models are like people who try to assemble IKEA furniture by looking at only one page of the manual and guessing where the screws go. They are great at understanding language ("Put the cap up!"), but they struggle with 3D space. They can't easily tell if an object is behind another or if it's rotated the right way just by looking at a flat image.

2. The Solution: The "Iterative Art Director"

The authors' method turns the AI into an active agent that doesn't just guess once. It plays a game of "Hot and Cold" with a 3D simulator.

The Setup: You give the AI a 3D scene (a digital table) and a text instruction.
The Loop:
1. The Judge (Evaluator): The AI looks at the current scene from multiple angles (like walking around the table). It asks, "Does this look like the instruction? Is the marker in the mug? Is the cap up?"
2. The Critic: If the answer is "No," the AI picks the specific camera angle that shows the problem best. Maybe the front view hides the fact that the cap is upside down.
3. The Fixer (Proposer): The AI then says, "Okay, I see the problem. I need to rotate the marker 90 degrees around the Z-axis and move it 2 inches to the left."
4. The Update: The computer simulates this move, renders a new image, and the loop starts again.

It keeps doing this—Judge, Critic, Fixer, Update—until the scene perfectly matches the text instruction.

3. The Three Secret Weapons

To make this "Art Director" actually good at 3D, the authors gave it three special tools (inference-time techniques):

Tool 1: The "Walk-Around" Strategy (Multi-View Reasoning)
- Analogy: Imagine trying to find a lost earring on a carpet. If you only look from above, you might miss it. But if you walk around the room and look from the side, you see it clearly.
- How it helps: The AI doesn't just look at one picture. It generates views from all sides to find the "supporting view" that proves the object is in the right place or reveals exactly what's wrong.
Tool 2: The "Glow-in-the-Dark" Compass (Object-Centered Coordinate System)
- Analogy: Imagine trying to tell someone to "turn left" while they are wearing a blindfold and spinning around. They will get confused. But if you tape a glowing arrow to their head pointing "Forward," and another pointing "Right," they can instantly understand directions.
- How it helps: The AI draws red, green, and blue arrows directly onto the 3D object. This stops the AI from getting confused about what "left" or "up" means when the camera angle changes. It gives the AI a physical sense of direction.
Tool 3: The "One-Spin" Rule (Single-Axis Rotation)
- Analogy: Trying to untangle a knot by twisting it in every direction at once is impossible. But if you say, "Just twist it left, then stop," it becomes easy.
- How it helps: Instead of asking the AI to calculate a complex 3D spin all at once, the AI is told to fix one axis at a time (e.g., "Just rotate it around the vertical axis"). It does this step-by-step until the object is perfectly oriented.

4. The Result: A Robot That Actually Works

The authors tested this on real-world robot simulations.

Old Way: A robot using standard AI would try to put a hammer in a mug, miss the handle, or drop the hammer because it couldn't "see" the 3D depth.
New Way: The robot using this "Closed-Loop Agent" successfully placed the hammer, the marker, and the teapot exactly as described, even for tricky tasks like "pour tea" or "stack blocks."

Summary

Think of this paper as teaching an AI to stop guessing and start checking. Instead of making a blind leap of faith, the AI acts like a careful human: it looks at the object from different angles, uses a visual compass to understand directions, and makes small, one-step adjustments until the job is done perfectly.

This allows robots to follow complex instructions like "Put the book on the second shelf, facing the right way" with a level of precision that was previously impossible without expensive retraining.

1. Problem Statement

The paper addresses the challenge of Text-Guided 6D Object Pose Rearrangement. Given a 3D scene (represented by RGB-D images or 3D meshes) and a natural language instruction (e.g., "Place the marker in the mug with the cap facing up"), the goal is to infer the precise 6D pose (3D position + 3D orientation) of a target object such that the scene satisfies the instruction.

Key Challenges:

Spatial Reasoning Gap: While Vision-Language Models (VLMs) excel at describing images, they struggle with 3D spatial reasoning, particularly inferring precise goal poses from text.
Depth Ambiguity & Occlusion: Single-view, feed-forward prediction often fails due to depth ambiguities and occlusions, leading to physically invalid configurations.
Complex Rotations: Inferring full 3D rotations (SO(3)) from a single image is difficult for current VLMs, especially when the goal state requires specific orientations (e.g., "cap facing down").

2. Methodology

The authors propose a training-free, closed-loop framework where a pre-trained VLM acts as an active agent. Instead of a one-shot prediction, the system iteratively refines the object's pose through a loop of evaluation and proposal.

Core Framework: The Closed-Loop Agent

The process involves two roles played by the same VLM:

Evaluator: Observes the current multi-view rendered scene and judges whether it is "faithful" to the text instruction. If not, it identifies the most informative view (supporting view) that reveals the error.
Proposer: Takes the supporting view (augmented with specific visual aids) and predicts an incremental 6D pose update (translation and rotation) to move the object closer to the goal.

The loop repeats: Render Scene $\rightarrow$ Evaluate $\rightarrow$ Propose Update $\rightarrow$ Apply Update $\rightarrow$ Render New Scene until the evaluator confirms faithfulness or a maximum iteration limit is reached.

Key Inference-Time Techniques

To stabilize the closed-loop process and improve geometric grounding, three specific techniques are introduced:

Multi-View Reasoning with Supporting View Selection:
- Instead of relying on a single image, the system renders $K$ views from a circular trajectory around the object.
- The Evaluator selects the "supporting view" that best explains why the current state is incorrect (e.g., a view that reveals occlusion or depth errors). This mitigates single-view ambiguity.
Object-Centered Coordinate System Visualization:
- The system renders explicit 3D axes (Red=X, Green=Y, Blue=Z) overlaid on the target object.
- Purpose: This provides visual cues for direction (left/right/forward/back) and scale. It helps the VLM distinguish between clockwise/counterclockwise rotations and resolves directional confusion caused by different camera viewpoints.
- The axes are normalized (unit length), allowing the VLM to infer translation relative to the object's size rather than absolute world scale.
Single-Axis Rotation Prediction:
- Instead of asking the VLM to predict a complex 3D rotation matrix or Euler angles simultaneously, the Proposer predicts a rotation around a single dominant axis ( $x, y,$ or $z$ ) and an angle.
- Purpose: This simplifies the SO(3) search space, making the reasoning task tractable for current VLMs. Iterative application of single-axis rotations allows the system to converge on complex orientations over multiple steps.

Context Memory

The system maintains a context memory containing previous evaluator judgments, predicted updates, and rationales. This prevents oscillating predictions and helps the agent learn from past errors during the iterative process.

3. Key Contributions

Closed-Loop Agentic Framework: A novel, training-free approach that transforms VLMs into iterative agents for 6D pose rearrangement, moving beyond passive one-shot prediction.
Inference-Time Geometric Grounding: The introduction of multi-view reasoning, coordinate visualization, and single-axis rotation prediction significantly enhances the 3D spatial reasoning capabilities of pre-trained VLMs without fine-tuning.
State-of-the-Art Performance: The method achieves superior results on standard benchmarks compared to existing VLM-based and structured methods.
Zero-Shot Robotic Manipulation: The predicted poses are directly usable by standard robot motion planners, demonstrating high success rates in simulated robotic manipulation tasks without task-specific training.

4. Experimental Results

The method was evaluated on two benchmarks: Open6DOR V2 (6D pose rearrangement) and SIMPLER (robot manipulation).

Open6DOR V2 (Pose Prediction):
- Metrics: Success rate on Position, Rotation, and 6-DoF tracks.
- Results: The proposed method outperformed baselines (SoFar, Open6DOR-GPT) across all tracks.
  - Rotation Track: Significant improvement (e.g., 56.6% vs. 31.3% for SoFar on Level 2), demonstrating the effectiveness of the closed-loop and single-axis rotation techniques.
  - 6-DoF Track: Achieved 46.4% overall success (vs. 26.3% for SoFar), showing the ability to handle both position and orientation simultaneously.
- Generalization: The method worked consistently across different VLM backbones (GPT-5.2, Gemini-2.5, InternVL3).
Robot Manipulation (Open6DOR V2 & SIMPLER):
- Setup: Coupled with GraspNet and OMPL for a Franka arm (Open6DOR) and WidowX (SIMPLER).
- Results: The method achieved significantly higher task success rates than end-to-end Vision-Language-Action (VLA) models (OpenVLA, SpatialVLA) and the SoFar baseline.
- Qualitative: The system successfully executed complex tasks like "Pour tea" and "Place hammer handle upright," where baselines often failed due to incorrect orientation predictions.
Ablation Studies:
- Removing Coordinate Visualization caused the largest drop in performance, highlighting its critical role in directional reasoning.
- Context Memory was essential for stable convergence.
- Multi-view reasoning was less critical for simple tabletop tasks but vital for handling occlusions.

5. Significance and Limitations

Significance:

Bridging the Gap: The paper demonstrates that VLMs can be effectively leveraged for complex 3D spatial tasks without expensive fine-tuning, provided they are placed in a closed-loop visual feedback system.
Practical Robotics: It offers a viable path for zero-shot robotic manipulation where precise 6D pose estimation is required but training data is scarce.
Interpretability: The iterative nature and explicit coordinate visualization make the reasoning process more transparent than black-box VLA policies.

Limitations:

Inference Speed: The closed-loop nature requires multiple VLM queries and rendering steps per task, resulting in higher latency compared to one-shot models. This makes it less suitable for real-time, low-latency control.
Model Dependency: Performance is inherently bounded by the visual understanding capabilities of the underlying VLM.
Complexity: The system relies on a 3D renderer and segmentation models, adding computational overhead compared to purely image-based approaches.

In conclusion, this work establishes that iterative refinement via visual feedback is a powerful paradigm for overcoming the spatial reasoning limitations of current VLMs in 3D object manipulation tasks.