💻 computer science

AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement

The paper introduces AnySlot, a framework that achieves zero-shot, sub-centimeter precision in slot-level robotic placement by decoupling language grounding from control through an explicit visual goal representation, accompanied by the new SlotBench benchmark for evaluating such structured spatial reasoning tasks.

Original authors: Zhaofeng Hu, Sifan Zhou, Qinbo Zhang, Rongtao Xu, Qi Su, Ci-Jyun Liang

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaofeng Hu, Sifan Zhou, Qinbo Zhang, Rongtao Xu, Qi Su, Ci-Jyun Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a very specific game: "Put this red block into the third slot from the left, but only if it's the biggest one, and make sure it doesn't touch the blue box."

For a human, this is easy. You look, you think, you point, and you move your hand. For a robot, this is a nightmare.

This paper introduces a new system called AnySlot that solves this problem by changing how the robot thinks. Instead of trying to do everything in one giant brain-melt, it breaks the job into two distinct steps: The Architect and The Builder.

Here is the breakdown using simple analogies:

1. The Problem: The "Confused Chef"

Current robots (called "Flat VLA policies") are like a chef who tries to read a recipe, chop the vegetables, stir the pot, and plate the food all at the exact same time.

The Issue: If the recipe says "put the soup in the second bowl from the left," the robot gets confused. It tries to guess the coordinates while simultaneously moving its arm.
The Result: It often picks the wrong bowl or drops the soup because it's trying to do too much math in its head at once. It's like trying to solve a complex math equation while riding a unicycle.

2. The Solution: The "Architect and Builder" (AnySlot)

The authors realized that to be precise, you need to separate planning from doing. They created a two-stage system:

Stage 1: The Architect (High-Level Goal)

Instead of asking the robot to "calculate the X and Y coordinates," they ask an Image Generator (like a digital artist) to draw a picture of the goal.

The Analogy: Imagine you are giving directions to a friend. Instead of saying, "Walk 42 steps north, then turn 15 degrees," you simply draw a big blue dot on a map right where they need to go.
How it works: The robot reads the sentence ("Put the block in the second slot") and asks an AI artist to generate an image where a blue sphere is drawn exactly on top of that specific slot.
The Magic: This blue sphere becomes a "visual target." It turns a confusing language puzzle into a simple visual game: "Go to the blue dot."

Stage 2: The Builder (Low-Level Policy)

Now, the robot's arm doesn't have to think about which slot is the "second largest." It just has to follow the blue dot.

The Analogy: The robot is now a delivery driver who just needs to drive to the house with the blue flag on the roof. It doesn't need to know the street name or the zip code; it just follows the flag.
The Result: Because the robot only has to focus on "move arm to blue dot," it can be incredibly precise (sub-centimeter accuracy). It ignores the complex language and focuses purely on the visual target.

3. The New Test: "SlotBench"

The authors realized there was no good way to test if robots could actually do this kind of precise, logic-heavy work. So, they built a video game called SlotBench.

Think of it as a gym for robot brains. It has 9 different levels of difficulty, ranging from simple ("pick the top slot") to very tricky ("pick the slot that is furthest from the red block but closest to the Pepsi can").
They tested old robots on this gym, and most failed miserably. They couldn't handle the logic.
AnySlot walked into the gym and aced almost every level (90% success rate) because it used the "Blue Dot" trick to simplify the task.

4. Why This Matters

Precision: Old robots were like a person trying to thread a needle while wearing boxing gloves. AnySlot is like a surgeon with steady hands.
Zero-Shot Learning: This means the robot can learn to do these tasks without being specifically trained on every single possible arrangement of blocks. If you give it a new puzzle it has never seen before, it can still figure out where the "blue dot" should go and place the object perfectly.

Summary

AnySlot is a framework that stops robots from trying to be poets and mathematicians at the same time.

Translate: Turn complex language instructions into a simple visual picture (a blue dot on the target).
Execute: Let the robot's arm simply follow that picture.

By turning "thinking" into "drawing," the robot can finally place objects with the precision needed for real-world factories and assembly lines.

1. Problem Definition

The paper addresses the challenge of slot-level robotic manipulation, a task requiring a robot to place an object into a specific, discrete "slot" within a dense environment based on complex, compositional natural language instructions (e.g., "place the red block in the second largest slot" or "avoid the lower-left cells").

Key Challenges:

Precision vs. Semantics: Unlike coarse pick-and-place tasks, slot-level placement demands sub-centimeter geometric accuracy combined with high-level semantic reasoning (ordinality, size comparison, logical negation).
Limitations of Flat VLA Policies: End-to-end Vision-Language-Action (VLA) models struggle because they tightly couple language grounding with low-level control. They often lack the capacity for reliable compositional spatial reasoning and fail to maintain geometric precision under unseen layouts.
Limitations of Modular Approaches: Existing modular pipelines (VLM for grounding + Controller for execution) often rely on VLMs to predict discrete coordinates or keypoints. These predictions are typically coarse, token-based, and lack the pixel-level spatial context (e.g., slot boundaries) required for stable, precise alignment.

2. Methodology: AnySlot Framework

The authors propose AnySlot, a two-stage, goal-conditioned framework that decouples high-level reasoning from low-level execution by introducing an explicit spatial visual goal as an intermediate representation.

Stage 1: High-Level Goal Construction (Grounding)

Instead of outputting coordinates, the system uses an image generation model (e.g., Nano-Banana) to render a visual marker directly onto the robot's observation image.

Process:
1. The robot captures a head-view RGB-D image ( $I_h^0$ ) and receives a language instruction ( $\ell$ ).
2. An image generator renders a blue sphere marker at the target slot location within the image based on the instruction.
3. 3D Lifting: The 2D center of the sphere is detected, and its depth is retrieved from the depth map. Using camera intrinsics and extrinsics, this point is back-projected to a 3D world-space anchor ( $P^W$ ).
4. Multi-view Projection: The 3D anchor is projected back into all camera streams (head and wrist cameras) to create view-consistent overlays.
Output: A spatial visual goal $G_t$ (the blue sphere overlay) that serves as the target for the low-level policy.

Stage 2: Low-Level Goal-Conditioned Policy (Execution)

A continuous generative VLA policy ( $\pi_{0.5}$ ) executes the placement.

Input: Multi-view observations augmented with the rendered goal overlays ( $G_t$ ) and a fixed low-level instruction (e.g., "move the object into the slot with blue sphere guidance").
Mechanism: The policy is trained to track the visual marker and perform the pick-and-place action. By fixing the language instruction, the policy is relieved of the burden of spatial reasoning, focusing solely on precise visuomotor control relative to the visual goal.
Training: The low-level policy is trained on a synthetic dataset where the "oracle" goal is provided, teaching the robot to track colored markers with sub-centimeter precision.

3. Key Contributions

AnySlot Framework: A novel architecture that bridges the gap between language and control by converting compositional language into explicit, spatially coherent visual goals via image generation. This avoids the precision limitations of coordinate regression from VLMs.
SlotBench Benchmark: The introduction of a comprehensive simulation benchmark featuring nine distinct task categories designed to test structured spatial reasoning:
- Ordinal reasoning, Size/Height comparison, Distance reasoning, Compositional relations, Logical negation, Vague language, Affordance reasoning, and World knowledge.
- Unlike existing benchmarks, SlotBench specifically evaluates zero-shot generalization to unseen slot layouts and complex instructions.
Empirical Validation: Extensive experiments demonstrating that AnySlot significantly outperforms both flat VLA baselines and modular VLM-based methods in zero-shot settings.

4. Experimental Results

The evaluation was conducted on SlotBench using a zero-shot protocol (no task-specific fine-tuning on the test tasks).

Performance Comparison:
- Flat VLA Policies: Achieved near-zero success rates (0–18%) across most categories, failing to generalize beyond simple ordinal tasks.
- Modular VLM Baselines (e.g., AnyPlace): Showed partial success but struggled significantly with precision (Instruction Accuracy < 50% in many cases) due to coarse coordinate grounding.
- AnySlot: Achieved an average Success Rate (SR) of ~90% and Instruction Accuracy (IA) of ~94% across all nine categories. It was the only method to reliably handle complex reasoning like logical negation and vague language.
Ablation Studies:
- Goal Construction: Replacing the image generator with VLMs for coordinate prediction resulted in a drastic drop in precision, confirming that visual markers are superior to token-based coordinates for this task.
- Low-Level Policy: Fully fine-tuned policies ( $\pi_{0.5}$ ) significantly outperformed parameter-efficient tuning (PEFT) or diffusion policies, highlighting the need for robust low-level adaptation.
- Synthetic Data: Training the low-level policy with visual goal augmentation (synthetic dataset) was critical; training without it resulted in near-zero success.

5. Significance and Impact

Paradigm Shift: The paper challenges the "flat" end-to-end approach for precision tasks, proposing that explicit visual intermediate representations are necessary to decouple semantic understanding from geometric execution.
Robustness: By using image generation to create spatial goals, the system achieves robustness to small grounding errors and maintains geometric consistency across multiple camera views.
Benchmarking: SlotBench fills a critical gap in the literature by providing a standardized testbed for structured spatial reasoning in robotics, moving beyond simple object manipulation to complex, constraint-based placement.
Real-World Feasibility: The authors demonstrated the pipeline on a real UR10e robot, showing that the 3D reconstruction and multi-view projection of the visual goal align accurately with physical targets, validating the method's applicability beyond simulation.

In summary, AnySlot demonstrates that for high-precision, language-guided robotic tasks, transforming abstract language instructions into concrete, visual spatial goals is a more effective strategy than direct coordinate prediction or end-to-end mapping.