💻 computer science

Geometry Conditioning in an Embodied SLM: Training Controls and Robustness Diagnostics in a 0.8B Hybrid Model

This paper investigates the impact of geometric conditioning on a 0.8B hybrid embodied language model, revealing that training-time alignment of physical-state inputs yields no reliable advantage over shuffled or absent geometry and highlighting a significant gap between coordinate invariance and physical-layout generalization in visual policies.

Original authors: Hao Li, Haofei Sun, Lin He

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Li, Haofei Sun, Lin He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of robotics, teaching a machine to move its arm to pick up a cup is often a matter of showing it thousands of examples. The robot watches a human perform the task, then tries to copy the motion. For years, scientists have wondered if giving the robot a better understanding of the physical world—specifically, the exact distances and angles between objects—would make it smarter. Imagine a robot that doesn't just see a picture of a cup and a table, but also understands the precise mathematical relationship between them. The hope is that this extra layer of geometric knowledge would help the robot adapt when the cup is moved slightly or when the camera angle changes. This question sits at the heart of a new study involving a small, specialized computer brain designed to control robot arms. The researchers wanted to know if feeding this brain explicit instructions about physical space actually helps it learn, or if the robot can figure it out on its own just by watching.

The study focused on a tiny artificial intelligence model, containing only 0.8 billion parameters, which is small by modern standards but large enough to be a functional robot controller. The team trained this model on a set of tasks involving moving objects in a simulated environment, using a total of 6.2 million adjustable settings to teach it how to act. They tested two main ways of giving the robot geometric information. In the first method, they fed the data directly into the robot's decision-making "gates," which act like switches that control how the robot processes information over time. In the second method, they fed the same geometric data into the robot's input stream, treating it like a word in a sentence. To ensure a fair test, they also trained a version of the robot where the geometric data was scrambled during the learning phase, so the robot saw the numbers but they didn't match the actual movements. Finally, they tested a version that used a simple clock signal instead of geometry to see if the robot was just learning to follow a timer.

The results were surprising and challenged the common assumption that more geometric detail leads to better performance. When the robot was trained with the correct, unscrambled geometric data, it succeeded in about 29 percent of its attempts. However, the version trained with scrambled, meaningless geometric data actually performed slightly better, succeeding in nearly 37 percent of attempts. The version that received no geometric data at all succeeded about 24 percent of the time. This suggests that, under the specific conditions of this experiment, providing the robot with precise, correct geometric instructions did not offer a clear advantage over giving it random or scrambled numbers. The researchers found that the robot did not rely on the correct alignment of these numbers to succeed; in fact, the scrambled version sometimes outperformed the correct one. This indicates that the robot might be learning to solve the tasks through other cues, such as the visual patterns of the camera or the sequence of actions, rather than by calculating the physical distances between objects.

The study also tested how well these robots could handle changes in the environment, a crucial test for any real-world application. When the researchers rotated the entire coordinate system—essentially telling the robot that "up" was now "left"—a robot that relied only on absolute positions failed completely. However, a robot trained to understand relative positions, meaning it focused on where the object was in relation to the robot's hand rather than a fixed map, managed to succeed in seven out of ten attempts. This showed that understanding relative relationships is vital for flexibility. Yet, when the researchers physically moved the object by just five centimeters on the table, all the visual policies, including those with geometric training, collapsed. Their success rate dropped to near zero. This revealed a significant gap: while the robots could handle a change in perspective, they could not handle a small physical shift in the object's location. The geometric training did not protect them from this failure.

Ultimately, the paper concludes that simply adding physical state information to a small robot brain does not guarantee better performance or robustness. The researchers found no reliable evidence that training-time geometric alignment helps the robot generalize to new situations. The slight variations in success rates across different training runs suggested that the results were sensitive to chance and specific task details rather than a fundamental improvement from the geometric data. The study serves as a careful check on the field, showing that while robots can learn to be flexible with perspective, they remain fragile when the physical layout of their world changes slightly. The findings suggest that the path to truly robust robot manipulation may require more than just feeding the system better maps of the physical world; it may require a deeper change in how these systems learn to interact with reality.

Technical Summary: Geometry Conditioning in an Embodied SLM

Problem Statement
The paper investigates how physical-state inputs, specifically geometric information regarding object and goal positions, affect the performance of a small language model (SLM) adapted for robotic manipulation. While recent work has shown that hybrid language models (combining attention with gated recurrence) can be adapted into robot policies with only a few million trainable parameters, it remains unclear whether injecting geometric signals through input tokens or recurrent decay gates improves learning. Furthermore, it is uncertain whether apparent performance gains rely on the geometric signal being temporally and spatially aligned with the demonstrations during training, or if the model can learn robustness through other means.

Methodology
The authors conduct experiments using a 0.8B hybrid language model (Qwen3.5-0.8B) adapted for the LIBERO-Spatial manipulation benchmark. The base model weights are frozen; adaptation is achieved via Low-Rank Adaptation (LoRA) on attention projections, a linear input projection, and an action head. The total number of trainable parameters is tightly matched across all conditions (approximately 6.2M).

The study compares several conditioning routes for injecting geometric data:

Token-based Conditioning:
- c1: A token-MLP adapter receives 6-dimensional invariant geometric increments ( $g_t$ ) and projects them to 2 dimensions before the input projection.
- c2: A token adapter receives richer 14-dimensional relative state features.
- c0 (Baseline): No explicit object/goal geometry is provided.
Decay-Gate Conditioning:
- f0: The 6-dimensional invariant increments ( $g_t$ ) are added as a bias to the decay logits of the 18 Gated Delta Network (GDN) layers via a learnable scalar and tanh-squashed transformation.
- f0-shuffled: The same $g_t$ is used, but the temporal alignment is shuffled (permuted) during training to test the necessity of correct temporal alignment.
- b1 (Clock Control): Instead of geometry, six normalized "token clock" channels are fed into the decay gates to control for the presence of any temporal signal.

Experimental Protocol

Tasks: Three tasks from LIBERO-Spatial.
Training: Joint training on 35 demonstrations per task using behavior cloning with action chunking ( $K=8$ ).
Evaluation: 540 held-out rollouts (3 seeds $\times$ 3 tasks $\times$ 10 initial states).
Robustness Tests:
- Frame Relabeling: Rotating the coordinate frame of the input poses without moving the physical scene.
- Physical Displacement: Shifting the target object by 5 cm and 10 cm in the table plane.

Key Results

Training-Time Alignment: Correct temporal alignment of geometry (f0) did not outperform shuffled training (f0-shuffled).
- f0 (Correct): 28.9% success (26/90).
- f0-shuffled: 36.7% success (33/90).
- c0 (No Geometry): 24.4% success (22/90).
- The difference between correct and shuffled geometry was not statistically significant ( $p=0.28$ ), suggesting that correct alignment provides no clear benefit in this setup.
Token vs. Gate Pathways:
- The token adapter c1 (27.8% success) performed similarly to the gate adapter f0 (28.9%), with inconclusive differences across seeds.
- The richer token adapter c2 (34.4% success) showed a trend toward improvement over the baseline but was not statistically significant ( $p=0.093$ ) given the sample size.
Clock Conditioning: The clock condition (b1) performed poorly (11.1% success) and included one seed that failed to converge, indicating that the decay gate interface is sensitive to input semantics and optimization stability.
Robustness to Coordinate Changes:
- Frame Relabeling: A state-only policy using relative coordinates maintained 7/10 success under 45° and 90° rotations, whereas absolute-coordinate policies failed completely. This confirms relative coordinates are invariant to rigid transforms.
- Physical Displacement: All tested visual policies (including those with geometry conditioning) collapsed under physical object displacement. Success rates dropped to at most 3/20 for a 5 cm shift and 0/20 for a 10 cm shift. This demonstrates that coordinate invariance does not equate to generalization to new physical layouts.

Significance and Claims
The paper claims to provide controls for interpreting the apparent benefits of geometry conditioning. The primary finding is that under the specific adaptation recipe used (frozen SLM, LoRA, small parameter budget), training-time geometric alignment does not reliably improve performance over shuffled training.

The study highlights a critical gap between coordinate invariance (handling rotated frames) and physical-layout generalization (handling displaced objects). While relative coordinate features successfully handled frame relabeling, no policy, regardless of geometry conditioning, could generalize to physically displaced objects. The authors conclude that the study identifies the limits of geometry conditioning under small adaptation budgets and illustrates that observed gains in similar works may depend heavily on the alignment of geometric signals with demonstrations rather than an inherent ability to learn physical causality.

Limitations
The authors note that the baseline performance (24.4%) is relatively low, leaving open the possibility that a stronger training recipe might reveal geometry benefits. Additionally, the matched-input comparison (c1 vs. f0) is limited by a 2D bottleneck in the token adapter, and the robustness tests were conducted on specific seeds and tasks, limiting the generalizability of the failure modes observed.

Technical Summary: Geometry Conditioning in an Embodied SLM

More like this