💻 computer science

Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation

This paper introduces RGMP-S, a novel framework that combines a Long-horizon Geometric Prior Skill Selector for robust 3D scene understanding and a Recursive Adaptive Spiking Network for data-efficient motion synthesis, enabling humanoid robots to generalize complex manipulation tasks across diverse simulated and real-world environments.

Original authors: Xuetao Li, Wenke Huang, Mang Ye, Jifeng Xuan, Bo Du, Sheng Liu, Miao Li

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuetao Li, Wenke Huang, Mang Ye, Jifeng Xuan, Bo Du, Sheng Liu, Miao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to act like a human waiter. You want it to hear a request like, "Can I have a Coke?" and then successfully grab the can and hand it to you.

The problem is that robots are often terrible at this. If you ask a standard robot for a Coke, it might grab a normal can perfectly. But if you hand it a crushed, flattened Coke can, the robot gets confused. It sees the word "Coke" and tries to grab it the same way it grabs a round can (squeezing the sides). But because the can is flat, that grip fails, and the robot drops it.

This paper introduces a new system called RGMP-S to fix exactly this kind of problem. The authors built a robot brain that combines two special skills: Geometric Common Sense and Memory-Enhanced Learning.

Here is how it works, using simple analogies:

1. The "Geometric Common Sense" Brain (LGSS)

The Problem: Standard robot brains (called Vision-Language Models) are great at reading signs and recognizing objects, but they are bad at understanding physics. They know what a "Coke can" is, but they don't intuitively understand that a crushed can needs to be pinched, not squeezed.

The Solution: The authors added a "Geometric Prior" module. Think of this as giving the robot a set of common-sense rules about shapes.

The Analogy: Imagine a human waiter. If they see a round cup, they grab the sides. If they see a flat napkin, they pinch it. They don't need to be taught this every time; they just "know" it based on the shape.
How it works: The robot's system looks at the object, figures out its shape (e.g., "cylinder" vs. "squashed"), and uses a rulebook to decide the best way to grab it. It stops the robot from trying to squeeze a flat object. This allows the robot to handle new, weird shapes it has never seen before without crashing.

2. The "Spiking Memory" Muscle (RASNet)

The Problem: To learn how to move, robots usually need thousands of examples (demonstrations) from humans. This is like trying to learn to ride a bike by watching 1,000 videos of people riding bikes. It's slow, expensive, and the robot might just memorize the specific videos instead of learning the skill. Also, standard learning models often get "noisy" and forget important details when there is too much visual clutter.

The Solution: The authors created a new learning engine called RASNet that uses "Spiking" features.

The Analogy: Think of a standard robot learning like a student who tries to memorize every single word of a textbook. It's heavy and slow.
The "Spiking" Difference: The RGMP-S robot learns more like a human brain using spikes. Imagine a neuron that only "fires" (sends a signal) when it sees something truly important, like the edge of a cup, and ignores the background noise (like a patterned tablecloth).
The Benefit: Because it only pays attention to the "spikes" (the important details) and remembers them over time, it can learn a complex task from just 40 examples instead of 200. It's like learning to ride a bike after watching just a few videos because you intuitively understand the balance.

3. Putting It All Together

The system works in two steps:

The Planner (LGSS): The robot hears "Give me the Coke," looks at the table, sees a crushed can, and says, "Ah, that's flat. I need to use a 'pinch' move, not a 'grab' move."
The Doer (RASNet): The robot then executes the move. Because of its "spiking" memory, it ignores the messy background and focuses entirely on the can's shape and position to move its arm smoothly.

The Results

The authors tested this on a real humanoid robot and in computer simulations.

Better Success: The robot succeeded in tasks about 19% more often than the previous best methods.
Faster Learning: It learned new tasks using 5 times less data than other robots.
Real-World Proof: They tested it on tasks like pouring water, folding towels, and picking items from a messy bin. The robot handled these "long and tricky" tasks much better than before.

In short: This paper presents a robot that doesn't just "see" objects; it understands their shape and uses smart memory to learn how to move them efficiently, even when it has never seen that specific object before.

Technical Summary: RGMP-S for Humanoid Robot Manipulation

1. Problem Statement

Humanoid robot manipulation requires bridging high-level semantic reasoning with low-level action generation. Current frameworks face two critical bottlenecks:

Semantic-Geometric Gap: Vision-Language Models (VLMs) excel at semantic planning but often fail to translate abstract instructions into physically executable skills. They lack explicit geometric reasoning, leading to failures when object geometries deform (e.g., attempting to side-grasp a crushed can) or when spatial constraints are violated.
Data Inefficiency and Latency: State-of-the-art generative policies, such as Diffusion Policies, offer high expressiveness but suffer from prohibitive inference latency due to iterative denoising processes. Conversely, Transformer-based approaches often require massive datasets to achieve generalization, ignoring the intrinsic spatiotemporal mechanisms of manipulation and leading to overfitting in sparse demonstration scenarios.

The paper addresses the dual challenge of enabling robust, data-efficient generalization in unseen environments while maintaining real-time responsiveness.

2. Methodology: RGMP-S

The authors propose RGMP-S (Recurrent Geometric-prior Multimodal Policy with Spiking features), a unified framework comprising two core modules:

A. Long-horizon Geometric-prior Skill Selector (LGSS)

This module addresses the semantic-geometric gap by integrating geometric commonsense into VLMs without heavy 3D reconstruction.

Mechanism: It employs a lightweight, rule-based geometric tuning strategy. The system takes natural language instructions and RGB images, using a VLM (Qwen-vl) to localize objects and a segmentation model (YOLOv8n-seg) to extract shape properties (e.g., "cylindrical," "squashed").
Reasoning: A Chain-of-Thought (CoT) protocol enforces geometric consistency checks. The system aligns semantic instructions with latent geometric constraints to select appropriate manipulation primitives (e.g., "Side Grasp" for cylinders, "Pinch" for crushed objects) from a predefined library.
Efficiency: This approach avoids computationally expensive 3D reconstruction, operating with minimal fine-tuning and low latency (~105ms total inference).

B. Recursive Adaptive Spiking Network (RASNet)

This module addresses data efficiency and spatiotemporal modeling for action generation.

Architecture: RASNet is a feed-forward recursive architecture designed to capture intrinsic spatiotemporal mechanisms. It replaces standard recursive models to avoid vanishing gradients and noise accumulation.
Key Components:
- Adaptive Decay Mechanism (ADM): Dynamically regulates the retention of historical context, allowing the model to focus on task-critical features while suppressing redundant background noise.
- Rotary Position Embeddings (RoPE): Encodes positional information via geometric rotations, enhancing the model's ability to capture relative spatial dependencies without learnable position parameters.
- Adaptive Spike Neurons (ASN): Integrates spiking dynamics to modulate feature retention. Unlike fixed-decay spiking neurons, ASN learns an adaptive decay factor ( $\tau$ ) via channel-wise attention, enabling the network to filter temporal noise and amplify task-relevant signals.
- Spiking Dense Feature Extraction (SDFE): Extracts fine-grained spatial features from dense visual inputs, reducing redundancy.
Action Generation: The network outputs features fused via a multi-scale mechanism and approximates the action distribution using a Gaussian Mixture Model (GMM). This allows the policy to capture multi-modal action distributions, avoiding the "regression to the mean" problem common in single-Gaussian models.

3. Key Contributions

LGSS Framework: A novel skill selector that fuses VLMs with low-rank geometric adapters. It enables precise selection of parameterized skills based on shape-level commonsense biases, ensuring physical feasibility without task-specific fine-tuning.
RASNet Architecture: A policy architecture utilizing recursive spiking dynamics and adaptive decay to distill robust skills from sparse demonstrations. It effectively captures directional spatial dependencies and mitigates overfitting in data-scarce regimes.
Data-Efficient Generalization: The framework demonstrates robust performance across diverse physical platforms and simulation benchmarks, achieving high success rates with significantly fewer training trajectories compared to diffusion-based baselines.

4. Experimental Results

The authors evaluated RGMP-S on the ManiSkill2 simulation benchmark and three real-world robotic platforms (a custom humanoid, a desktop dual-arm manipulator, and a commercial Aloha robot).

Performance vs. SOTA: RGMP-S outperformed state-of-the-art baselines (including Diffusion Policy, Octo, OpenVLA, and Dex-VLA) across 10 manipulation tasks.
- Zero-Shot Grasping: Achieved an average success rate of 89%, surpassing the best baseline (Dex-VLA) by 12%. It demonstrated robust transfer to unseen objects (e.g., spray bottles, human hands) trained only on Fanta can data.
- Long-Horizon Tasks: In tasks like towel folding and water pouring, RGMP-S achieved success rates of 0.86 and 0.92, respectively, significantly outperforming models lacking explicit geometric guidance.
- Simulation Benchmarks: RGMP-S achieved the highest average score (16.24%) on ManiSkill2, notably outperforming the Diffusion Policy (10.01%) and ManiSkill2-1st (10.69%) in articulated object manipulation (e.g., Cabinet Door, Cabinet Drawer).
Data Efficiency: RGMP-S achieved performance comparable to a Diffusion Policy trained on ~200 trajectories using only 40 trajectories (a 5× improvement in data efficiency).
Real-Time Inference: The framework operates at 75.2 Hz, significantly faster than Diffusion Policy (1.01 Hz) and OpenVLA (3.6 Hz), making it suitable for real-time control.

5. Significance and Claims

The paper claims that RGMP-S represents a substantial advancement in embodied intelligence by effectively bridging the gap between high-level semantic reasoning and low-level physical execution. Its significance lies in:

Overcoming the Semantic-Geometric Gap: By injecting geometric priors directly into the decision-making process, the system ensures that semantic plans are physically executable, even in unseen or deformed scenarios.
Efficiency without Sacrifice: The integration of spiking dynamics and adaptive mechanisms allows for robust learning from limited data, challenging the prevailing "brute-force scaling" paradigm that relies on massive datasets and compute.
Reproducibility: The authors emphasize that the framework is fully reproducible, with source code and video demonstrations publicly available, providing a strong baseline for future research in efficient and generalizable robotic learning.

The authors acknowledge limitations in scenarios requiring extreme perceptual precision (e.g., millimeter-level insertion or manipulating fabric flush against a table), noting that these challenges reflect intrinsic physical complexities rather than solely algorithmic flaws.