💻 computer science

Learning Transferable Motor Skills for Geometry-Aware Robotic Surface Tasks

This paper proposes a modular framework that decouples geometric motion planning from execution-level expertise by training a multimodal neural network to infer interpretable, atomic motor rules from kinematic data and CAD geometry, thereby enabling the transfer of expert-like motor skills to robotic surface tasks across varying object topologies.

Original authors: Miroslav David, Karla Stepanova, Robert Babuska

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Miroslav David, Karla Stepanova, Robert Babuska

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to paint a wall or weld a metal frame. You have two main problems to solve:

The Map: The robot needs to know where to go. It needs a perfect geometric path that covers every inch without crashing into anything.
The Touch: The robot needs to know how to move while it's there. A human expert doesn't just move in a straight line at a constant speed. They slow down when turning a sharp corner, tilt their tool slightly to get a better angle, or speed up on long straightaways.

The Problem
Current robots are great at the "Map" part but terrible at the "Touch." If you teach a robot by showing it how to paint a square box, it learns to paint that specific square box. If you then give it a round table, it gets confused because it tried to memorize the exact movements for the square, not the skill of painting.

The Solution: A "Rule Book" for Robots
The authors of this paper propose a clever middle ground. Instead of teaching the robot the whole dance, they teach it a vocabulary of tiny, specific rules.

Think of it like this:

The Geometric Planner draws the skeleton of the path (the "Map").
The Expert Rules are like a set of "post-it notes" that tell the robot how to tweak that skeleton.

These rules are simple and interpretable, such as:

"Slow Down Rule": "If you are at a corner, reduce your speed by 20%."
"Tilt Rule": "If you are on a straight line, tilt your tool slightly to the left."

How They Taught the Robot
The team built a "brain" (a neural network) that acts like a detective. It looks at two things at once:

The Motion: A video of a human expert moving their arm.
The Shape: A 3D digital model of the object being painted.

The detective's job is to figure out: "Ah, the human slowed down here because it was a corner," or "They tilted the tool here because the surface was flat."

Once the robot learns these rules, it can apply them to any new shape. It doesn't need to have seen that specific shape before. It just needs to recognize the "corner" or the "straight line" and apply the correct rule from its vocabulary.

The "Simulator" Gym
To train this robot, they didn't use a real factory floor (which is slow and expensive). They built a physics simulator—a virtual gym.

They created fake objects (L-shaped and window-shaped frames).
They generated thousands of fake painting paths.
They let their "detective brain" practice finding the rules in this virtual world.

The Results
When they tested the system in the simulator:

The robot became incredibly good at identifying when to use a "corner rule" versus a "straight line rule" (almost 100% accuracy).
It could predict exactly how much to slow down or how much to tilt the tool.
Most importantly, when they gave it a new shape it had never seen before (like a window frame), it successfully applied the same rules it learned on the L-shaped objects.

The Bottom Line
This paper shows that you can separate "knowing where to go" from "knowing how to move." By breaking expert skills down into a simple list of rules (like "slow down at corners"), robots can learn from human demonstrations and then apply those skills to completely new shapes, rather than just copying the exact movements of the original demonstration.

Note: The authors mention that while this works perfectly in their computer simulation, the next big challenge is getting real robots to learn these rules from real humans, which is currently difficult because real sensors aren't fast enough to catch the tiny, quick adjustments humans make.

Technical Summary: Learning Transferable Motor Skills for Geometry-Aware Robotic Surface Tasks

Problem Statement

Robotic surface-interaction tasks, such as spray painting and welding, require a dual capability: accurate geometric path planning and precise execution strategies that mimic human expertise. While modern motion planners effectively generate collision-free paths satisfying geometric constraints, they often fail to capture the nuanced motor patterns of skilled human operators, such as adaptive velocity scaling and tool orientation adjustments based on local geometry. Conversely, Learning from Demonstration (LfD) approaches preserve these expert motor skills but typically couple execution tightly to the specific geometry of the demonstration. This coupling limits the transferability of learned skills to new workpieces with different shapes or topologies. The core challenge addressed is how to decouple high-level geometric planning from low-level execution expertise to enable transferable, geometry-aware motor skills.

Methodology

The authors propose a modular framework that represents expert behavior as a vocabulary of interpretable, atomic motor rules. These rules systematically modify a geometrically planned reference path based on local segment characteristics.

1. Rule-Based Skill Representation

Expert skills are modeled as a structured set of parametric transformations ( $\rho$ ) applied to spatial segments of a trajectory. The raw geometric path is decomposed into segments (e.g., straight lines, corners). The rule vocabulary includes:

Velocity Scaling ( $\rho_{vel}$ ): Applies a continuous scaling factor to the target linear speed.
Orientation Offsets ( $\rho_{ori}$ ): Introduces systematic tilts (represented as quaternion offsets) relative to the surface normal to mimic expert tool handling.

2. Multimodal Neural Inference

A multimodal neural network is trained to infer rule parameters and segment types jointly from kinematic trajectory data and CAD model geometry.

Inputs: The model processes a kinematic trajectory (position, orientation, velocity) and the underlying target CAD model (represented as a point cloud).
Architecture:
- Trajectory Encoder: A 1D convolutional encoder (three stacked Conv1d layers) processes time-indexed trajectory features. Orientations are mapped to a continuous 6D rotation representation for stability.
- Geometry Encoder: A PointNet++ encoder processes the CAD point cloud to extract global geometric features.
- Fusion: The embeddings from both modalities are concatenated and passed through a shared Multi-Layer Perceptron (MLP).
Outputs: The network utilizes two heads:
- A classification head predicting the geometric segment type (straight, corner, or none) to which a rule applies.
- A regression head predicting the continuous execution parameter (e.g., specific speed multiplier or tilt angle).

3. Trajectory Refinement and Simulation

To ensure physical continuity and avoid instantaneous velocity jumps, the inferred rules are injected into a physics-based simulation. A pure pursuit controller tracks a lookahead pose along the reference path, applying proportional forces and torques to position and orientation errors. The resulting wrench is numerically integrated using a fourth-order Runge-Kutta (RK4) method, modeling the end-effector as a rigid body with mass, inertia, and viscous damping.

4. Data Generation and Transfer

Training data is generated synthetically using L-shaped and window-shaped topologies. For transfer to unseen geometries, the system automatically segments a new geometric path using a sliding window and Singular Value Decomposition (SVD) to measure local curvature, labeling points as "straight" or "corner" without manual annotation. The inferred rules are then applied to the corresponding segments during simulation.

Key Contributions

The paper outlines two primary contributions:

A Modular Framework: A system that represents expert motor skills as a parametric rule vocabulary, allowing these skills to be injected into geometrically planned trajectories, thereby decoupling planning from execution-level expertise.
A Multimodal Neural Architecture: A specific network design combining a 1D convolutional trajectory encoder and a PointNet++ CAD encoder to infer rule parameters and segment types directly from demonstration data.

Experimental Results

The approach was evaluated on simulated datasets involving L-shaped and window-shaped objects.

Rule Classification: On L-shaped geometries, the model achieved a combined F1-score of 0.999 across three classes (straight, corner, none). On complex window geometries, it achieved an F1-score of 1.000 for velocity scaling and 0.984 for orientation rules.
Parameter Regression:
- L-shapes: Mean Absolute Error (MAE) was 0.020 rad for orientation offsets and 0.056 for velocity scaling factors.
- Window Topologies: The model maintained accurate orientation estimation (MAE 0.038 rad). Velocity scaling showed a moderate error of 0.131, which the authors attribute to the shorter, rapidly changing segments characteristic of window frames, suggesting a need for finer-grained segmentation or normalization for dense curvature changes.
Generalization: The results demonstrate that the architecture successfully disentangles dynamic overrides from the geometric trajectory, allowing the model to extract and apply velocity and orientation rules across different topologies.

Significance and Future Work

The paper claims that its primary significance lies in successfully decoupling execution strategies from workpiece geometry using interpretable, atomic rules. This allows for the transfer of expert motor skills to novel workpieces without retraining on the specific new geometry.

The authors note that the current rule vocabulary operates purely on segment type and does not yet exploit runtime proximity to specific CAD features (like edges or crossings) during inference. They identify bridging the sim-to-real gap as the critical next step, noting that current physical teaching interfaces suffer from low sampling rates that limit the capture of high-frequency motor adjustments. Future work aims to expand the rule vocabulary to include proximity-based rules conditioned on structural CAD features and to evaluate zero-shot transfer to held-out topologies such as U-shaped and T-shaped geometries.