💻 computer science

Motion Primitive Discovery in a Humanoid Robot via Self-Organising Maps for Phase Recognition

Inspired by the Mirror Neuron System, this paper proposes a two-level architecture for the NICO humanoid robot that utilizes Self-Organising Maps to encode arm and hand kinematics and an Echo State Network to achieve accurate online phase recognition of motor primitives.

Original authors: Radovan Gregor, Igor Farkaš

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Radovan Gregor, Igor Farkaš

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot to understand a human gesture, like a wave or a high-five, without just memorizing a list of "if-then" rules. This is the heart of a field called social cognition, where scientists try to figure out how machines can "get" what people are doing and why. To do this, researchers often look at the human brain for inspiration, specifically a fascinating group of cells called "mirror neurons." These are special brain cells that fire up not just when you do an action yourself, but also when you watch someone else do it. It's like your brain has an internal movie theater where it replays the action to understand it. The big question in robotics is: Can we build a computer system that works the same way? Instead of just looking at a video and guessing, can a robot build its own internal "vocabulary" of movements and use that to instantly recognize what phase of a movement is happening right now? This is crucial for robots that need to work alongside humans, because if a robot can predict that you are about to pick up a cup, it can move out of the way or hand you a napkin before you even ask.

This paper, written by Radovan Gregor and Igor Farkaš, explores exactly that idea using a humanoid robot named NICO. The team built a two-level system inspired by the mirror neuron concept to teach the robot how to recognize the "phases" of its own movements in real-time. Think of it like teaching a robot to dance. The first level of their system uses two special maps, called Self-Organising Maps (SOMs), which act like a library of movement patterns. One map learns the movements of the robot's arm, and the other learns the shapes of its hand. As the robot practices seven different actions—like picking up an object, eating, or waving—these maps automatically sort the thousands of tiny movements into neat, organized groups. It's as if the robot is discovering its own "motion primitives," which are like the basic Lego bricks of movement. The researchers found that the arm map and the hand map learn different, complementary things: the arm map is great at tracking the path and timing of the reach, while the hand map is a master at recognizing exactly how the fingers are shaped (like a power grip vs. a precision pinch).

Once these "Lego bricks" are discovered, the second level of the system kicks in. This part uses a type of neural network called an Echo State Network (ESN), which acts like a short-term memory bank. It watches the sequence of "bricks" the robot is using as it moves. The big question the researchers wanted to answer was: Does the robot need to know everything about the situation—like what object it is holding, how far away it is, or what the final goal is—to recognize what it is doing right now? Or is the sequence of movements alone enough?

To find out, they ran thousands of simulations where the robot performed these actions. They tested the system in two ways: first, by letting the ESN guess the current movement phase using only the sequence of arm and hand movements it had just seen. Then, they gave it extra "context" clues, like the name of the action or the distance to the object. The results were quite clear. When the system relied only on the movement sequence (the "motion primitives"), it got the current phase right about 93.9% of the time. When they added all the extra context clues, the accuracy only went up slightly to about 94.9%.

This suggests that the "story" told by the movement itself is the most important part. The extra information about the object or the goal is helpful, like a little hint, but it isn't the main driver of understanding. The robot's internal map of movements was already doing most of the heavy lifting. The authors conclude that by letting the robot organize its own movements into these topographic maps and then watching the flow of those maps over time, it can accurately recognize what it is doing in the moment. While this was all tested in a computer simulation with the NICO robot, the findings support the idea that self-organized motor representations are a powerful way to build robots that can understand and anticipate human-like actions without needing a massive database of every possible scenario.

Technical Summary: Motion Primitive Discovery in a Humanoid Robot via Self-Organising Maps for Phase Recognition

1. Problem Statement

The central challenge addressed is the computational basis of action recognition within human–robot interaction (HRI). Specifically, the paper investigates how a robot can perform online phase recognition—identifying the current movement phase of an ongoing action in real-time—using only sensory-motor streams.

The authors posit that action understanding operates on two levels: a repertoire of elementary motor primitives and the goal-directed sequences they compose. While modern deep learning approaches often classify entire sequences end-to-end, this work focuses on a biologically inspired architecture that separates the discovery of reusable motion primitives from the temporal integration required to recognize the current phase. The core research question is: To what extent is the structure of a current movement phase determined by the temporal dynamics of learned motor primitives alone, versus how much additional contextual information (e.g., object state, action identity) is required for accurate online recognition?

2. Methodology

The proposed system employs a two-level architecture inspired by the Mirror Neuron System (MNS), processing data from the NICO humanoid robot in a physics-based Unity simulation.

2.1 Data Collection and Preprocessing

Dataset: 4,200 simulated trials involving seven motor actions (Pick, Eat, Place, Push, Tap, Point, Wave) performed on 14 diverse objects.
Feature Selection: From 108 recorded kinematic variables, a hierarchical correlation analysis (using Pearson correlation and Ward's linkage) identified and removed redundancy. This resulted in 22 non-redundant features:
- 13 Arm features: TCP velocities, arm joint velocities, and palm orientation.
- 9 Hand features: Aperture, velocities, finger positions, and joint velocities.
Windowing: Data was segmented into 0.5s overlapping windows (30 frames at 60Hz) with 50% stride. Features were normalized via z-score.

2.2 Level 1: Motion Primitive Discovery (SOMs)

Two Self-Organising Maps (SOMs) were trained to learn topographic representations of the kinematic data, serving as a discrete vocabulary of motion primitives:

A-SOM (Arm SOM): A $35 \times 35$ grid trained on arm kinematics.
H-SOM (Hand SOM): An $18 \times 18$ grid trained on hand kinematics.
Output: For each timestep, the system identifies the Best Matching Unit (BMU) on both maps. These BMUs represent the current "primitive" state of the arm and hand.

2.3 Level 2: Online Phase Recognition (ESN)

An Echo State Network (ESN) acts as the temporal integrator.

Input: The sequence of BMU weight vectors ( $b_A(t)$ and $b_H(t)$ ) from the SOMs.
Reservoir: A fixed, randomly connected recurrent network ( $N=300$ neurons) with a spectral radius of 0.99 and a leaking rate of 0.50.
Readout: A linear readout layer trained via ridge regression to map the reservoir state to one of 27 possible movement phase labels.
Contextual Variations: The study evaluated four input configurations to test the necessity of context:
1. ctx-no: Primitive descriptors only (22D).
2. ctx-act: Primitives + Action identity one-hot (29D).
3. ctx-dst: Primitives + Action + Normalized palm-object distance (30D).
4. ctx-all: Primitives + Action + Distance + Object contact flag (31D).

3. Key Results

3.1 SOM Topography and Purity

A-SOM: Achieved 90.0% neuron utilization and 73.9% phase purity. The map exhibited clear topographic organization, grouping temporally similar movements (e.g., "reach" phases on the left, "terminal" phases on the right) without explicit temporal supervision.
H-SOM: Achieved 51.5% utilization (due to discrete hand synergies) but 97.2% grasp purity. It successfully isolated distinct grasp types (Power vs. Precision) and action-specific hand configurations (e.g., Point, Wave).
Complementarity: The two maps encoded complementary aspects; A-SOM captured transport dynamics, while H-SOM captured configuration states.

3.2 Phase Recognition Accuracy

The ESN was tested on a held-out set of 840 trials.

Baseline Performance (ctx-no): Using only the temporal trajectory of SOM activations (without any world-state context), the system achieved 93.9% phase recognition accuracy.
Impact of Context:
- Adding action identity (ctx-act) improved accuracy by +0.7 pp (94.6%).
- Adding distance (ctx-dst) maintained 94.6%.
- Adding full context (ctx-all) yielded a final accuracy of 94.9%.
Conclusion on Context: The marginal gain (+1.0 pp) from adding contextual information indicates that the temporal integration of primitive descriptors preserves the dominant phase-discriminative structure. Context serves as a secondary refinement rather than a primary signal.

3.3 Hyperparameter Sensitivity

Accuracy was found to be highly dependent on reservoir capacity ( $N$ ). Increasing $N$ from 30 to 1,000 improved accuracy monotonously (from ~73% to 97.3%). The benefit of contextual information diminished as reservoir capacity increased, suggesting that larger reservoirs can compensate for missing context through memory, whereas smaller reservoirs rely more heavily on external cues.

3.4 Single-Trial Analysis

In a representative "Eat" trial, the context-free model (ctx-no) achieved 98.4% accuracy, failing only once during a phase transition where the reservoir required an additional step to resolve ambiguity. The context-rich model (ctx-all) achieved 100% by using action identity and distance to resolve the transition one step earlier. The analysis confirmed that neither SOM provided unambiguous information at every step; the ESN successfully integrated the complementary streams to resolve local ambiguities.

4. Contributions and Significance

The paper's primary contribution is the integration of established SOM and ESN methods within an MNS-inspired architecture for motor primitive representation and online phase recognition.

Validation of Self-Organised Representations: The results support the computational hypothesis that self-organised motor representations, when temporally integrated, are sufficient to support accurate online recognition of ongoing movement phases.
Efficiency of Primitive Trajectories: The study demonstrates that the temporal trajectory of SOM activations carries the majority of the information required for phase discrimination, rendering explicit world-state context largely secondary for this specific task.
Dual-Map Architecture: The work validates the design choice of separating arm and hand kinematics into distinct topographic maps, reflecting the biological reality that these signals originate from distinct neural pathways and provide complementary discriminative structure.

The authors maintain a modest scope, noting that the model is currently trained and evaluated on the robot's own actions (self-observation). They explicitly state that extending this to cross-embodiment observation (watching humans) would require additional mechanisms to handle viewpoint and scale differences, which is reserved for future work. The current results establish the viability of the approach for internal state recognition and prediction.