💻 computer science

Emergent Neural Automaton Policies: Learning Symbolic Structure from Visuomotor Trajectories

The paper introduces ENAP, a bi-level neuro-symbolic framework that adaptively infers an interpretable Mealy state machine from visuomotor demonstrations to guide a low-level controller, thereby achieving superior sample efficiency and long-horizon task performance compared to state-of-the-art end-to-end policies without requiring hand-crafted symbolic priors or task-specific labels.

Original authors: Yiyuan Pan, Xusheng Luo, Hanjiang Hu, Peiqi Yu, Changliu Liu

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiyuan Pan, Xusheng Luo, Hanjiang Hu, Peiqi Yu, Changliu Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to build a complex Lego castle.

If you use traditional "End-to-End" AI (like the current popular models), it's like handing the robot a massive, blurry photo of the finished castle and saying, "Just copy this." The robot tries to memorize every single pixel and muscle movement at once. It works okay for simple tasks, but if you ask it to build a castle with a different color scheme or a slightly different shape, it gets confused. It's a "black box"—we don't know why it made a mistake, and it needs to see thousands of examples to learn.

If you use old-school "Symbolic" AI, it's like giving the robot a rigid, handwritten instruction manual: "Step 1: Pick up red block. Step 2: Move to X. Step 3: Place block." This is very clear and logical, but it breaks immediately if the robot drops a block or if the table is slightly tilted. It can't adapt to the real world because the rules are too strict.

ENAP (Emergent Neural Automaton Policy) is the "Goldilocks" solution. It's a framework that teaches the robot to figure out its own logic while learning how to move.

Here is how it works, broken down into three simple parts:

1. The "Detective" Phase (Finding the Story)

Instead of giving the robot a manual, we show it a video of a human expert building the castle. The robot watches the video and acts like a detective.

The Analogy: Imagine watching a movie and realizing, "Oh, the hero is in the 'Chase' scene now," and then, "Now they are in the 'Hide' scene."
What ENAP does: It looks at the messy, continuous video data and automatically groups moments into distinct "phases" or "modes." It discovers that there is a "Reach" phase, a "Grasp" phase, and an "Insert" phase. It doesn't need a human to tell it these exist; it finds them on its own.

2. The "Map Maker" Phase (Drawing the Flowchart)

Once the robot has identified these phases, it draws a flowchart (a state machine).

The Analogy: Think of a subway map. The stations are the phases (Reach, Grasp, Insert), and the lines are the connections between them.
The Magic: This map isn't rigid. If the robot tries to insert a peg and it misses (a "failure"), the map knows there is a "loop" back to the "Align" station to try again. It learns that mistakes happen and that the logical next step is to "go back and try," rather than just crashing. This is called autonomous failure recovery.

3. The "Driver" Phase (The Fine Motor Skills)

Now the robot has a high-level map (the flowchart), but it still needs to know exactly how to move its fingers to pick up a specific block.

The Analogy: The flowchart is the GPS telling you, "Turn left at the next intersection." The "Driver" is your actual hands steering the car.
How it works: The flowchart gives a "coarse" instruction (e.g., "Move hand toward the block"). Then, a small, fast neural network (the residual network) adds the tiny, precise adjustments needed to actually grab the block without knocking it over.

Why is this a big deal?

It Learns Faster: Because the robot understands the structure of the task (the flowchart), it doesn't need to memorize every single movement from scratch. It's like learning the rules of chess instead of memorizing every possible game. The paper shows it needs 39% fewer data points to learn than other top methods.
It's Explainable: If the robot fails, we can look at the flowchart and say, "Ah, it got stuck in the 'Align' loop because the peg was crooked." We can see why it failed. Traditional AI is a black box; ENAP is a white box with a clear map.
It Handles "What Ifs": If you change the task slightly (e.g., "Stack the blocks in a different order"), the robot can often just follow a different path on its existing map, whereas other robots would need to be retrained from scratch.

In a Nutshell

ENAP is a robot learning system that teaches a robot to write its own instruction manual while it learns to move. It separates the "big picture thinking" (planning the steps) from the "muscle memory" (moving the arm), allowing the robot to be smart, adaptable, and easy to understand, even when things go wrong.

1. Problem Statement

Scaling robot learning to long-horizon tasks remains a critical challenge. Current approaches face a dichotomy:

End-to-End Policies (e.g., VLA models): While effective for short tasks, they often lack structural priors, resulting in "black-box" policies that are data-intensive, semantically opaque, and struggle with multi-stage reasoning.
Traditional Neuro-Symbolic Methods: These offer interpretability and structure but rely heavily on hand-crafted symbolic priors (e.g., manually defined temporal logic predicates), which limits their adaptability and requires significant expert engineering.

Core Question: How can the logical structure of high-level planning (System 2) naturally emerge from the perceptual flow of low-level control (System 1) without manual labeling or predefined symbols?

2. Methodology: The ENAP Framework

ENAP (Emergent Neural Automaton Policy) proposes a bi-level neuro-symbolic framework that unsupervisedly reconstructs a task-level state machine from raw visuomotor demonstrations. It combines a discrete high-level planner with a continuous low-level controller.

A. Core Architecture

The policy is represented as a Probabilistic Mealy Machine (PMM) coupled with a Residual Network:

High-Level (System 2): A PMM ( $M = (Q, \Sigma, \Gamma, \delta, \lambda, q_0)$ $M = (Q, Σ, Γ, δ, λ, q_{0})$ ) that captures latent task phases (states) and transitions.
- States ( $Q$ ): Discrete task modes (e.g., "Reach," "Grasp").
- Transitions ( $\delta$ ): Probabilistic transitions between modes based on observations.
- Outputs ( $\lambda$ ): Coarse action priors (mean actions) for each state-input pair.
Low-Level (System 1): A residual network ( $\pi_\psi$ $π_{ψ}$ ) that takes the coarse action prior and the current observation to predict precise continuous control actions.
- Final Action: $\hat{a}_t = a_{base} + \Delta a_t$ , where $a_{base}$ comes from the PMM and $\Delta a_t$ is the residual.

B. Three-Stage Pipeline

The framework operates via an Iterative Expectation-Maximization (EM) process (Algorithm 2):

Symbol Abstraction (E-Step):
- Encoding: Continuous observations (images + proprioception) are encoded into feature vectors $z_t$ using a neural encoder ( $\phi_\theta$ ).
- Clustering: An adaptive clustering algorithm (HDBSCAN) groups these features into discrete symbols ( $\Sigma$ ), effectively discretizing the continuous space without labels.
- RNN Embedding: An RNN maps variable-length history sequences of actions and symbols into fixed-size embeddings ( $h_t$ ) to serve as continuous surrogates for discrete states. A contrastive loss ensures embeddings cluster around saturated centers, facilitating state distinction.
Structure Extraction (Extended $L^*$ Algorithm):
- The authors extend Angluin's $L^*$ algorithm to work with continuous data and without an oracle.
- Self-Supervised Membership Query (MQ): Instead of querying an oracle, the algorithm mines the demonstration dataset. It checks if new history embeddings fall within a similarity threshold ( $\tau_{sim}$ ) of existing centroids. If not, a new state is added (ensuring Closedness).
- Non-deterministic Equivalence Query (EQ): Validates the hypothesis by checking if test trajectories can be explained by at least one valid path in the PMM. If a trajectory fails, the prefix causing failure is added as a counter-example to refine the structure.
- Pruning: Stable phases (self-loops) are merged to create a compact automaton.
Policy Optimization (M-Step):
- With the PMM structure fixed, the residual network and the encoder are fine-tuned via Behavior Cloning (BC).
- The objective minimizes the error between the predicted action and the expert action, while a regularization term ( $L_{center}$ ) keeps features close to cluster centers to maintain structural separability.

3. Key Contributions

Emergent Structure Discovery: ENAP is the first framework to unsupervisedly learn a Mealy state machine directly from raw visuomotor trajectories, eliminating the need for hand-crafted symbolic priors or task-specific labels.
Bi-Level Neuro-Symbolic Policy: It unifies discrete reasoning (via the automaton) and continuous control (via the residual network), offering a representation that is both kinematically feasible for robots and semantically interpretable for humans.
Extended $L^*$ for Robotics: The paper introduces a novel adaptation of the $L^*$ algorithm that handles continuous inputs, stochasticity, and offline data mining, bridging the gap between formal automata learning and deep reinforcement learning.
Theoretical Justification: The authors provide a rigorous derivation showing how the bi-level architecture factorizes the Bellman equation, reducing the complexity of function approximation by separating global dynamics from local residuals.

4. Experimental Results

Experiments were conducted on Complex Manipulation, Long-Horizon TAMP, and Real-World tasks.

Performance vs. State-of-the-Art (SoTA):
- In low-data regimes, ENAP outperforms end-to-end VLA policies (e.g., OpenVLA, $\pi_0$ ) by up to 27% in success rate.
- On the DualStackCube task, ENAP achieved 76.0% success with only 22.9M parameters, compared to 69.8% for OpenVLA (7.6B parameters).
- In Long-Horizon TAMP (CALVIN), ENAP distilled from a pre-trained VLA (FLOWER) improved sequential task completion from 91.0% to 97.0%.
Sample Efficiency: ENAP maintains high efficacy even with only 25% of the training data, demonstrating that the extracted PMM acts as a strong inductive bias.
Interpretability & Recovery:
- The learned automata successfully identified logical branching (e.g., choosing different grasp strategies based on object location).
- Autonomous Failure Recovery: The PMM learned to transition back to previous states (e.g., $q_2 \to q_1$ ) to retry actions upon error, a behavior not explicitly programmed but emergent from the data.
Generalization: In multi-modal tasks (pushing to multiple goals), ENAP successfully separated conflicting data modes into distinct topological paths, outperforming monolithic policies that tend to average out conflicting gradients.

5. Significance

This work represents a significant step toward generalizable and interpretable robotic intelligence.

Bridging the Gap: It solves the "alignment" problem between high-level cognitive planning and low-level sensorimotor control by allowing the structure to emerge from data rather than being imposed by humans.
Data Efficiency: By decomposing the problem into discrete phases and continuous residuals, it drastically reduces the data required to learn complex, long-horizon tasks.
Safety and Trust: The explicit state machine provides a transparent map of the agent's decision logic, enabling formal verification and easier debugging compared to black-box neural networks.

In summary, ENAP demonstrates that complex robotic behaviors can be decomposed into a hierarchical, emergent structure that combines the robustness of symbolic planning with the flexibility of neural control, paving the way for more efficient and explainable autonomous agents.