🤖 machine learning

LM-X: Explainable Action Modeling with Progress, Event, and Uncertainty Prediction for Generalist Robot Manipulation

LM-X is an explainable action modeling framework for generalist robot manipulation that enhances long-horizon control performance by explicitly predicting task progress, semantic events, and local reliability, achieving superior success rates on real-robot benchmarks compared to state-of-the-art models.

Original authors: Jin Lou, Jingxuan Zhu, Andong Chen, Xupeng Wang, Yuan Xu, Yuexuan Li, Xingdong Zhu, Zhijie Zhu, Yingwei Ji, Wenpeng Nie, Jingyi Li, Liangliang Chen, Jinyan Liu, Zhiqi Song, Jidong Zhang, Hongming Li

Published 2026-08-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jin Lou, Jingxuan Zhu, Andong Chen, Xupeng Wang, Yuan Xu, Yuexuan Li, Xingdong Zhu, Zhijie Zhu, Yingwei Ji, Wenpeng Nie, Jingyi Li, Liangliang Chen, Jinyan Liu, Zhiqi Song, Jidong Zhang, Hongming Li, Yuchen Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that can learn to perform complex tasks by watching humans are no longer science fiction; they are a rapidly evolving reality in the field of artificial intelligence. These systems, often called generalist robots, use a combination of vision and language to understand instructions like "fold the shirt" or "pick up the cup" and then translate those words into physical movements. For years, the standard approach has been to train these robots to predict the very next movement they should make, step by step. While this method works well for simple, short actions, it hits a wall when tasks become long and complicated. The robot sees the immediate command but remains blind to its own progress, unsure if it is moving closer to the goal or drifting away from it. It cannot tell if a hesitation is a moment of careful thought or a sign of confusion, nor can it explain why it chose a specific motion. This lack of internal awareness makes it difficult to trust the robot in unpredictable environments or to fix it when it goes wrong.

A new approach, detailed in recent research, seeks to solve this by giving the robot a voice inside its own control system. Instead of just outputting a movement, the new model, called LM-X, is designed to constantly answer three specific questions about what it is doing. First, it asks, "Am I getting closer to finishing the job?" Second, it asks, "What is the next major step I am trying to achieve?" And third, it asks, "How confident am I that this specific movement will work?" By training the robot to predict these answers alongside its physical actions, the researchers have created a system that is not only more capable but also transparent about its own state. The robot no longer just acts; it monitors its own progress and uncertainty in real time, allowing it to handle complex, multi-step tasks with a level of reliability that previous models struggled to achieve.

The researchers built this system by drawing inspiration from how biological brains organize movement. In nature, the brain does not just fire muscles randomly; it maintains a hierarchy of predictions. It holds a slow-moving sense of the overall goal, a faster sense of the immediate sub-goal, and a constant check on the reliability of the motor commands being sent. The LM-X model translates this biological principle into software. It does not try to copy the anatomy of a brain, but rather adopts its functional strategy of using different time scales to guide behavior. The model is trained on a massive dataset of over 20,000 hours of real robot movements, including thousands of hours of failed attempts. This inclusion of failure is crucial. By watching the robot struggle, miss a grasp, or get stuck, the system learns to recognize what a regression looks like, rather than just memorizing what success looks like.

The core of the system consists of three distinct signals that run in parallel with the robot's movements. The first signal, which the researchers call "return-to-go," acts as a progress bar. It is a single number that estimates how much of the task remains. If the robot is moving smoothly toward the goal, this number rises. If the robot makes a mistake, such as dropping an object or moving in the wrong direction, the number drops immediately. This allows the system to detect a problem the moment it happens, rather than waiting until the task is completely finished. The second signal, "event-to-go," identifies the next meaningful transition in the task. In a complex job like assembling a piece of furniture, there are distinct phases: picking up a part, inserting it, and tightening a screw. This signal tells the robot which of these phases it is currently pursuing, ensuring it stays on the right track even if the immediate movements look similar to a different part of the process. The third signal measures uncertainty. It calculates how much the robot's prediction varies, essentially asking how sure it is about the next move. When the robot hesitates or oscillates back and forth, this uncertainty signal spikes, flagging that the robot is unsure of its action.

To test if these extra signals actually helped, the researchers first ran a smaller experiment on a set of five difficult tasks before committing to a massive, expensive training run. They compared a standard robot model that only predicted movements against versions that included one or more of the new signals. The results were clear: the model that included all three signals performed significantly better than the standard model, and also better than any version that only had one of the new features. This proved that the signals worked together to provide a complete picture of the task. The researchers then trained the full model on the large dataset of real-world robot data. When tested on a benchmark of 50 different tasks, the new model succeeded in 74.1% of the attempts, a substantial improvement over the previous best model, which succeeded in only 55.4% of the cases.

The improvement was even more pronounced when the model was tested on seven different real-world tasks using four different types of robots. In these physical tests, the new model succeeded 68.6% of the time, compared to 50.7% for the previous standard. The researchers observed that the progress signal was highly responsive to the robot's actual performance. In one experiment involving a bimanual task where the robot had to sort objects with two hands, the progress score rose as the robot successfully moved items into place. However, the moment the robot dropped an object or moved away from the target, the score fell instantly, accurately reflecting the setback. Similarly, the uncertainty signal provided a clear window into the robot's hesitation. When the robot was trying to grasp an object and kept adjusting its position without committing, the uncertainty signal rose. Once the robot made a firm decision and closed its gripper, the signal dropped, indicating confidence.

This work suggests that for robots to become truly reliable partners in complex environments, they need more than just the ability to move their arms. They need to understand their own state. By explicitly predicting progress, intent, and reliability, the LM-X model bridges the gap between raw mechanical action and intelligent decision-making. The researchers emphasize that these signals are not just for human observers to read; they are part of the robot's own control loop, helping to shape the next movement based on the current state of the task. While the system is not perfect and still struggles with some specific tasks, the results demonstrate that adding these layers of self-awareness makes the robot significantly more robust. The study concludes that this multi-layered approach offers a practical path forward for building robots that can not only perform difficult tasks but also explain their own behavior and recover from mistakes when things go wrong.

Technical Summary: LM-X – Explainable Action Modeling with Progress, Event, and Uncertainty Prediction

1. Problem Formulation

The paper identifies a dual bottleneck in current generalist Vision-Language-Action (VLA) policies for robot manipulation:

Single-Target Supervision Bottleneck: Most VLAs compress the entire behavior hierarchy into a single short-horizon action prediction target. Consequently, critical intermediate variables—task progress, intermediate intent (event-level transitions), and local action reliability—are not explicitly supervised. They must emerge implicitly from the same loss function used for action prediction, often failing to distinguish between nominal progress, regression, and ambiguity.
Control Observability Bottleneck: Operators can observe the robot's physical actions but cannot access the policy's internal state regarding whether execution is advancing, which semantic transition is being pursued, or the reliability of the local command. This lack of transparency hinders diagnosis and recovery, especially in long-horizon tasks where similar visual configurations may correspond to different intentions (e.g., a failed grasp vs. a new target).

The authors argue that an action is not its own explanation. Post-hoc natural language rationales or visual attributions often disconnect from the features driving the command. Instead, useful explanation for control requires exposing compact, operationally defined variables that are available before action execution and are intrinsic to the policy's control computation.

2. Methodology: LM-X Architecture

The authors propose LM-X, a VLA policy inspired by biological sensorimotor organization, which posits that slow contextual estimates constrain event-scale behavior, which in turn constrains fast motor predictions, while uncertainty regulates local control. LM-X translates this principle into a computational architecture with three explicit, supervised explanatory signals:

2.1 Core Components

Return-to-Go (RTG): A task-scale scalar signal representing progress and state quality. It answers: "Is the observed state progressing at the task scale?"
- Definition: An empirical return calculated from the terminal outcome and remaining duration. Successful trajectories approach zero as they near completion; failed trajectories receive a distinct negative margin.
- Role: It provides a dense learning signal for progress, distinguishing nominal advancement from regression (e.g., returning to a start pose). It conditions the event prediction head.
Event-to-Go (ETG): An event-scale action chunk representing the next semantic transition. It answers: "What task-relevant transition is being pursued?"
- Definition: An action chunk extending to the next verified semantic boundary (e.g., "grasp complete," "insertion started"). It is trained via conditional flow matching.
- Role: It exposes the policy's intermediate intent. Unlike standard action chunking, ETG is explicitly tied to verified semantic boundaries, allowing the model to distinguish between subgoal selection errors and execution errors.
Heteroscedastic Action Flow (Uncertainty): A motor-scale signal predicting action variance. It answers: "How reliable is the motor-scale prediction?"
- Definition: A flow-matching head that predicts both the mean velocity and the log-variance of the action distribution.
- Role: It captures local ambiguity. High variance is associated with hesitation, oscillation, or off-manifold states, while low variance indicates confident execution.

2.2 Architecture and Information Flow

LM-X is built on the Cosmos-Reason2-2B vision-language backbone. The architecture enforces a directed factorization (Eq. 3):
$p(R_t, E_t, A_t | x_t) = p(R_t | z_t) \cdot p(E_t | z_t, s_t, R_t) \cdot p(A_t | z_t, s_t, R_t, E_t)$

RTG Expert: Operates solely on visual-language embeddings ( $z_t$ ) to assess progress without relying on specific joint states, encouraging generalization across embodiments.
Event Expert: Receives $z_t$ , proprioception ( $s_t$ ), and the predicted RTG. It outputs an event chunk ( $E_t$ ) and a hidden state ( $z_{event}$ ).
Action Expert: A Diffusion Transformer conditioned on $z_t$ , $s_t$ , RTG, and $z_{event}$ . It generates the fine-grained action chunk ( $A_t$ ) and the associated uncertainty ( $q_t$ ).

This structure ensures that semantic assessment precedes transition prediction, which precedes motor control, making explanation intrinsic to the control graph rather than a post-hoc annotation.

2.3 Training and Data

Dataset: The model is trained on over 20,000 hours of real-robot trajectories from 15+ embodiments (humanoids, dual-arm, single-arm), including 1,000+ hours of failed policy rollouts.
Annotation: A pipeline converts diverse physical transitions into shared event-conditioned targets. Failed rollouts are crucial for teaching the model to distinguish progress from regression and commitment from ambiguity.
Pretraining Gate: Before the costly 20-day pretraining run on 64 NVIDIA B200 GPUs, a controlled ablation study on 5 RoboTwin2.0 tasks validated that retaining all three modules yields significant gains over single-component variants.

3. Key Contributions

Dual-Bottleneck Formulation: A formal separation of single-target supervision limitations from control-state observability limitations in VLA learning.
LM-X Model: A brain-inspired, explainable VLA that explicitly predicts task progress (RTG), event-level intention (ETG), and local reliability (variance) as supervised targets within the control pathway.
Cost-Aware Pretraining Strategy: A "pretraining gate" using controlled ablations to justify the inclusion of complex modules before committing to large-scale resource-intensive training.
Scalable Annotation: A method for generating dense progress and event labels from heterogeneous real-robot data, including failure cases, to support multi-timescale learning.

4. Experimental Results

4.1 Pretraining Gate (Simulation)

On a 5-task RoboTwin2.0 benchmark, the complete LM-X model achieved a 79.6% mean success rate, a 16.0 percentage point (pp) gain over the action-only backbone (63.6%) and a 10.8 pp gain over the strongest single-head variant. This confirmed the complementary nature of the three signals.

4.2 RoboTwin2.0 Benchmark (Simulation)

Evaluated on 50 randomized-hard tasks:

LM-X: 74.1% mean success.
GR00T N1.7: 55.4% mean success.
Result: LM-X outperformed the baseline on 8 of 10 representative tasks shown, with significant gains in complex tasks like "stamp_seal" (+52 pp) and "blocks_ranking" (+30 to +33 pp).

4.3 Real-World Evaluation

Evaluated on 7 tasks across 4 real-world embodiments (Aloha, Astribot S1, Tianji-Marvin, Loong S1):

LM-X: 68.6% mean success.
GR00T N1.7: 50.7% mean success.
Result: LM-X improved 5 tasks, tied 1, and regressed on 1. The largest gain was on "tableware organization" (+60 pp). The median task-wise gain was +10 pp.

4.4 Diagnostic Signal Analysis

RTG: Demonstrated non-monotonic behavior, decreasing during visible regressions (e.g., missed grasps, target switches) and increasing upon semantic completion. It responded to state quality rather than just elapsed time.
Uncertainty: Variance spikes correlated with hesitation, oscillation, and failed episodes. Failed rollouts exhibited frequent high-variance intervals, while successful ones remained stable after initial transients.

5. Significance and Claims

The paper claims that explicit multi-timescale predictive states serve as a practical interface between stronger control and intrinsic policy explainability.

Intrinsic Explainability: Unlike detached decoders, LM-X's explanatory signals are operationally defined, available online, and coupled to the action generation process. This allows for process-level inspection (what the policy believes about state and intent) rather than just outcome-level inspection.
Functional, Not Mechanistic: The authors emphasize that LM-X is a functional realization of hierarchical predictive state principles observed in biology, not a biological replica. The "brain-inspired" label refers to the computational hierarchy (slow context $\to$ intermediate events $\to$ fast control), not anatomical correspondence.
Limitations: The authors note that the uncertainty analysis is qualitative and temporal correspondences are not yet calibrated detectors. The model does not guarantee causal faithfulness, and safety-critical deployment requires combining these signals with external constraints.

The results support the hypothesis that separating supervision for progress, intent, and reliability improves both control performance and the ability to diagnose execution dynamics in long-horizon manipulation.