🤖 machine learning

Reward-Free Continual Adaptation for Resilient Space Robots

This paper introduces a reward-free continual learning framework that enables space robots to adapt to severe hardware degradation by leveraging a pre-trained latent-state world model to update transition dynamics via unsupervised rollouts and train policies on imagined trajectories, eliminating the need for infeasible reward signals during deployment.

Original authors: Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Space is a place where machines must work alone, far from the help of human hands. For decades, engineers have taught robots to move by giving them strict instructions or by letting them learn through trial and error, a method called reinforcement learning. In this learning process, a robot tries an action, and if it succeeds, it receives a digital "prize" or reward signal that tells it to do that action again. This system works well in controlled environments where computers can instantly calculate exactly how well a robot is doing. However, in the vast, untracked emptiness of space, there are no cameras or sensors to tell a robot if it is moving correctly. Without a way to measure success, the robot cannot receive the reward signals it needs to learn or adapt. This creates a dangerous problem: if a space robot breaks or wears down, it cannot fix its own behavior because it has no way to know it is failing.

A team of researchers at the University of Luxembourg has proposed a new way to solve this problem, allowing robots to adapt to severe damage without needing to know if they are winning or losing. Their approach relies on a concept called a "world model," which is essentially a mental map the robot builds inside its computer. Before the robot ever leaves Earth, scientists train it in thousands of different computer simulations, letting it experience every kind of terrain, gravity, and mechanical failure it might face. During this training, the robot learns not just how to move, but also how to predict what will happen next and what a "good" outcome looks like, even without seeing the final result. The researchers found that this mental map contains a hidden understanding of success, a kind of internal compass that points the robot in the right direction even when it cannot see the finish line.

The core of their discovery is a method to update this mental map when the robot is already in space and something has gone wrong. Imagine a rover driving on Mars that suddenly loses the ability to steer its front right wheel. In a traditional system, the robot would be stuck, unable to learn how to drive with a broken wheel because it cannot calculate a reward signal to guide its new movements. The researchers' new system, however, freezes the part of the robot's brain that understands what a reward looks like. It then uses the robot's own movements to update only the part of the brain that predicts how the world changes. By watching how the broken wheel actually moves the robot, the system learns the new reality of the damage. It then uses its frozen, pre-trained understanding of "good" outcomes to teach itself how to drive again, entirely on its own.

To test this idea, the researchers ran a series of rigorous simulations involving three very different space tasks. First, they simulated a rover with twelve motors trying to cross rough, rocky ground. They then simulated a spacecraft with twelve thrusters trying to navigate through orbit. Finally, they simulated a robotic arm with seven joints trying to screw a bolt into a nut with extreme precision. In each case, they introduced a sudden, severe failure: locking a wheel on the rover, disabling three thrusters on the spacecraft, and bending the tip of the screwdriver tool on the arm. These failures were designed to be so severe that a robot trained only on a working machine would fail completely, unable to perform its task.

The results showed that a robot using this new method could recover from these disasters, though with some limits. When the researchers compared the adaptive robot to a robot that had no way to learn from its mistakes, the adaptive robot showed promising initial recovery where the other failed. It was able to figure out how to move its broken parts to still reach its goals. However, the study revealed that the reward-free agent consistently underperformed compared to a version that was allowed to see the true reward signals, which is something a real space robot cannot do. Furthermore, the learning profiles showed that after an initial performance gain, the agent exhibited significant volatility and decay, particularly in the orbital and assembly tasks. This suggests that while the robot's internal map is powerful enough to guide it through the initial shock of failure, it struggles to maintain perfect accuracy over a long period without the constant correction of real-world feedback.

This work represents a significant step toward making space robots truly resilient. It proves that a machine does not need to be told it is doing well to learn how to fix its own broken movements, provided it has learned a deep understanding of how the world works beforehand. The researchers demonstrated that by separating the knowledge of "what is good" from the knowledge of "how things move," a robot can adapt to the unexpected realities of space. While the current tests were performed entirely in computer simulations, the findings offer a promising path forward for missions where human intervention is impossible, and where a robot must be able to survive and work even when its body is damaged. The ability to learn without a reward signal turns a potential mission-ending failure into a solvable challenge, bringing us closer to a future where our robotic explorers can truly endure the harshness of the cosmos.

Technical Summary: Reward-Free Continual Adaptation for Resilient Space Robots

Problem Statement

Space robots operate in extreme environments where hardware degradation (e.g., wheel damage, thruster failures, kinematic misalignments) can critically compromise traditional control strategies. While Continual Reinforcement Learning (RL) offers a mechanism for online adaptation, it fundamentally relies on access to accurate reward signals during deployment. In space, precise reward computation is often infeasible due to the absence of external tracking systems, limited sensing capabilities, and the complexity of the environment. For instance, estimating the volume of excavated regolith or the efficacy of orbital maneuvers in real-time without privileged state information is unreliable. This "unobservable reward" challenge creates a bottleneck for deploying adaptive learning systems in space, where human intervention is limited and failure stakes are high. Existing approaches often rely on manual reward engineering or domain shift handling but fail to address severe morphological failures without new reward data.

Methodology

The authors propose a Reward-Free Continual Adaptation framework built upon the DreamerV3 latent-state world model architecture. The approach is divided into two distinct phases:

1. World Model Pre-Training

The agent is pre-trained in diverse simulations using domain randomization (varying gravity, friction, inertial properties, etc.).

Architecture: The model utilizes a Recurrent State-Space Model (RSSM) to compress high-dimensional observations ( $x_t$ ) into compact stochastic latent states ( $z_t$ ).
Components: The system jointly optimizes a sequence model, forward dynamics, encoder, decoder, reward predictor, and continuity predictor.
Training Objective: The model reconstructs observations, predicts rewards ( $\hat{r}_t$ ) and episode continuity, and minimizes the KL divergence between prior and posterior dynamics. An actor-critic policy is trained entirely within the latent trajectories generated by the world model, using the predicted rewards to learn the underlying task structure.

2. Reward-Free Adaptation

Upon deployment to an environment with severe hardware degradation, the agent adapts without receiving new external rewards.

Freezing Mechanism: The observation encoder, decoder, and crucially, the reward predictor, are frozen. This preserves the "latent reward landscape" learned during pre-training.
Unsupervised Rollouts: Only the transition dynamics (sequence model and forward dynamics) are updated via unsupervised rollouts on the new, degraded hardware. The agent collects proprioceptive data to learn a corrective delta to the transition dynamics.
Policy Update: The actor-critic policy is retrained entirely on "imagined" trajectories generated by the updated world model. The agent leverages the frozen reward predictor to receive meaningful reward signals based on the updated latent representations, allowing it to adapt to altered dynamics without external reward feedback.
Stabilization: To mitigate catastrophic forgetting, the learning rate is reduced by an order of magnitude, and Gaussian exploration noise is injected during online rollouts.

Key Contributions

Framework for Reward-Free Adaptation: The paper introduces a method that enables continual learning in space robotics by decoupling dynamics updates from reward prediction, relying on a pre-trained latent reward landscape.
Latent Reward Hypothesis: It demonstrates the hypothesis that world models pre-trained across diverse simulations inherently encode a robust reward structure within their latent space, which can guide adaptation even when the transition dynamics change significantly.
Benchmarking: The authors provide a benchmark (Space Robotics Bench) and evaluate the approach across three distinct domains:
1. Planetary Traversal: A rover with a locked steering/drive joint (wheel failure).
2. Orbital Navigation: A spacecraft with failed thrusters (SE(3) navigation).
3. Precision Assembly: A manipulator with a misaligned screwdriver tool (kinematic deviation).

Experimental Results

The evaluation was conducted in simulation (NVIDIA Isaac Lab) with a strict separation between a data-abundant pre-training phase (20 million steps) and a resource-constrained adaptation phase (60 minutes of interaction).

Baselines: The proposed method was compared against a zero-shot baseline, an agent retrained from scratch, and an adaptive agent with access to "privileged" (true) rewards.
Performance:
- Zero-Shot: Failed catastrophically across all domains due to unmodeled dynamics shifts.
- Retrained from Scratch: Confirmed task solvability but highlighted extreme sample inefficiency.
- Privileged Reward Agent: Achieved rapid recovery, nearing the performance of the retrained baseline.
- Reward-Free Agent: Demonstrated promising initial recovery, proving that latent world models can guide adaptation without external rewards. However, the agent consistently underperformed compared to the privileged baseline.
Limitations Observed: The reward-free agent exhibited significant volatility and performance decay in later stages, particularly in orbital and assembly tasks. The authors attribute this to the latent representation drifting away from the original space as the transition dynamics are updated on degraded morphologies, causing the frozen reward predictor to become less accurate over time.

Significance and Claims

The paper claims to represent a significant step toward truly resilient autonomy for extreme environments. By resolving the critical bottleneck of unobservable rewards, the framework allows robots to adapt to unforeseen hardware failures without human intervention or the need for complex external tracking systems.

However, the authors maintain a modest stance regarding the current limitations:

Capacity Limits: The late-stage performance decay suggests a capacity limit within the RSSM, where continuous dynamics updates eventually overwrite the generalized principles acquired during pre-training.
Sim-to-Real Gap: The study is simulation-only and does not address the peculiarities of the sim-to-real gap.
Computational Constraints: The demanding optimization required for online adaptation currently exceeds the strict power constraints of space-grade embedded compute modules, necessitating future breakthroughs in efficient in-situ learning.

Future work is proposed to investigate localized latent-space adapters to bound dynamics updates and prevent the drift that causes performance decay.