🤖 machine learning

SEAR: Sample Efficient Action Chunking Reinforcement Learning

The paper introduces SEAR, a sample-efficient off-policy reinforcement learning algorithm that leverages a causal transformer critic with multi-horizon targets and a receding horizon replanning strategy to enable effective online learning with action chunks, outperforming state-of-the-art methods on challenging manipulation tasks.

Original authors: C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov, Florian Seligmann, Weiran Liao, Aryan Kaushik, Gerhard Neumann

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov, Florian Seligmann, Weiran Liao, Aryan Kaushik, Gerhard Neumann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a complex video game, like a maze where it has to pick up keys, open doors, and solve puzzles. In the world of artificial intelligence, this is called Reinforcement Learning. Think of it like training a dog: you give it a treat (a reward) when it does something right, and it learns to repeat that behavior. But here's the tricky part: if the game is very long, the dog might forget that the treat at the very end was because of the first thing it did. It needs to connect the dots between the start and the finish.

To help the robot learn faster, scientists use a trick called Action Chunking. Instead of telling the robot, "Move your arm left," then "Move your arm up," then "Grab the cup," one tiny step at a time, they say, "Here is a whole sequence: Move left, up, and grab." It's like teaching a pianist a whole measure of music at once instead of one note at a time. This helps the robot see the bigger picture and plan better. However, there's a catch. If you just tell the robot to play a whole chunk of music without letting it listen to the music as it plays, it might get lost if it hits a wrong note. It needs to be able to stop, listen, and adjust its plan constantly. This paper tackles the problem of how to teach robots these big "chunks" of action without making them too slow or clumsy to react when things go wrong.

Meet SEAR (Sample Efficient Action Chunking Reinforcement Learning), a new method designed to teach robots how to plan ahead without losing their cool. The researchers found that while telling a robot to execute a whole sequence of moves at once (a "chunk") is great for learning, doing it the old-fashioned way often leads to a disaster. If you just hand the robot a list of 10 moves and say "go," and it makes a mistake on move 3, it keeps going blindly until the list is done. By the time it realizes it's off track, it's too late.

The authors discovered that the main reason previous methods failed was that they didn't give the robot enough "practice" on the little pieces of the plan. Imagine trying to learn a dance routine by only practicing the entire 2-minute song once a day. You'd never get the individual steps right. SEAR fixes this by using a special kind of teacher, called a Causal Transformer Critic. Instead of just grading the robot on the final result of the whole dance, this teacher grades the robot on every single step of the sequence as it happens. It's like a dance instructor who stops the music after every move to say, "Good footwork!" or "Watch your elbow!" This gives the robot a massive amount of feedback from every single attempt, making it learn much faster and more efficiently.

But learning the big chunks is only half the battle. The robot also needs to be able to react quickly. To solve this, SEAR uses a clever strategy called Random Replanning. Instead of forcing the robot to execute the whole list of moves it planned, SEAR tells it to execute only a random part of the list—maybe just the first three moves—before stopping and asking, "Okay, what's the plan now?" based on where it actually is. This is like a GPS that doesn't just give you a route for the whole trip and say "drive," but recalculates your path every few blocks to avoid traffic jams. This keeps the robot's "state-action coverage" high, meaning it learns to handle all kinds of situations, not just the ones it planned for perfectly.

The results are impressive. When tested on the Metaworld benchmark, a collection of 20 very difficult robotic manipulation tasks (like picking up objects or pushing buttons), SEAR achieved a success rate of over 80%. This is a huge jump compared to previous methods, which only managed around 60%. Even more interestingly, the researchers found that SEAR could learn using these big chunks during training, but then switch to making single-step decisions during the actual test, effectively becoming a super-fast, single-step learner that was trained by thinking in big chunks.

The paper also tested SEAR in a "offline-to-online" setting, where the robot first learns from a huge dataset of past videos (offline) and then tries to improve by actually doing the task (online). By applying SEAR's techniques to an existing method called QC, the researchers saw a significant boost in performance on the OGBench cube-triple tasks, proving that these ideas work even when the robot starts with a head start of past data.

However, the authors are careful to note that this isn't a magic bullet for every robot. They point out that SEAR works best for tasks that take a while to complete and don't require split-second reactions, like assembling furniture or moving objects. It wouldn't work well for things that need instant reflexes, like a robot dog running through a forest, because the "chunking" approach can make the data quality drop if the robot needs to react too fast. But for the slow, deliberate, and complex tasks of robotic manipulation, SEAR suggests a powerful new way to teach machines how to think ahead, learn faster, and stay on their toes.

Technical Summary: SEAR (Sample Efficient Action Chunking Reinforcement Learning)

Problem Statement

Action chunking, a technique where a policy predicts a sequence of future actions rather than a single step, has shown remarkable success in imitation learning and offline reinforcement learning (RL). In RL, it facilitates faster value propagation and structured exploration, which is particularly beneficial for long-horizon tasks with sparse rewards. However, applying action chunking to standard off-policy online RL algorithms without specific design considerations leads to poor sample efficiency and performance degradation.

The paper identifies two primary failure modes when extending action chunking to online RL:

Critic Approximation Errors: The enlarged action space (a sequence of $N$ actions) makes Q-function approximation difficult. Existing methods often train critics to predict a single Q-value for the entire chunk. This results in a mismatch where the action space dimension increases with chunk size $N$ , but the number of training targets remains constant, leading to poor actor updates.
Reduced State-Action Coverage: Executing action chunks in an open-loop manner reduces the policy's decision frequency. If a policy predicts $N$ actions but only queries the environment every $N$ steps, the actor is rarely evaluated on intermediate states. This creates "holes" in state coverage, especially in environments with narrow initial state distributions. When such a policy is deployed with a receding horizon (executing only a prefix of the chunk), it encounters out-of-distribution states, causing performance to drop and negating the benefits of chunking.

Methodology: SEAR

The authors introduce SEAR (Sample Efficient Action Chunking Reinforcement Learning), an off-policy algorithm designed to overcome these limitations within the Maximum Entropy RL (MaxEnt-RL) framework. SEAR integrates three core components:

1. Multi-Horizon Q-Function Targets

To address the critic approximation issue, SEAR employs a multi-horizon target strategy. Instead of training the critic on a single target for the full $N$ -step return, it generates targets for every prefix of the action chunk ( $k=1$ to $N$ ).

Mechanism: For an action chunk $a_{t:t+N}$ , the critic is trained to predict Q-values $Q^{(k)}(s_t, a_{t:t+k})$ for all prefixes.
Benefit: This effectively increases the number of training targets per sample, making the ratio of action space dimension to training targets independent of the chunk size $N$ . This provides a denser training signal, stabilizing learning even for large chunk sizes.

2. Causal Transformer Critic

To handle the temporal dependencies inherent in action sequences and the multi-horizon targets, SEAR utilizes a causal transformer as the critic backbone.

Architecture: The transformer processes the state and the action sequence, predicting Q-values for all prefixes simultaneously.
Justification: The causal structure ensures that the $k$ -step target $G^{(k)}$ depends only on the first $k$ actions, aligning perfectly with the multi-horizon training objective. This architecture outperforms standard MLP critics in this setting.

3. Random Replanning for State Coverage

To restore state-action coverage and enable effective receding horizon control, SEAR introduces random replanning during data collection.

Mechanism: Instead of executing a fixed prefix length $k$ or the full chunk, the agent executes a random prefix of length $k \sim \text{Uniform}(1, N)$ before re-planning.
Benefit: This breaks the rigid pattern of evaluating the actor only every $N$ steps. It ensures that the actor is queried at various intermediate states, creating a uniform distribution of state coverage. This allows the policy to be robustly evaluated with shorter receding horizons at test time without suffering from distribution shift.

Key Contributions

The paper claims the following contributions:

SEAR Algorithm: An off-policy RL method for action chunking grounded in the MaxEnt-RL framework, specifically designed for online settings.
Multi-Horizon Targets & Causal Critic: The integration of a causal transformer critic with multi-horizon N-step returns, which significantly improves sample efficiency by providing per-action training targets within a chunk.
Random Replanning: A novel data collection strategy that executes random prefixes of action chunks to restore state-action coverage, enabling the effective use of receding horizon control.
Empirical Validation: Extensive experiments demonstrating that SEAR consistently outperforms state-of-the-art online and offline-to-online RL methods.

Experimental Results

The authors evaluated SEAR on two benchmarks: the Metaworld online RL benchmark and the OGBench offline-to-online benchmark.

Metaworld (Online RL):
- SEAR was tested on the 20 hardest environments in Metaworld.
- It achieved an Interquartile Mean (IQM) success rate of over 80%, substantially outperforming SimbaV2 (a state-of-the-art non-chunked method) and CQN-AS (a state-of-the-art chunked method), which both plateaued around 60%.
- An extension of SimbaV2 with action chunking achieved only ~20% success, highlighting the difficulty of transferring offline chunking techniques to online settings without SEAR's specific design choices.
- Ablation studies confirmed that removing multi-horizon targets or random replanning caused performance to drop to baseline levels (~20%).
OGBench (Offline-to-Online RL):
- Applying SEAR's components (multi-horizon targets and random replanning) to the QC framework improved performance on the demanding cube-triple tasks compared to standard QC and QC-FQL.
- The improvements were most pronounced early in the online training phase.
Receding Horizon Analysis:
- SEAR policies trained with large chunks (e.g., $N=10$ or $20$) but deployed with a short receding horizon (e.g., 3–5 steps) achieved the best performance.
- Notably, SEAR trained with chunking and deployed with a receding horizon of 1 (effectively a single-step policy) outperformed both the non-chunked SEAR-1 and SimbaV2, suggesting chunking can be used as a training-only mechanism to learn better single-step policies.

Significance and Claims

The paper positions SEAR as a solution to the specific challenges of applying action chunking in online reinforcement learning. The authors claim that previous methods failed in online settings because they did not address the dual challenges of critic approximation in high-dimensional action spaces and the lack of state coverage caused by open-loop execution.

The significance of SEAR lies in its ability to:

Decouple Training and Evaluation: It allows for the use of large action chunks during training to accelerate value propagation and structured exploration, while maintaining high reactivity at evaluation time through receding horizon control.
Enable Online Chunking: It demonstrates that action chunking is viable in purely online settings without prior offline data, provided the critic is trained with multi-horizon targets and the actor's state coverage is maintained via random replanning.
General Applicability: The core ideas (multi-horizon targets and random replanning) are shown to transfer effectively to offline-to-online settings, suggesting they are fundamental improvements for action chunking methods broadly.

The authors remain modest regarding limitations, noting that action chunking is best suited for long-horizon tasks (like robotic manipulation) and may not be suitable for high-reactivity domains (like locomotion) where the degradation of data quality from open-loop execution is too severe. They also acknowledge the increased computational cost of the transformer critic compared to single-step methods.