🤖 machine learning

Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning

The paper proposes Shortcut Trajectory Planning (STP), a single-stage offline reinforcement learning framework that employs conditional shortcut models to enable efficient, adjustable-step trajectory generation with reduced training and inference costs while maintaining strong performance across diverse benchmarks.

Original authors: Guanquan Wang, Yoshimasa Tsuruoka

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guanquan Wang, Yoshimasa Tsuruoka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a robot to walk, navigate a maze, or pick up a pen, but you can't let the robot practice in the real world. You only have a giant video library of someone else trying (and sometimes failing) to do these tasks. This is the world of Offline Reinforcement Learning. The robot has to learn by watching the tapes, not by bumping into walls.

For a while, the best way to do this was using "Diffusion Planners." Think of these like a very talented but slow artist. To draw a perfect path for the robot, the artist starts with a messy scribble and slowly, step-by-step, erases the noise to reveal the picture. It takes a long time to erase all the noise (many "sampling steps"), which makes the robot slow to react.

Then, a new idea came along: Consistency Planners. These are like a student who watches a master teacher draw the picture once, then tries to learn how to draw the whole thing in just one or two big strokes. It's super fast! But there's a catch: you first have to train the master teacher, and then train the student to copy them. It's a two-step process that is expensive to set up and can be a bit shaky if the student doesn't copy perfectly.

The New Shortcut: STP

The authors of this paper, Guanquan Wang and Yoshimasa Tsuruoka, propose a new method called Shortcut Trajectory Planning (STP).

Instead of the slow artist or the two-step teacher-student system, they suggest using a "Shortcut Model." Imagine a superhero who can look at a messy scribble and, in a single leap, jump straight to the finished drawing. Or better yet, imagine a video game character who can choose to take one giant leap or a few smaller hops to reach the destination, all using the same set of powers.

The Main Finding:
The paper suggests that STP can generate these perfect robot paths just as well as the complex teacher-student systems, but with a much simpler setup. They trained the model in just one stage (no teacher needed). When they tested it on standard robot challenges (like the D4RL benchmarks, which include walking, navigating mazes, and moving objects), STP performed very strongly.

In walking tasks, it averaged a score of 73.9, slightly beating the previous best shortcut method (CTP) which scored 73.3.
In maze navigation, it reached an average score of 183.8, outperforming almost every other method.
In complex hand-manipulation tasks (like moving a pen), it achieved an average of 114.3, the highest among the methods they compared.

What They Argue Against

The paper explicitly argues against the idea that you need a two-stage teacher-student pipeline to get fast, high-quality planning. They show that the "distillation" process (training a teacher, then a student) adds unnecessary cost and instability. They also argue that while older diffusion methods are powerful, their slow, step-by-step denoising is too expensive for real-time control. STP suggests you can have the speed of the shortcut without the complexity of the two-step training.

How They Made It Work (The Secret Sauce)

To make sure the robot doesn't just pick a path that looks good on paper but crashes into a wall in real life, the authors added two clever tricks:

The "Warm-Start" Strategy: Instead of starting from scratch (a blank, noisy canvas) every time the robot needs to make a new move, STP takes the path it just drew and tweaks it slightly for the next step. It's like a hiker who doesn't stop and re-plan the whole trip from the beginning every time they take a step; they just adjust their current direction. This made the robot's movements much smoother. In the maze tests, using this trick boosted the average score from 150.8 to 183.8.
The "Feasibility Penalty": Sometimes, the robot's "critic" (the part that judges how good a path is) gets too excited about a high score and picks a path that goes through a wall because it looks like a shortcut. The authors added a "feasibility penalty" that acts like a reality check. If a path hits a wall, it gets a big point deduction. This was especially helpful in the hardest maze (Maze2D Large), where the score jumped from 181.9 to 215.1 when they turned this penalty on.

How Sure Are They?

The authors are confident in their results based on simulations and experiments on standard datasets. They ran their method on 150 different random seeds (essentially trying 150 different starting conditions) to make sure the results weren't just luck. They found that STP consistently performed well across walking, mazes, and manipulation tasks.

However, they don't claim this is a magic bullet that solves every problem in robotics. They note that while STP is great, other methods (like Diffusion-QL) still win on some specific tasks because they use different math to optimize values. But for the specific goal of fast, single-stage planning, the paper suggests STP is a highly effective and practical alternative.

In short, the paper suggests that by using a "shortcut" model that learns in one go and checks its own work for reality, we can build robots that plan faster and smarter without needing a complicated two-step training school.

Technical Summary: Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning

Problem Statement

Offline reinforcement learning (RL) has increasingly relied on diffusion-based trajectory planners to model behavior distributions and perform decision-making via sampling. While these methods demonstrate strong performance, their iterative denoising processes incur high inference costs, rendering them computationally expensive for real-time or high-frequency control. Recent alternatives based on consistency models (e.g., Consistency Planning, Consistency Trajectory Planning) reduce inference steps by replacing multi-step diffusion with consistency-based generation. However, these methods typically require a two-stage teacher–student distillation pipeline. This approach increases total training costs and introduces instability, as the final planner's performance depends on the successful training of a teacher model and the subsequent successful distillation into a student model. In offline RL, where learning is already challenged by multimodal trajectory distributions and distributional shift, such instability is particularly undesirable.

Methodology

The paper proposes Shortcut Trajectory Planning (STP), a framework that integrates shortcut models (Frans et al., 2025) as efficient trajectory generators for offline model-based RL. Unlike consistency-based planners that rely on distillation, STP trains a conditional shortcut trajectory model in a single stage.

Core Components

Shortcut Trajectory Model:
- Architecture: The planner is parameterized as a conditional shortcut model $S_\theta(x_t, t, d | s_k)$ , where $x_t$ is a noisy future-state sequence, $t$ is the noise level, $d$ is the step size, and $s_k$ is the current state.
- Training Objective: The model is trained to predict finite-step updates along a generative trajectory. The training loss combines a flow-matching term (grounding the model at the finest scale) and a recursive self-consistency term. The self-consistency constraint ensures that a shortcut of size $2d$ matches the composition of two successive shortcuts of size $d$ . This allows the model to support one-step, few-step, and multi-step generation within a single network without a teacher model.
- Trajectory Representation: The model uses a "jumping-step" representation, modeling a subsequence of future states $(s_k, s_{k+M}, \dots, s_{k+(H-1)M})$ where $M$ is the planning stride. This allows for longer effective horizons with compact generated sequences.
Critic and Feasibility Correction:
- A critic network $V_\alpha$ is trained to predict the accumulated discounted return of denoised trajectories.
- To address the issue that critics may assign high scores to physically infeasible trajectories (e.g., those intersecting obstacles), STP introduces a feasibility-aware correction. The final plan selection score is adjusted by subtracting a feasibility penalty $\Psi$ that measures constraint violations (e.g., collision with walls).
Inference Process:
- Sampling: The planner generates candidate trajectories by iteratively applying the shortcut model under a prescribed sampling budget (step size $d = 1/K$ ).
- Warm-Start Strategy: For subsequent environmental timesteps, the planner does not restart from pure noise. Instead, it initializes the next trajectory by perturbing the denoised trajectory from the previous step. This improves temporal coherence and reduces redundant computation.
- Action Extraction: The highest-scoring trajectory (after feasibility correction) is selected. The first stride transition $(s_k, s_{k+M})$ is passed to a stride-based inverse dynamics model to recover the executable action $a_k$ .

Key Contributions

Single-Stage Training: STP eliminates the need for the two-stage teacher–student distillation pipeline required by consistency-based planners, simplifying the training process and reducing instability.
Adjustable Inference: By conditioning on step size, the shortcut model supports flexible inference budgets (one-step to multi-step) within a single network, balancing planning speed and trajectory quality.
Feasibility-Aware Planning: The integration of a feasibility penalty into the plan selection process prevents the selection of high-reward but physically unrealizable trajectories, a common failure mode in generative planning.
Warm-Start Initialization: The adoption of a warm-start strategy enhances temporal consistency and execution smoothness, particularly in long-horizon tasks.

Experimental Results

STP was evaluated on standard D4RL offline RL benchmarks, including locomotion (Gym MuJoCo), navigation (Maze2D, AntMaze), manipulation (Kitchen), and dexterous control (Adroit).

Performance: STP achieves competitive or superior performance compared to diffusion-based (Diffuser, Diffusion-QL), consistency-based (CP, CTP), and other actor-critic baselines.
- Locomotion: STP improved the average score from 73.3 (CTP) to 73.9, outperforming CTP on five of six datasets.
- Navigation: On Maze2D, STP achieved the best average performance (183.8) among all compared methods. On AntMaze, it improved the average score from 83.33 (CTP) to 85.33.
- Manipulation: STP achieved the best average performance on both Kitchen and Adroit benchmarks.
Ablation Studies:
- Warm-Start: Removing the warm-start strategy significantly degraded performance in both Maze2D (average drop from 183.8 to 150.8) and AntMaze (drop from 85.3 to 68.0), confirming its importance for temporal consistency.
- Feasibility Penalty: The penalty yielded substantial gains on complex Maze2D-Large tasks (improving scores from 181.9 to 215.1) by filtering out trajectories that collide with obstacles, while having minimal effect on simpler tasks.

Significance and Claims

The paper claims that Shortcut Trajectory Planning provides a practical alternative to distillation-based generative planners. By replacing the complex teacher–student pipeline with a single-stage shortcut model, STP simplifies the training pipeline while preserving the ability to perform fast, low-step sampling. The results suggest that shortcut models are effective for offline model-based RL, offering a robust balance between training complexity, inference cost, and planning quality. The authors emphasize that the combination of single-stage training, efficient trajectory generation, and robust inference-time selection (via warm-start and feasibility correction) makes shortcut-based planning a promising direction for efficient offline reinforcement learning.