🤖 machine learning

Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models

This paper introduces LDM-v0, a large-scale, multi-task transformer policy trained offline on diverse trajectories from thousands of environments, demonstrating that a single unified model can match the performance of specialized policies across domains ranging from robotics to cybersecurity.

Original authors: Thibaut Kulak

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thibaut Kulak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do everything: play video games, drive a car, manage a warehouse, and even control a robotic arm. Traditionally, you would have to hire a different expert coach for each job. The video game coach knows nothing about driving, and the warehouse manager has no idea how to play Super Mario. This is slow, expensive, and requires a lot of custom tuning for every single task.

This paper introduces LDM-v0, a new approach that tries to solve this by training one single "super-coach" that can learn to do all these different jobs at once.

Here is a breakdown of how they did it, using simple analogies:

1. The Problem: Too Many Different "Languages"

Reinforcement Learning (RL) is like teaching a student by letting them try things and giving them a score (reward) when they do well. The problem is that every environment (like a video game or a robot simulator) speaks a different "language."

One environment might use numbers to describe a robot's arm.
Another might use pictures.
Another might use text.
The rules for what counts as a "good move" are totally different in each one.

Trying to train one model on all of these at once is like trying to teach a student to speak French, Japanese, and Morse code simultaneously while they are also learning to juggle.

2. The Solution: The "Universal Translator" (LDM-v0)

The researchers built a massive model called LDM-v0 (Large Decision Model). Think of this model as a universal translator that converts all those different "languages" into a single, common format that the computer can understand.

The Input: Instead of feeding the model raw data, they translate everything into "tokens" (like words in a sentence). Whether it's a picture, a number, or a robot's position, it gets turned into a standardized token.
The Brain: They used a specific type of AI architecture (based on "Llama," similar to the ones used for chatbots) that is really good at remembering long stories. In this case, the "story" is the history of what the robot saw, what it did, and what score it got.

3. How They Trained It: The "Shadowing" Method

You can't just throw the model into a room with 1,000 different games and hope it figures it out. It needs a teacher. But since there is no human who knows how to play all these games perfectly, the researchers used a clever trick called Automated Reference-Policy Supervision.

Imagine you want to learn to play chess, soccer, and poker. Instead of a human teaching you, you hire 1,000 different expert bots.

Hire the Experts: They trained thousands of specialized AI agents (using standard algorithms like PPO or DQN) to master specific environments.
Record the Masters: They recorded the "perfect" moves these expert bots made.
The Shadowing Game: They then fed these recordings to LDM-v0. The model's job wasn't to "think" or "explore"; its job was simply to mimic the expert bots. It looked at the history of a game and predicted, "What would the expert do next?"

By doing this, LDM-v0 learned to imitate the best strategies across 1,000 different environments without needing to be told the rules of each one explicitly.

4. The Results: One Model to Rule Them All?

The team tested this single model on about 1,000 different environments, ranging from:

Robotics: Controlling robotic arms and drones.
Driving: Simulating cars on highways.
Business: Managing inventory and stock trading.
Gaming: Playing classic arcade games and Super Mario.

The Big Win:
The single LDM-v0 model performed just as well as the specialized "expert" bots on about 1,000 of those environments. In other words, one generalist model could do the job of 1,000 different specialists.

They also found that making the model bigger (giving it more "brain power") generally made it better, up to a point.

5. What This Means (and What It Doesn't)

What it proves: It is possible to train one giant AI model on a huge mix of totally different tasks and have it perform well on all of them. It suggests that these different tasks share hidden patterns that the model can learn.
What it doesn't claim: The paper does not say this model is ready to drive your car tomorrow or manage your actual stock portfolio. The tests were done in simulations (video games and digital twins).
The Limitation: The model is very good at tasks it has seen before (or very similar ones). The authors admit they haven't fully tested if it can handle a brand new type of task it has never seen before. It's a master of imitation, not necessarily a master of invention yet.

Summary Analogy

Think of previous AI models as specialized apprentices: a carpenter who can only build chairs, a plumber who can only fix sinks.
LDM-v0 is like a super-apprentice who has watched thousands of videos of carpenters, plumbers, electricians, and chefs. It hasn't built a house or cooked a meal itself yet, but if you show it a new situation, it can instantly recall, "Oh, the expert plumber would do this," and do it correctly.

The paper shows that this "super-apprentice" approach works surprisingly well, paving the way for future AI systems that can learn many skills from a single, massive training session.

Technical Summary: Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models

Problem Statement

Reinforcement Learning (RL) has achieved significant success in specific domains such as games, robotics, and control. However, applying RL to real-world settings remains challenging due to the need for extensive environment interaction, careful reward engineering, domain-specific architectures, and substantial hyperparameter tuning. While offline RL and offline-to-online approaches mitigate the cost of online interaction, selecting and adapting a suitable algorithm for a new environment remains difficult.

Parallel to these challenges, large-scale sequence models have transformed natural language processing and computer vision. In RL, sequence-modeling approaches like Decision Transformers have demonstrated that policies can be represented as autoregressive models over trajectories. This raises a critical question: Can multi-domain offline RL trajectories be consolidated into a single scalable transformer policy while maintaining strong task performance across many domains? The primary obstacle is that multi-domain RL ecosystems are fragmented, heterogeneous, and difficult to unify.

Methodology

The authors introduce LDM-v0 (Large Decision Model v0), a unified multi-task, multi-modal transformer policy trained offline on trajectories from thousands of heterogeneous environments. The methodology consists of three core components:

1. Automated Reference-Policy Supervision

To generate high-quality training data across diverse domains without manual reward engineering, the authors treat dataset construction as an AutoRL problem:

Candidate Selection: A pool of candidate algorithms and configurations (from Stable-Baselines3 and SB3-Contrib, including A2C, PPO, SAC, DQN, etc.) is defined.
Ranking: For each environment library, compatible candidates are trained on a subset of five environments. Configurations are ranked using pairwise comparisons based on a one-tailed Welch's t-test of final performance.
Data Generation: The top-ranked configurations are trained for 3 million transitions. Trajectories are recorded, and the final trained policy is replayed to generate reference action annotations.
Filtering: Low-quality runs (those without statistically significant performance improvement or those falling below 95% of the best policy's performance) are removed.
Result: A dataset of approximately 9.3 billion transitions from ~3,000 distinct environments, annotated with reference policy actions rather than exploratory actions.

2. Unified Environment Infrastructure

To handle the fragmentation of RL libraries (different Python versions, dependencies, and simulators), the authors developed an internal orchestration framework:

Environments are encapsulated in isolated Docker containers.
A unified interaction API compatible with modern Gymnasium interfaces is exposed.
This pipeline integrated 146 environment libraries covering approximately 15,000 individual environments.

3. Model Architecture and Training

Architecture: LDM-v0 is a decoder-only transformer based on the Llama architecture, trained from scratch.
Input Processing: The model receives an interaction history (past observations, actions, rewards, termination signals) and the current observation.
- Tokenization: Continuous inputs are $\mu$ -law encoded and discretized into 1024 bins. Discrete inputs use lookup tables. Images are resized to 64x64 and encoded via a convolutional encoder.
- Embedding: Observation, action, reward, and termination signals are aligned and stacked into transition-level meta-tokens. This compact representation reduces sequence length compared to per-dimension tokenization (e.g., Gato), allowing for longer histories within a fixed context budget.
Training Objective: The model is trained via supervised next-action prediction using cross-entropy loss over discretized action bins. It learns to predict the reference action $a_t$ conditioned on the trajectory history $(o_i, a_i, r_i, d_i)$ .
Scale: The primary model (LDM-v0) has 308M parameters, 12 layers, and a context length of 2048 transitions. It was trained for six days on two nodes with 16 NVIDIA H200 GPUs.

Key Contributions

LDM-v0: A large-scale, multi-modal transformer policy trained on a unified dataset spanning 146 libraries and ~3,000 environments.
Scalable Infrastructure: A robust pipeline for automating reference-policy generation and environment orchestration across highly heterogeneous RL ecosystems.
Compact Transition-Level Modeling: A tokenization strategy that packs observation, action, reward, and termination signals into single meta-tokens, enabling efficient long-horizon sequence modeling.
Empirical Validation: Evidence that a single shared parameterization can achieve competitive performance across diverse domains without task-specific fine-tuning.

Results

The authors evaluated LDM-v0 across the training environments (in-distribution) and compared its performance against independently trained, task-specific reference policies.

Performance Thresholds: LDM-v0 achieved >80% of the reference policy performance on over 1,600 environments and matched reference performance on approximately 1,000 environments.
Domain Coverage: Strong performance was observed across:
- Robotics and control (e.g., gymnasium_robotics, panda-gym).
- Autonomous driving (highway-env).
- Financial trading (gym_trading_env) and inventory optimization.
- Cybersecurity (cymnasium).
- Game-like environments (Atari-style, procgen, Super Mario Bros).
Scaling Laws: Experiments with model sizes ranging from 32M to 736M parameters showed substantial performance improvements from 32M to 308M. Gains appeared to plateau between 308M and 736M, suggesting that while larger capacity helps, the 308M model captures significant statistical regularities.

Significance and Claims

The paper claims to demonstrate the feasibility of large-scale offline pretraining across heterogeneous reinforcement learning environments using a single transformer policy.

Unified Representation: The results suggest that despite substantial heterogeneity in observation modalities, action spaces, reward scales, and temporal horizons, large sequence models can leverage statistical regularities shared across subsets of environments.
Automation: The work highlights the potential of automated, unified pipelines to overcome the fragmentation of current RL ecosystems, reducing the reliance on expert knowledge and task-specific design choices.
Future Direction: The authors view LDM-v0 as a step toward scalable, pretrained reinforcement learning systems. They explicitly note that their primary objective was not to study out-of-distribution generalization, but to investigate whether a single policy can jointly model diverse RL behaviors at scale.

The authors remain modest regarding generalization, acknowledging that evaluations were primarily conducted on environments within the training distribution. They identify understanding transferable decision-making strategies and improving generalization to unseen environments (via in-context adaptation or finetuning) as important open questions for future work.