🤖 AI

Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

This paper proposes Style-Conditioned Implicit Q-Learning (SCIQL), a framework that unifies behavior style definition and employs offline goal-conditioned RL techniques with Gated Advantage Weighted Regression to effectively align high task performance with robust style supervision in offline reinforcement learning.

Original authors: Mathieu Petitbois, Rémy Portelas, Sylvain Lamprier

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mathieu Petitbois, Rémy Portelas, Sylvain Lamprier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a video game or run a race. Usually, we just want the robot to win or reach the finish line as fast as possible. But what if you want the robot to do it in a specific way? Maybe you want it to run like a cheetah, or draw a circle with a shaky, artistic hand, or move gracefully like a dancer.

This paper tackles a very specific problem: How do we teach a robot to perform a task with a specific "style" using only old, recorded videos of other robots doing it, without letting the robot practice in the real world?

Here is the breakdown of their solution, SCIQL, explained through everyday analogies.

The Problem: The "Style vs. Skill" Conflict

The authors identify three big headaches in this field:

Defining "Style" is Hard: Is "style" the whole video? Or just a few seconds? If you label a whole 10-minute video as "aggressive," but the robot was calm for 9 minutes and aggressive for 1, the label is confusing. The authors argue that style happens in short bursts (sub-trajectories), like a single dance move or a specific turn.
The "Missing Puzzle Pieces" Problem (Distribution Shift): Imagine you have a cookbook of recipes. You want to learn how to cook "Spicy Italian." But your cookbook only has photos of people eating the food, not the ingredients. If you try to cook it, you might run out of pasta or use the wrong sauce because the book didn’t show you how to get there, only what the final plate looked like. In AI terms, the robot might end up in a situation it has never seen before (like tripping while trying to run fast) and doesn’t know how to recover because that specific "tripped-and-running" moment wasn’t in the training videos.
Style vs. Performance: Often, being stylish makes you slower. A robot that dances while running will be slower than one that just sprints. Previous methods had to choose: be stylish but bad at the task, or be good at the task but boring. The authors wanted both.

The Solution: SCIQL (Style-Conditioned Implicit Q-Learning)

The authors created a new method called SCIQL. Think of it as a smart tutor that uses two main tricks to solve the problems above.

Trick 1: The "Windowed Labeler" (Solving Definition)

Instead of labeling an entire video as "Fast," they use a sliding window. Imagine watching a movie with a magnifying glass that only looks at 5 seconds at a time. They label those 5 seconds: "This part is fast," "This part is slow."

Why it helps: It’s easier to judge style in small chunks. It also makes it cheaper and easier for humans (or simple rules) to create these labels, because you don’t need to watch the whole video to decide.

Trick 2: The "Gated Advantage" (Solving Style vs. Performance)

This is the core innovation. They use a mechanism they call Gated Advantage Weighted Regression (GAWR).

Imagine you are learning to drive.

The "Style" Signal: Your instructor says, "Keep your hands at 10 and 2."
The "Task" Signal: Your instructor says, "Speed up to pass that car."

Sometimes these conflict. If you speed up, you might forget to keep your hands at 10 and 2.

GAWR acts like a smart gatekeeper:

If the robot is doing a good job with the Style (hands at 10 and 2), the gate opens wide, and the robot listens to the Task advice (speed up).
If the robot is failing at the Style (hands are slipping), the gate closes. The robot ignores the "speed up" advice and focuses entirely on fixing its hand position.

This ensures the robot never sacrifices its style to get a quick reward. It only gets "better" at the task when it’s already being stylish.

Trick 3: "Stitching" and "Relabeling" (Solving Missing Pieces)

To handle the "missing puzzle pieces," the method uses a technique from goal-conditioned learning. It’s like editing a video.

If the training data has a clip of "starting to run" and a separate clip of "running fast," but no clip of the transition, the AI learns to stitch them together.
It also uses "hindsight relabeling." If the robot fails to run fast, the AI looks back and says, "Well, you didn't run fast, but you did crawl well. Let’s label this part as 'crawling' so we can learn from it." This helps the robot learn from mistakes and fill in the gaps in the data.

The Results

The authors tested this on several tasks:

Circle2D: Drawing circles with different speeds and styles.
HalfCheetah: A robot cheetah running with different gaits (crawling, normal, upright).
HumEnv: A humanoid robot moving in different ways (standing, lying down, fast, slow).

What they found:

Better Style: SCIQL was much better at sticking to the requested style than previous methods.
Better Performance: Because of the "Gated" mechanism, the robots didn’t just look stylish; they actually performed the task (like running far or drawing accurately) better than methods that forced a trade-off.
Robustness: Even when the data was messy or incomplete, SCIQL could "stitch" together behaviors to maintain the style.

In Simple Terms

Think of SCIQL as a strict but fair dance coach.

Old Methods: Would say, "Just look good!" (Style) or "Just hit the notes!" (Task). The student would either look great but miss the beat, or hit every note but look awkward.
SCIQL: Says, "First, make sure your posture is perfect (Style). Only if your posture is perfect, then I will tell you how to hit the next note faster (Task). If your posture slips, stop worrying about speed and fix your posture."

By using this "gate," the robot learns to be both stylish and skilled, using only old videos as its reference, without needing to practice in the real world.

Technical Summary: Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment

Problem Statement
The paper addresses the challenge of learning style-conditioned policies from fixed offline datasets (Offline Reinforcement Learning) while simultaneously optimizing for high task performance. The authors identify three primary challenges in existing literature:

Style Definition: Existing definitions range from discrete trajectory labels to unsupervised clusters, often suffering from poor interpretability, high labeling costs, or credit assignment issues due to the multi-timescale nature of behaviors.
State-Style Distribution Shift: Offline methods suffer from distribution shift. Adding style conditioning exacerbates this, as agents may encounter out-of-distribution state-style configurations during inference, lacking the ability to recalibrate without environment interaction.
Task-Style Misalignment: Optimizing for task performance (e.g., speed) often conflicts with style alignment (e.g., specific gait). Previous methods either compromise on style alignment to improve performance or fail to robustly align style while optimizing tasks.

Methodology
The authors propose Style-Conditioned Implicit Q-Learning (SCIQL), a framework that generalizes Goal-Conditioned Reinforcement Learning (GCRL) to style alignment.

Style Definition via Data Programming: To address interpretability and labeling costs, the authors define style using explicit labeling functions $\lambda$ applied to subtrajectory windows rather than full trajectories. This captures multi-timescale behaviors and mitigates credit assignment issues. The dataset $\mathcal{D}$ is annotated such that each state-action pair $(s_t, a_t)$ is labeled based on the style of the surrounding window.
Surrogate Markov Style Objective: Direct optimization of style alignment is non-Markovian. The authors construct a surrogate Markovian reward $r^\lambda_D(s, a, z) = p^{\pi_D}_\lambda(z | s, a)$ , estimated from the annotated dataset. This transforms the style alignment problem into a standard offline RL optimization problem.
SCIQL Algorithm: Building on Implicit Q-Learning (IQL), SCIQL learns style-conditioned value functions $V^\lambda$ and $Q^\lambda$ using expectile regression. Crucially, it employs style relabeling (sampling style labels $z$ independently of the current state-action pair during training) and trajectory stitching to handle distribution shift and enable long-term decision-making for styles not present in single continuous trajectories.
Gated Advantage Weighted Regression (GAWR): To jointly optimize task performance and style alignment, the authors introduce GAWR. Instead of simply summing task and style advantages, GAWR uses the style advantage $A^\lambda$ as a gate for the task advantage $A^r$ . The gated advantage is defined as:
$\xi(A^\lambda, A^r)(s, a, z) = A^\lambda(s, a, z) + \sigma(A^\lambda(s, a, z)) \cdot A^r(s, a)$
where $\sigma$ is the sigmoid function. This mechanism ensures that task performance improvements are prioritized only when they do not significantly harm style alignment (i.e., when style advantage is positive).

Key Contributions

A general definition of stylized policy learning as a generalization of GCRL.
A supervised data-programming framework using subtrajectory labeling functions to define interpretable styles with low labeling costs.
The SCIQL algorithm, which leverages offline goal-conditioned RL techniques (hindsight relabeling, value learning) for robust style alignment.
The GAWR mechanism, which uses advantage functions as gates to balance style alignment and task performance optimization.
Empirical demonstration that SCIQL outperforms prior offline methods in both style alignment and style-conditioned task performance.

Results
Experiments were conducted on three environments: Circle2d (drawing circles with various styles), HalfCheetah (locomotion with varied gaits/speeds), and HumEnv (humanoid control with varied postures/speeds).

Style Alignment: SCIQL achieved the highest style alignment scores across all datasets compared to baselines including Behavior Cloning (BC), Conditioned BC (CBC), BCPMI, and SORL. For instance, on halfcheetah-vary-v0, SCIQL achieved 78.9% alignment versus 31.7% for SCBC and 46.7% for SORL.
Task Performance: SCIQL demonstrated superior joint performance. Using hypervolume (HV) metrics to evaluate the Pareto frontier between style alignment and task performance, SCIQL showed improvements of +41% to +163% over SORL across different environments.
GAWR Efficacy: The inclusion of GAWR allowed SCIQL to increase task performance scores while maintaining high style alignment, whereas methods prioritizing task performance alone suffered significant drops in alignment.

Significance
The paper claims to provide a practical framework for offline RL that achieves robust style alignment without requiring further environment interactions. By framing style alignment as a goal-conditioned problem and using GAWR to manage the conflict between style and task objectives, SCIQL offers a more controllable and high-performing alternative to existing diverse policy learning methods. The authors highlight the method's ability to handle long-term decision-making and trajectory stitching, which are critical for complex style behaviors.