🤖 machine learning

Foundations of Reinforcement Learning and Control:Connections and New Perspectives

This tutorial bridges the gap between reinforcement learning and control theory by introducing adaptive control and actor-critic algorithms, demonstrating their combined application to a locomotion control problem to foster mutual understanding and data-driven decision-making between the two communities.

Original authors: Claire Vernade, Onno Eberhard, Martha White, Florian Dörfler, Csaba Szepesvári, Miroslav Krstic, Michael Muehlebach

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Claire Vernade, Onno Eberhard, Martha White, Florian Dörfler, Csaba Szepesvári, Miroslav Krstic, Michael Muehlebach

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot dog how to run across a field. You have two very different teachers for this job. The first teacher is a Control Theorist. They are like a meticulous engineer who knows exactly how the dog's legs are built. They build a safety harness and a set of rules to make sure the dog doesn't trip, even if the ground gets muddy or a leg gets a little stiff. Their main goal is safety and stability; they want the dog to stand still or walk in a straight line without falling over, no matter what. The second teacher is a Reinforcement Learning (RL) Expert. They are like a playful coach who doesn't care about the dog's anatomy. Instead, they just throw the dog into the field and say, "Run! If you run fast, you get a treat. If you fall, you get nothing." Through millions of tries, the dog learns to run incredibly fast by figuring out what works through trial and error.

For decades, these two teachers worked in separate schools. The engineers worried that the "playful coach" was too dangerous and might break the robot. The "playful coach" thought the engineers were too rigid and couldn't teach the robot to do anything fancy. But now, both fields are realizing they need each other. The robots of the future need to be fast and smart, but they also need to be safe and not fall apart when the world changes. This paper is a friendly guidebook that explains how to introduce these two teachers to each other, showing how their different styles can actually work together to make a robot that is both a champion runner and a safe traveler.

The Paper's Big Idea: Mixing the Safety Harness with the Playful Coach

This paper, written by a team of experts from universities and research institutes in Germany, Canada, and the US, acts as a bridge between the worlds of Control Theory and Reinforcement Learning. The authors argue that while these fields have different histories and languages, they are actually trying to solve the same problem: how to make a machine move and act intelligently when we don't know everything about how it works.

To explain this, the authors use a running example: a simulated robot called the Half-Cheetah. Imagine a robotic cheetah with six joints (like knees and hips) that needs to run as far and as fast as possible.

The Two Approaches
The paper first breaks down how each field usually tackles the Half-Cheetah:

The Control Theory Way (Model-Reference Adaptive Control or MRAC):
Think of this as giving the robot a "target gait." The engineer says, "Your left leg should move like this, and your right leg like that." They build a "reference model"—a perfect, imaginary version of how the robot should move. If the real robot starts to wobble because its joints get slippery or heavy, the MRAC system acts like a smart autopilot. It constantly checks the difference between the real robot and the perfect model, then tweaks the motors in real-time to fix it. It's great at keeping the robot stable and tracking a specific path, even if the robot's weight changes or the floor gets slippery. However, it is designed for low-level adaptation, meaning it excels at adjusting forces to hit a target, but it does not invent high-level strategies or complex running styles on its own.
The Reinforcement Learning Way (Actor-Critic Algorithms):
This is the "playful coach" approach. Here, the robot (the "actor") tries to run, and a "critic" watches and says, "Good job!" or "That was slow." The robot learns from scratch, without any pre-written rules about how to move its legs. It discovers complex, high-speed running styles that a human engineer might never have thought of. In the paper's simulations, a famous algorithm called Soft Actor-Critic (SAC) learned to make the Half-Cheetah run very fast. But there's a catch: if the robot's environment changes suddenly (like if the friction in its joints drops by half), the SAC robot gets confused and starts to stumble because it learned a specific way to run that only worked on the old, "sticky" floor.

The New Hybrid Solution
The paper's main contribution is a new way to combine these two teachers. The authors suggest using Reinforcement Learning for the "High-Level" thinking and Adaptive Control for the "Low-Level" muscle control.

Here is how it works in their experiment:

The High-Level Brain (RL): The SAC algorithm learns a strategy. Instead of telling the robot exactly how much torque (force) to apply to the motors, it learns to pick a set of desired angles for the joints. It's like the coach saying, "I want your knee bent at 45 degrees," rather than "Push the motor with 5 Newtons of force."
The Low-Level Muscles (MRAC): Once the brain decides on the desired angle, the MRAC system takes over. It acts as the muscle memory. It figures out exactly how much force is needed to get the joint to that angle right now, even if the joint is slippery or heavy. If the friction changes, the MRAC system instantly adjusts the force to make sure the joint still hits the target angle.

What They Found
In their simulations, the authors tested this hybrid system on the Half-Cheetah.

First, they trained the SAC robot to run on a normal surface. It learned a great running style.
Then, they deployed the learned policy into the environment. After the robot had been running for 1.2 million steps, they suddenly changed the physics: they made the joints twice as slippery (reducing the damping coefficient by a factor of 2).
The Result: The pure RL robot (SAC alone) crashed and failed miserably because it didn't know how to handle the slippery joints. The pure adaptive controller (MRAC alone) could track a simple path but couldn't run fast or complexly.
The Hybrid Winner: The combined system (SAC + MRAC) kept running smoothly. The "brain" kept choosing the same good running angles, and the "muscles" (MRAC) automatically adjusted the forces to handle the new slippery conditions. The robot didn't crash; it just kept running.

Why This Matters
The paper suggests that we don't have to choose between "safe, rigid engineering" and "smart, flexible learning." By layering them, we can get the best of both worlds. The RL part can learn complex, high-level strategies (like how to run fast or climb a hill), while the adaptive control part handles the messy, changing reality of the physical world (like wind, slippery floors, or broken parts).

The authors are careful to note that this is a simulation on a computer model of a robot, not a physical robot running in a real lab yet. They also point out that while this hybrid approach worked well in their specific test, it's not a magic bullet for every problem. For instance, the RL part still needs to learn the strategy first, and the adaptive part needs a good reference model to track.

Ultimately, the paper is a call to action for scientists in both fields to stop working in silos. It shows that by understanding each other's tools—like how "Lyapunov functions" (a math way to prove stability) relate to "value functions" (a math way to measure success)—we can build robots that are not only smart and fast but also robust enough to handle the unpredictable real world. The future of robotics, they argue, lies in this collaboration.

Technical Summary: Foundations of Reinforcement Learning and Control

Problem Definition

The paper addresses the persistent gap between two adjacent scientific fields: Reinforcement Learning (RL) and Control Theory. Both fields aim to optimize controllers for unknown dynamical systems using feedback, sharing a common root in dynamic programming. However, they have evolved with distinct methodologies, goals, and cultures.

Control Theory historically prioritizes stability, safety, and robustness in physical ("in-vivo") systems, often relying on formal mathematical certificates (e.g., Lyapunov functions) and gray-box models with parametric uncertainty.
Reinforcement Learning has largely developed within computer science ("in-silico" simulators), prioritizing algorithmic scalability, data-driven optimization, and performance on high-dimensional benchmarks, often using black-box, model-free approaches.

The core problem is to bridge these perspectives to create unified, data-driven decision-making frameworks that leverage the complementary strengths of both fields: the stability guarantees of control and the scalability of RL.

Methodology

The tutorial employs a structured comparative approach, using a Half-Cheetah locomotion task (simulated in MuJoCo) as a running example to illustrate concepts.

1. Unified Foundations

The authors establish a unified notation for dynamical systems, defining:

State Space ( $S$ ) and Action Space ( $A$ ).
Dynamics: Contrasting the RL view (stochastic Markov chains, $s_{t+1} \sim p(\cdot|s_t, a_t)$ ) with the Control view (deterministic dynamics with additive noise, $s_{t+1} = f(s_t, a_t) + \eta_t$ ).
Stability vs. Optimality: The paper highlights that while RL seeks to maximize cumulative rewards (optimality), control seeks to ensure trajectories converge to a target (stability). It notes that in Linear Quadratic Regulator (LQR) settings, the optimal cost-to-go function coincides with a Lyapunov function, linking the two concepts.

2. Representative Algorithms

The paper details two primary methodologies:

Model-Reference Adaptive Control (MRAC): A control-theoretic approach where a controller adjusts its parameters online to make a physical system track a stable "reference model." It relies on a "matching assumption" (that ideal gains exist within the parameterization) and uses Lyapunov-based update laws to ensure bounded tracking error.
Actor-Critic RL (specifically Soft Actor-Critic - SAC): A model-free RL approach where an "actor" (policy) selects actions and a "critic" (value function) evaluates them. The algorithm uses policy gradient theorems and temporal difference (TD) updates to learn optimal behavior directly from interaction data without an explicit system model.

3. Hybrid Approach: RL + MRAC

A key methodological contribution is the proposal of a low-level/high-level hybrid architecture:

High-Level (RL): An SAC policy learns a complex locomotion strategy (e.g., desired joint angles) in a static environment.
Low-Level (MRAC): An adaptive controller takes the high-level output as a reference signal and adjusts low-level control gains (torques) to track these angles, compensating for real-time changes in system dynamics (e.g., friction changes).
Implementation: The authors demonstrate that the MRAC component can be initialized using data from the static environment to define a reference model, then adapt online when dynamics shift, effectively restoring the nominal dynamics for the fixed SAC policy.

Key Contributions

Conceptual Unification: The paper provides a concise technical introduction that aligns the terminology and problem formulations of RL and control, clarifying that their differences often stem from environmental assumptions (simulator vs. physical plant) rather than fundamental problem definitions.
Methodological Bridge: It explicitly connects Lyapunov functions (control) with Value functions (RL), showing how stability certificates can be viewed through the lens of dynamic programming.
Hybrid Architecture Demonstration: The authors present a novel combination of actor-critic algorithms and MRAC. This approach separates high-level planning (learning task structure) from low-level adaptation (handling parametric uncertainty), demonstrating that the two paradigms are complementary rather than opposed.
Critical Analysis of Limitations: The paper honestly assesses the limitations of both fields:
- Control: Often struggles with highly nonlinear, unmodeled dynamics and requires complex manual construction of Lyapunov functions.
- RL: Often lacks formal stability guarantees, struggles with transient performance during learning, and relies on heuristics that can diverge under nonlinear function approximation.

Results

Using the Half-Cheetah benchmark, the authors demonstrate:

Pure SAC: Successfully learns a running gait in a static environment but fails catastrophically when joint friction is suddenly reduced (dynamics change), as the fixed policy does not account for the new physics.
SAC with Fixed Low-Level Controller: Shows marginal improvement over pure SAC after the dynamics change but still struggles to maintain performance.
SAC + MRAC (Proposed Hybrid): The adaptive low-level controller successfully tunes its parameters to compensate for the friction change. This restores the nominal low-level dynamics, allowing the fixed high-level SAC policy to maintain high performance despite the environmental shift. The results illustrate that the hybrid approach achieves both high performance (via RL) and continual adaptation (via MRAC).

Significance and Claims

The paper claims to provide a foundation for understanding the core differences between RL and control, offering insights to help experts in each field engage with the tools of the other.

Convergence: The authors argue that the time is ripe for convergence, driven by advances in data availability and computational infrastructure. Modern control is increasingly data-centric, while RL is moving toward physical systems requiring safety guarantees.
Complementarity: The primary significance lies in demonstrating that RL and adaptive control are not mutually exclusive. The hybrid approach shows that RL can handle high-level decision-making and task learning, while adaptive control can provide the necessary robustness and stability for low-level execution in uncertain environments.
Future Directions: The paper identifies open problems where synergies are needed, including long-horizon planning, hierarchical architectures, partial observability, and robust RL that ensures safety under distribution shifts. It emphasizes that for real-world deployment, safety (hard constraint satisfaction) is non-negotiable, requiring a careful interplay between feedforward (planning) and feedback (adaptation) mechanisms.

The paper concludes modestly, stating that while the hybrid example is illustrative, the broader theme is the potential for a unified framework for data-driven decision making that balances performance with reliability.