Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning

This paper systematically analyzes how Markov Decision Process design choices impact the sim-to-real gap in industrial process control, demonstrating through a color mixing task that physics-based dynamics models significantly outperform simplified models in achieving real-world success under strict precision constraints.

The Gist

Imagine you are teaching a robot chef to mix the perfect shade of paint. You want the robot to be able to mix a specific color, like "Sunset Orange," by combining Cyan, Magenta, and Yellow inks.

The problem is that you can't let the robot practice on real paint in a real kitchen immediately. It would waste tons of expensive ink and make a mess. So, you teach it in a video game simulation first.

But here's the catch: What works in the game often fails in real life. This is called the "Sim-to-Real Gap." The robot might be a master chef in the game but a disaster in the real kitchen because the simulation isn't perfect.

This paper is like a detective story where the authors try to figure out exactly how to design the robot's "training manual" (the MDP) so that what it learns in the game actually works when it picks up a real pipette.

Here is the breakdown of their findings using simple analogies:

1. The Training Manual (The MDP)

In Reinforcement Learning, the "MDP" is just the set of rules the robot follows. The authors tested different ways to write these rules to see which ones help the robot transfer its skills from the game to reality.

They looked at five main ingredients:

• What the robot sees (State): Does the robot know what color it is trying to make, or just what it has mixed so far?
• The Goal: Is the target color part of the instructions?
• The Scorecard (Reward): How do we tell the robot it's doing a good job?
• The Stop Sign (Termination): When does the robot stop mixing?
• The Physics Engine: How realistic is the simulation of how paint mixes?

2. The Big Discoveries (The "Aha!" Moments)

🎯 The "Target" Must Be Visible

The Analogy: Imagine playing a game of "Hot and Cold" to find a hidden treasure. If you don't tell the player where the treasure is, they will just wander aimlessly.
The Finding: The robot must be told the target color in every single step.

• Without the target: The robot learned a "compromise" strategy. It got good at mixing an average color. In the game, this was okay. But in the real world, when it needed a specific shade, it failed completely because it didn't know what it was aiming for.
• With the target: The robot learned specific strategies for specific goals. This worked perfectly in the real world.

📏 The "Ruler" Matters (State Representation)

The Analogy: Imagine you are baking a cake.

• Absolute Ruler: "Add 200 grams of flour." (This fails if you are making a tiny cupcake vs. a giant cake).
• Relative Ruler: "Add flour until it's 50% of the bowl." (This works for any size cake).
  The Finding: The robot learned much better when it was taught in ratios (percentages) rather than absolute amounts (microliters). Real-world machines have slight variations; ratios are more flexible and robust against those small errors.

🏆 The Scorecard (Reward Function)

The Analogy:

• Simple Score: "You get points for getting closer to the target color."
• Complex Score: "You get points for getting closer, but you lose points if you pour too much ink or pick the wrong bottle."
  The Finding: The simple score worked best. The complex score made the robot "overthink" and memorize the specific quirks of the simulation, causing it to fail when the real-world physics were slightly different. Keep it simple!

🧪 The Physics Engine (Dynamics)

The Analogy:

• Toy Physics: Mixing paint is like mixing water. You just add the amounts together. (Easy to calculate, but fake).
• Real Physics: Mixing paint is like mixing light and dust. It's messy, absorbs light, and scatters. (Hard to calculate, but real).
  The Finding:
• If you use Toy Physics, the robot learns fast but fails in the real world.
• If you use Real Physics (like the Kubelka-Munk model), the robot learns slower and struggles more in the game. BUT, when you put it in the real world, it succeeds 50% of the time, whereas the Toy Physics robot fails 100% of the time.
• Lesson: It's better to train on a hard, realistic simulation than an easy, fake one.

3. The "Strictness" Trap

The authors also tested how strict the rules should be.

• Loose Rules: "Stop when you are kind of close." -> The robot learns fast but is sloppy.
• Strict Rules: "Stop only when you are perfectly close." -> The robot learns slower and fails more in the game.
• The Twist: If you use Real Physics, the strict rules actually help! The robot learns to be precise. If you use Toy Physics, strict rules just break the robot.

The Bottom Line

To teach a robot to do a real-world job (like mixing medicine or paint):

1. Show it the goal every single time.
2. Teach it ratios, not just raw numbers.
3. Give it a simple scorecard (don't overcomplicate the rewards).
4. Use a realistic physics engine, even if it makes training harder.

By fixing the "training manual" (the MDP design), they turned a robot that failed completely in the real world into one that could successfully mix precise colors, bridging the gap between the video game and reality.

Technical Summary

Here is a detailed technical summary of the paper "Impact of Markov Decision Process Design on Sim-to-Real Reinforcement Learning" by Krau et al.

1. Problem Statement

Reinforcement Learning (RL) shows promise for industrial process control, but policies trained in simulation often fail when deployed on physical hardware due to the sim-to-real gap. This gap arises from discrepancies between simulated and real-world environments (e.g., lighting, material properties, sensor noise).

• Current Limitations: Most existing strategies to bridge this gap focus solely on transition dynamics (e.g., domain randomization, system identification) while keeping other Markov Decision Process (MDP) components (states, rewards, termination criteria) fixed.
• Research Gap: There is a lack of systematic analysis on how core MDP design choices—specifically state composition, target inclusion, reward formulation, and dynamics models—affect transferability to physical systems.
• Motivation: The work is motivated by industrial applications like CAR-T cell therapy, where precise fluid mixing is critical. The authors use a color mixing task as a controlled, reproducible physical testbed to study these effects.

2. Methodology

The authors conducted a systematic, phase-wise empirical study using a color mixing task where an agent mixes Cyan, Magenta, and Yellow inks to match a target color.

A. Experimental Setup

• Task: An episodic RL task where the agent adds ink volumes to reach a target RGB color within a tolerance $\tau$.
• Robustness Mechanisms: To simulate real-world noise, the system includes channel-wise observation noise and a lightweight adversarial perturbation scheme (injecting worst-case perturbations with 80% probability).
• Algorithm: Proximal Policy Optimization (PPO) was used for training.
• Evaluation Metrics:
  • Simulation: Final Performance (FP), Time-to-7.5 (sample efficiency), Coefficient of Variation (stability), and Non-Monotonicity.
  • Real-world: RGB distance, Steps-to-target, and Success rate (hitting tolerance $\tau$).

B. Three-Phase Optimization Strategy

The study isolated specific MDP components to determine their impact:

• Phase 1: Component Selection

  • Target Inclusion: Tested whether including the target color ($c_{target}$) in the state observation is necessary.
  • State Representation: Compared 5 volume encoding schemes (absolute vs. relative/normalized ratios).
  • Reward Function: Compared simple distance-based rewards ($R1$) vs. complex action-penalized rewards ($R2, R3$).
  • Hypothesis: Including the target and using relative state representations improves generalization; simple rewards prevent overfitting.

• Phase 2: Episode Design Optimization

  • Tested Termination Horizon ($T$) and Tolerance ($\tau$).
  • Hypothesis: Stricter training thresholds promote precision but may reduce success rates; relaxed thresholds accommodate noise.

• Phase 3: Dynamics Robustness

  • Tested three color prediction models with increasing physical realism:
    1. Linear Interpolation (Lerp): Simple, unrealistic.
    2. Kubelka-Munk (KM): Physics-based, accounts for absorption/scattering.
    3. Weighted Geometric Mean (WGM): Spectral mixing model.
  • Hypothesis: Physics-based models, while harder to learn, yield better real-world transfer.

3. Key Contributions

1. Systematic MDP Analysis: The first study to isolate and evaluate the specific impact of MDP design choices (state, reward, termination, dynamics) on sim-to-real transfer in a hardware-deployed setting.
2. Empirical Quantification: Provided concrete data showing that target state inclusion is critical; omitting the target transforms the MDP into a Partially Observable MDP (POMDP), causing policies to learn a "compromise" behavior that fails in the real world.
3. Design Guidelines: Identified specific patterns that improve transferability:
   • State: Normalized ratio-based representations generalize better than absolute volumes.
   • Reward: Simple distance-based rewards outperform complex, action-penalized rewards.
   • Dynamics: Physics-based models (KM, WGM) are essential for success under strict precision constraints, even if they slow down training.
4. Hardware Validation: Unlike many theoretical papers, this work validated all findings on physical hardware, demonstrating that simplified models fail entirely under strict constraints where physics-based models achieve ~50% success.

4. Key Results

• Target Inclusion (H1): Policies trained without the target color in the state ($M2$) achieved 0% success on hardware, despite moderate simulation performance. Policies with the target ($M1$) achieved 43.75% success. This confirms that goal-conditioned policies are necessary to adapt to real-world dynamics.
• State & Reward (H2 & H3): The combination of normalized ratio states (State 4) and simple Euclidean distance rewards ($R1$) provided the most stable training and best transfer. Complex rewards caused overfitting to simulation dynamics.
• Dynamics Models (H5):
  • Lerp (Simple): Converged fastest in simulation but failed completely on hardware under strict constraints.
  • KM (Physics-based): Slower to train but achieved 50% real-world success under strict precision constraints ($\tau=7.5$).
  • WGM: Offered a middle ground but did not outperform KM in real-world success rates.
• The "Unreachable" Paradox: A critical finding was that the target colors used in hardware experiments were mathematically unreachable in the simulation for all three dynamics models (the minimum required tolerance $\tau_{min}$ was always higher than the training $\tau=7.5$). Despite this, the physics-based models (KM) succeeded in the real world, proving that dynamics accuracy is more important than exact spectrum coverage for transfer.

5. Significance

This paper shifts the focus of sim-to-real research from purely "dynamics randomization" to holistic MDP formulation.

• Practical Impact: It provides actionable guidelines for industrial RL deployment: always include goal information in the state, use relative state representations, keep reward functions simple, and prioritize physics-based dynamics models over computationally cheap approximations.
• Theoretical Insight: It demonstrates that a policy optimized for an average goal (due to missing target state info) cannot adapt to specific real-world instances, highlighting the importance of the Markov property in goal-conditioned RL.
• Industrial Relevance: The findings are directly applicable to high-precision manufacturing and biological processes (like CAR-T therapy) where safety and precision preclude extensive real-world trial-and-error.