🤖 machine learning

Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluation

This paper proposes a novel offline reinforcement learning framework that utilizes off-policy evaluation on logged interaction data to automatically select optimal state representations and reward functions for human-robot collaboration, thereby eliminating the need for costly real-time environment interactions and enhancing the scalability of RL in safety-critical applications.

Original authors: Saurav Singh, Rodney Sanchez, Alexander Ororbia, Jamison Heard

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saurav Singh, Rodney Sanchez, Alexander Ororbia, Jamison Heard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to work alongside a human, like a co-pilot in a cockpit or a nurse in a hospital. To do this, you need to give the robot a "brain" (Reinforcement Learning) that learns by trial and error. But here's the problem: in the real world, you can't let the robot crash a plane or hurt a patient just to see what happens. That's too dangerous and expensive.

Usually, to teach the robot, humans have to spend years manually designing the robot's "senses" (what data it looks at) and its "motivation" (what it gets rewarded for). If the robot fails, you have to guess why and start over. It's like trying to tune a radio by turning the dial blindly while standing in a soundproof room.

This paper proposes a smarter way to tune that radio without ever leaving the room.

The Big Idea: The "Time-Travel" Coach

The authors introduce a method called Off-Policy Evaluation (OPE). Think of this as a "Time-Travel Coach."

Instead of letting the robot practice in the real world, the researchers take a huge pile of old logs—recordings of how humans and robots have interacted in the past. They use these logs to simulate thousands of different "what-if" scenarios.

The Problem: They need to figure out which set of "senses" (State Space) and which set of "rules for rewards" (Reward Function) will make the robot the best partner.
The Solution: They use the OPE Coach to look at the old logs and say, "If we had taught the robot using this specific set of senses and this specific reward rule, here is exactly how well it would have done."

This allows them to pick the best training recipe without ever risking a single real-world interaction.

The Experiments: Two Different Playgrounds

To prove this works, they tested their "Time-Travel Coach" in two very different environments.

1. The Lunar Lander (The Video Game Test)

First, they used a video game called Lunar Lander, where a robot has to land a spaceship on the moon.

The Test: They tried teaching the robot with different "senses." One version saw everything (position, speed, angle). Another version saw extra "noise" (fake data). A third version saw less (no position data).
The Result: The Coach looked at the logs and correctly identified that the "full view" was the best. It knew that adding fake noise confused the robot, and removing position data made it crash.
The Reward: They also tested different ways to give the robot points. They found that the best reward system wasn't just about landing safely; it was a mix of staying steady, using fuel wisely, and landing gently. The Coach picked the winner instantly, saving them from building a new robot to test it.

2. The NASA MATB-II (The Human Stress Test)

Next, they moved to a real human scenario using a simulation called MATB-II. Imagine a busy air traffic controller or a pilot juggling four tasks at once:

Tracking: Keeping a dot on a moving target.
Monitoring: Watching gauges for alarms.
Resource Management: Keeping fuel tanks balanced.
Communications: Listening to radio commands.

In this study, humans worked alongside an automation system. The researchers wanted to know: What data should the robot use to decide when to help?

The Data: They tried giving the robot just task data, just heart rate data, or a mix of task data, heart rate, and estimated stress levels.
The Finding: The "Time-Travel Coach" determined that the robot performed best when it could see everything: what the human was doing, their heart rate, and their stress level. However, it found that just looking at stress levels alone wasn't enough; the robot needed to see the actual tasks too.
The Reward: They also tested different ways to reward the robot. They found that a "balanced" reward system (which punished bad actions and rewarded good ones) worked best to teach the robot how to be a helpful partner.

The Real-World Test: Putting the Robot to Work

After using the Coach to pick the best recipe, they actually built the robot and put it in a room with 19 human volunteers. They compared three types of helpers:

The Rule-Based Robot: Follows a strict manual (e.g., "If the human is stressed, take over Task A").
The Standard RL Robot: Learned from the logs using the best recipe the Coach found.
The Fine-Tuned RL Robot: Learned from the logs, then tried to adjust its behavior in real-time based on the specific human it was working with.

The Results:

The Standard RL Robot was the most consistent. It was the best at tracking moving targets and monitoring alarms. It made the humans feel the least stressed and built the most trust. It was like a reliable, calm co-pilot who knew exactly when to step in.
The Rule-Based Robot was okay at simple tasks but struggled when things got chaotic.
The Fine-Tuned Robot was interesting. It was the best at the "Communications" task (listening to radio commands) because it adapted perfectly to that specific person. However, when the workload got heavy, it became a bit unpredictable. It was like a co-pilot who was great at chatting but sometimes forgot to watch the fuel gauge when things got crazy.

The Takeaway

The paper concludes that you don't need to crash a few rockets or stress out a few pilots to figure out how to build a good robot team. By using Off-Policy Evaluation, you can look at old data, simulate millions of training scenarios, and pick the perfect "senses" and "motivation" for your robot before you ever turn it on.

This makes building safe, effective human-robot teams much faster, cheaper, and safer. It turns the process of "guessing and checking" into a precise, data-driven science.

Technical Summary: Formulating Reinforcement Learning for Human-Robot Collaboration Through Off-Policy Evaluation

1. Problem Statement

Reinforcement Learning (RL) holds significant potential for optimizing sequential decision-making in dynamic environments, particularly in human-robot interaction (HRI). However, deploying RL in real-world, safety-critical settings (e.g., healthcare, aerospace, autonomous driving) faces substantial barriers. Traditional RL approaches rely heavily on extensive online interactions, trial-and-error, and domain expertise to define effective state representations and reward functions. This process is often costly, time-consuming, and unsafe.

While Offline Reinforcement Learning (ORL) mitigates the need for real-time interaction by training on pre-collected data, a critical challenge remains: selecting optimal state spaces and reward functions without online experimentation. Furthermore, ensuring that offline-trained policies are safe and effective in dynamic human-robot teaming scenarios requires robust evaluation methods that do not necessitate further real-world deployment.

2. Methodology

The authors propose a novel framework that leverages Off-Policy Evaluation (OPE) to systematically select optimal state space representations and reward functions using only logged interaction data. This approach eliminates the need for real-time environment access or human-in-the-loop feedback during the design phase.

2.1 State Space Selection via OPE

The framework evaluates multiple candidate state representations ( $S_1, S_2, ..., S_K$ ) derived from domain knowledge or feature selection.

Offline Training: An offline RL agent (specifically Constrained Q-Learning, CQL) is trained for each candidate state space using the logged dataset.
OPE Estimation: An OPE estimator (utilizing methods such as Importance Sampling, Direct Method, or Fitted Q-Evaluation) estimates the expected return of each trained policy without online interaction.
Selection: The optimal state space ( $S^*$ ) is selected based on the highest OPE-estimated return, effectively ranking representations by their ability to induce high-performing policies.

2.2 Reward Function Selection via OPE

The framework similarly evaluates candidate reward functions ( $r_1, r_2, ..., r_M$ ) by assessing their ability to distinguish between high-performing and low-performing policies.

Policy Evaluation: Pre-defined high-performing ( $\pi^+$ ) and low-performing ( $\pi^-$ ) policies are evaluated under each candidate reward function using OPE.
Separation Metrics: The effectiveness of a reward function is quantified by the divergence between the estimated returns of $\pi^+$ $π^{+}$ and $\pi^-$ $π^{-}$ . The authors utilize:
- Mean return difference ( $\Delta V$ ).
- Kullback-Leibler (KL) divergence.
- Jensen-Shannon (JS) divergence.
Selection: The optimal reward function ( $r^*$ ) is the one that maximizes the separation between good and bad policies, ensuring a strong learning signal.

3. Experimental Validation

The proposed methodology was validated in two distinct environments:

3.1 Controlled Setting: OpenAI Gym Lunar Lander

Objective: Assess the framework's ability to select state spaces and reward functions in a controlled environment.
State Space Results: Three candidates were tested: original ( $S_{orig}$ ), expanded with noise ( $S_{more}$ ), and reduced ( $S_{less}$ ). OPE correctly identified $S_{orig}$ as the optimal representation, demonstrating that unnecessary feature expansion degrades performance while excessive compression omits critical information.
Reward Function Results: The framework successfully identified a baseline reward function incorporating state-based, action-based, and terminal rewards as superior to alternatives (e.g., terminal-only or action-only). The selected function exhibited the highest JS divergence, confirming its ability to provide effective learning signals.

3.2 Real-World Setting: NASA-MATB-II Human Subjects Study

Objective: Evaluate applicability to human-robot teaming using data from a Multi-Attribute Task Battery-II (MATB-II) simulation involving nine participants under varying workload conditions.
State Space Selection: Candidates included combinations of task data, physiological data (heart rate, respiration), and estimated workload.
- Result: The multimodal state space ( $S_{task+physio+wl}$ ) yielded the highest OPE return. However, further analysis indicated that workload data provided marginal benefit when task and physiological data were present. Consequently, $S_{task+physio}$ was selected as the preferred representation, balancing performance with computational efficiency.
Reward Function Selection: Three reward functions were tested: penalizing bad behavior ( $r_1$ $r_{1}$ ), rewarding good behavior ( $r_2$ $r_{2}$ ), and a weighted sum ( $r_3$ $r_{3}$ ).
- Result: The weighted sum ( $r_3$ ) was selected as optimal, showing the highest KL and JS divergences, indicating the best separation between policy performances.

3.3 Online Human Subjects Validation

A subsequent IRB-approved study with 19 participants compared three agents:

Rule-Based (RB): Predefined rules activating automation during overload.
RL Agent: Trained offline using the OPE-selected state space and reward function.
RL Fine-Tuning (RLFT): The RL agent further fine-tuned via real-time interaction.

Key Findings:

Task Performance: The RL agent demonstrated the most consistent performance across tracking and system monitoring tasks, significantly outperforming RB and RLFT in response times and failure rates under varying loads. The RLFT agent excelled in the communications task but showed instability in resource management and higher failure rates under overload.
Trust and Workload: The RL agent received the highest trust ratings and resulted in the lowest perceived workload and highest fluency. Conversely, RLFT, despite receiving high post-trial trust scores regarding reliability, induced the highest subjective workload and frustration, suggesting that real-time fine-tuning can introduce unpredictability and cognitive strain.
Automation Frequency: The RL agent utilized selective, well-timed automation, whereas RLFT often over-automated or under-automated depending on the task, leading to inconsistent performance.

4. Key Contributions

OPE-Driven Design Framework: The introduction of a framework that uses Off-Policy Evaluation to select optimal state spaces and reward functions from logged data, bypassing the need for costly real-time interactions.
Data-Driven Optimization: A method to optimize RL formulations for both controlled (Lunar Lander) and complex human-robot interaction scenarios (NASA-MATB-II) by systematically evaluating candidate configurations.
Empirical Validation: Demonstration that OPE-based selection leads to RL agents that outperform rule-based baselines and fine-tuned variants in terms of adaptability, reliability, and user trust in safety-critical, multi-task environments.

5. Significance and Claims

The paper claims that this work enhances the feasibility and scalability of offline RL for real-world environments by automating critical design decisions. By reducing dependency on trial-and-error and online interactions, the framework offers a more reliable, effective, and sustainable approach to formulating RL for complex HRI settings.

The authors emphasize a critical trade-off observed in their results: while personalization through fine-tuning (RLFT) can improve specific task alignment, it may compromise system stability and increase cognitive load. Therefore, the proposed OPE-based offline approach offers a robust alternative that balances adaptability with predictability. The work suggests that for safety-critical domains, ensuring robust policy evaluation before deployment via OPE is essential, and that integrating physiological data into state representations can significantly improve human-robot teaming performance without the risks associated with extensive online experimentation.