💻 computer science

Towards an Adaptive Social Game-Playing Robot: An Offline Reinforcement Learning-Based Framework

This paper presents an offline reinforcement learning-based framework for an adaptive social game-playing robot that integrates multimodal emotion recognition to safely and efficiently train users with learning difficulties, demonstrating that algorithms like BCQ, DDQN, and CQL offer robust and bias-mitigated policies derived from real-world datasets.

Original authors: Soon Jynn Chu, Raju Gottumukkala, Alan Barhorst

Published 2026-03-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Soon Jynn Chu, Raju Gottumukkala, Alan Barhorst

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play checkers with a child. Your goal isn't just to win; it's to make the child feel happy, engaged, and not too frustrated. If the robot plays too hard, the child gets angry and quits. If it plays too easy, the child gets bored.

The robot needs to be a socially intelligent coach. It needs to read the child's face and body language, guess how they are feeling, and then decide: "Should I smile to cheer them up? Should I make the game easier? Or should I look serious to challenge them?"

This paper is about teaching a robot to do exactly that, but with a very important twist: The robot learns without ever actually playing with a real human during the training phase.

Here is the breakdown of how they did it, using simple analogies:

1. The Problem: The "Trial-and-Error" Trap

Usually, to teach a robot how to behave, you let it practice by interacting with real people. This is called Online Reinforcement Learning.

The Analogy: Imagine teaching a new chef by letting them cook for a real customer. If they burn the steak, the customer is unhappy. If they put salt in the dessert, the customer is disgusted.
The Issue: In the real world, you can't let a robot make hundreds of "bad" social mistakes (like being rude or making a game too hard) while it learns. It's unsafe, expensive, and uncomfortable for the human.

2. The Solution: The "Video Replay" Method (Offline RL)

The authors used Offline Reinforcement Learning.

The Analogy: Instead of letting the chef cook for a customer, you give them a library of video recordings of other chefs cooking. The robot watches these videos, studies what the chefs did, what the customers' reactions were, and learns the best strategies without ever touching a pan.
The Benefit: The robot learns from "logged" data (past interactions) so it never has to risk hurting a user's feelings during the learning process.

3. The Robot's "Super-Senses"

To learn from these videos, the robot needed to understand two things: The Game and The Human.

The Eyes: The robot used cameras to see the checkers board (who is winning?) and the human's face (are they smiling or frowning?).
The "Inner Ear": The human wore a wristband that measured their heartbeat and skin sweat. This is like a "stress-o-meter." Even if the human is smiling, the wristband might tell the robot, "Actually, they are super stressed right now."
The Brain: The robot combined all this info (Game Score + Face + Stress Level) to decide its next move.

4. The "Chef's School" (The Experiment)

The researchers gathered data from real people playing checkers with a robot. They recorded 232 "moves" (steps in the game). It wasn't a huge amount of data—think of it as a small cookbook rather than a massive library.

Then, they tested six different "learning algorithms" (six different ways for the robot to study the cookbook) to see which one was the best student.

5. The Results: Who Passed the Test?

They found that not all learning methods are created equal:

The "Wild Card" (NFQ): This method was like a student who got distracted easily. It tried to guess the answers but ended up with wild, crazy numbers. It was too unstable to trust.
The "Steady Eddies" (DDQN and BCQ): These two were very consistent. No matter how you tweaked their study habits (hyperparameters), they gave reliable answers. They are the "safe bets" for future robots.
The "Realist" (CQL): This was the star of the show. While other methods tended to overestimate how well they were doing (thinking they were geniuses when they weren't), CQL was humble and realistic. It learned the most accurate lessons from the data, avoiding the trap of thinking it knew more than it actually did.

6. Why This Matters

This research is a foundational step toward building robots that can be socially intelligent.

Before: Robots were like rigid puppets, following strict rules that often felt unnatural.
Now: We have a blueprint for robots that can learn from past data to understand human emotions, adjust their behavior, and be good friends or teachers without needing to practice on real people first.

In a nutshell: The authors built a system where a robot learns to be a great game-playing partner by studying a "highlight reel" of past games, rather than making mistakes in front of real people. They found that the CQL algorithm is the best at learning these lessons without getting confused or overconfident.

1. Problem Statement

Social robotics aims to create agents that can engage users through meaningful emotional interactions, particularly in educational settings for students with learning difficulties. However, current approaches face significant challenges:

Limitations of Rule-Based Systems: Naive mappings from detected emotions to robot actions often feel unnatural and lack the flexibility to handle the unpredictability of real-world user behavior.
Risks of Online Reinforcement Learning (RL): While online RL can learn nuanced strategies through trial-and-error, it is impractical for Human-Robot Interaction (HRI). It requires extensive real-world training data, is time-consuming, and poses safety risks where a robot might exhibit socially inappropriate or harmful behaviors during the learning phase, potentially damaging user trust.
Gap in Offline RL: Although Offline RL (learning from pre-collected datasets) offers a safe alternative, there is a lack of comprehensive frameworks and comparative studies evaluating which algorithms are most robust and effective for emotionally adaptive social robots.

2. Methodology

System Architecture

The authors propose a four-layer system architecture implemented on a Baxter robot:

Sensing: Captures multimodal data via a wearable device (Empatica E4 for biometric/physiological arousal), a head-mounted camera (for facial expressions), and an arm-mounted camera (for game board state).
Interpretation: Processes raw data into meaningful states.
- Emotion Recognition: Uses biometric signals for arousal detection and a mini-Xception network (trained on FER2013) for facial expression classification (Angry, Happy, Neutral).
- Game Status Analyzer: Extracts board state and calculates a game score based on piece positions.
Decision-Making (The RL Core): Formulated as a Markov Decision Process (MDP).
- State Space ( $s_t$ ): Defined by Game Status (Win/Loss/Draw), Facial Emotion, and Physiological Arousal (Present/Absent). Total: 18 states.
- Action Space ( $a_t$ ): Defined by Robot Facial Display (Angry/Happy/Neutral) and Difficulty Adjustment (Increase/Decrease/Constant). Total: 9 actions.
- Reward Function ( $r_t$ ): A weighted sum of GameScore (encouraging appropriate challenge) and EmoScore (based on Skin Conductance Response peaks and facial expression valence).
Actuation: The robot displays facial expressions on a screen and physically moves game pieces via arm control.

Data Collection

Scenario: American Checkers game played between the robot and 5 human participants.
Protocol: To ensure fairness and safety, the robot's initial actions were sampled from a random uniform distribution.
Dataset: 232 total steps collected across 5 participants (approx. 46 steps/participant). The dataset covers 63% of the state-action space, representing a realistic, partially covered HRI dataset.

Offline RL Implementation

The study evaluated six offline RL algorithms using the d3rlpy library:

Neural Fitted Q Iteration (NFQ)
Deep Q-Network (DQN)
Double DQN (DDQN)
Soft-Actor Critic (SAC)
Batch Constrained Q-Learning (BCQ)
Conservative Q-Learning (CQL)

The researchers performed a grid search over hyperparameters (learning rate, batch size, hidden layers, units, activation functions) and used Stratified Group K-Fold cross-validation to ensure generalization across different subjects.

3. Key Contributions

Multimodal Offline RL Framework: A novel system that integrates physiological and visual data to create a structured, socially rich testbed for training adaptive robots without real-time exploration risks.
Comparative Algorithmic Analysis: A rigorous evaluation of six offline RL algorithms specifically for HRI, determining which are most robust to hyperparameter variations and least prone to estimation biases (over/underestimation) in data-scarce environments.
Baseline for Safe Social Robotics: Establishes a foundational approach for deploying emotionally intelligent agents that learn entirely from offline datasets, ensuring user comfort and scalability.

4. Results & Analysis

Hyperparameter Sensitivity

Most Robust: DDQN and BCQ demonstrated the lowest standard deviation (0.10 and 0.14, respectively) in performance across hyperparameter variations, making them reliable for deployment where extensive tuning is not feasible.
Least Robust: NFQ showed extreme instability with a standard deviation of 51.47 and massive overestimation (max value ~323), indicating it is unsuitable for this specific data-scarce context.

Algorithm Performance (Bias & Accuracy)

Best Accuracy: CQL achieved the lowest absolute error (0.23) against the ground truth.
Bias Mitigation: CQL was the most effective at mitigating overestimation bias. By conservatively lower-bounding value estimates, its predictions clustered closest to the true environment dynamics.
Overestimation Trend: Most algorithms (except CQL) tended to overestimate the value function ( $\hat{V}(s_0)$ ), a common issue in offline RL when data coverage is incomplete.

Convergence

Five of the six algorithms converged successfully.
NFQ diverged, likely due to the lack of a separate target network, creating a "moving target" problem.
CQL showed the most stable loss curve with the least spike during target synchronization.

5. Significance and Future Work

This research provides a critical pathway for developing socially intelligent agents that can learn complex, emotion-adaptive behaviors without the safety risks and high costs of online training.

Practical Impact: The findings suggest that CQL is ideal for accuracy and bias control, while DDQN/BCQ are preferred for robustness in deployment scenarios with limited tuning resources.
Limitations: The current dataset is limited in size and demographic diversity. The system treats the environment as fully observable, whereas human emotion is inherently latent.
Future Directions: The authors propose expanding data collection with diverse modalities, incorporating teleoperation for richer state exploration, and modeling the problem as a Partially Observable MDP (POMDP) to better track the latent belief distributions of user affect.