💻 computer science

StARS: Socially Appropriate Robot Actions via a Recommender System-Driven Approach

The paper introduces StARS, a model-agnostic framework that leverages recommender system techniques to address inter-subject variability in human-robot interaction by generating personalized, socially appropriate robot actions tailored to individual user norms and specific contexts.

Original authors: Erencem Ozbey, Fethiye Irmak Dogan, Jin Huang, Hatice Gunes

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Erencem Ozbey, Fethiye Irmak Dogan, Jin Huang, Hatice Gunes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a room full of people, and a friendly robot is trying to help out. To us, "being helpful" might look the same to everyone, but in the world of Human-Robot Interaction (HRI), social rules are surprisingly personal. What one person finds charming and helpful, another might find creepy or even dangerous. This is the tricky puzzle scientists are trying to solve: how do we teach robots to read the room and the specific person in it?

The paper tackles this by borrowing a clever trick from a technology we all use every day: the recommendation system. You know how Netflix or Spotify suggests movies or songs? They don't just guess what everyone likes; they look at what you specifically liked in the past, mixed with what people similar to you enjoyed. The researchers realized that deciding if a robot action is "socially appropriate" is actually very similar to picking a movie. It's not a single universal truth; it's a matter of personal taste. By treating robot actions like "items" and people like "users," they can use math designed for movie recommendations to help robots figure out exactly what a specific human wants them to do.

The Problem: One Size Does Not Fit All

For a long time, robots were taught to act based on what the "average" person thinks is okay. If 51% of people thought a robot waving its arm was fine, the robot would do it. But the authors point out a major flaw in this approach: social appropriateness isn't universal. In a crowded party, one person might love a robot serving them a drink, while another might think it's a tripping hazard or an invasion of privacy. If a robot keeps guessing wrong based on the "average," people will get annoyed, lose trust, and stop interacting with it.

The paper argues that we need to stop trying to find one "perfect" robot behavior and start building robots that can adapt to your specific preferences.

The Solution: StARS (Socially Appropriate Robot Actions)

To solve this, the team created a new framework called StARS. Think of StARS as a "personal stylist" for robot behavior. Instead of just looking at the scene (the room, the people, the objects), StARS looks at who is watching.

Here is how it works, using a simple analogy:
Imagine you are trying to guess which pizza topping a friend will like.

The Old Way: You ask 100 people what they like, take the average, and guess "Pepperoni" because it's the most popular. Your friend hates pepperoni.
The StARS Way: You look at your friend's past pizza orders. You also notice that your friend usually likes the same toppings as their best friend, who loves mushrooms. StARS combines these clues. It says, "This specific person, in this specific situation, probably wants mushrooms, even if the crowd wants pepperoni."

Technically, StARS uses a method called Collaborative Filtering. This is the same math Netflix uses. It builds a giant mental spreadsheet where it tracks:

Users: The different people (annotators) who rated the robot actions.
Items: The different robot actions (like "serving food" or "moving out of the way").
Context: The scene (a kitchen, a hallway, a park).

By analyzing this spreadsheet, StARS learns hidden patterns. It might realize that "User A" always thinks robots are too close in narrow hallways, while "User B" thinks they are too far away. It then uses these patterns to predict what that specific user will think is appropriate in a new situation, even if the robot has never met them before.

What They Found

The researchers tested StARS on two different datasets: one about robots doing chores in a living room (MannersDB+) and another about robots navigating through crowds (SocNav1). They tried it with many different "brain" models (backbones) to see if it worked generally.

The results were consistently positive:

Better Accuracy: When they added StARS to their robot models, the predictions got significantly better. For example, on the living room dataset, the error rate (RMSE) dropped from around 1.27 to 1.09 for their best model.
Better Agreement: The robot's guesses matched human opinions much more closely. The correlation (a measure of how well the robot's opinion matched the human's) jumped from about 0.30 to 0.57 on the living room dataset.
It Works with Little Data: One of the biggest challenges in personalization is that you don't have a lot of data for every single person. The team found that StARS is very good at learning from just a few examples. Even when they only gave the system 10% of the available data, it still made huge improvements compared to the old "average" method. As they added more data, the robot got even better, but the biggest leap happened early on.

Why This Matters

The paper suggests that we don't need to completely redesign how robots think to make them more personal. Instead, we can just add a "recommendation layer" on top of whatever robot brain we already have. This makes it possible to deploy robots in real homes and hospitals where every person is different.

The authors are careful to note that they haven't "solved" the problem of social robots forever. Social situations are complex, and there is still a lot of ambiguity. However, their work suggests that treating social appropriateness as a personal preference problem—rather than a universal rule—is a powerful and data-efficient way to make robots that actually fit in with us. By using the same logic that helps you find your next favorite song, we might finally get robots that know exactly how to behave around you.

Technical Summary: StARS – Socially Appropriate Robot Actions via a Recommender System-Driven Approach

1. Problem Statement

Social appropriateness in Human-Robot Interaction (HRI) is not universal; it exhibits significant inter-subject variability. The same robot action in an identical context may be perceived as helpful by one user and intrusive or unsafe by another. Existing approaches often rely on aggregated human ratings or shared representations of appropriateness, which smooth over meaningful disagreements and fail to capture individual user norms. This limitation hinders the deployment of robots in close-proximity settings (e.g., homes, healthcare) where personalization is essential.

The core challenge addressed is how to model and predict socially appropriate actions that align with specific user preferences, particularly when preference feedback is sparse and costly to collect. The authors reformulate socially appropriate action generation as a preference modeling problem, drawing inspiration from recommender systems (RS) to treat annotators as users, scenes as items, and appropriateness scores as targets.

2. Methodology: The StARS Framework

The authors propose StARS (Socially Appropriate Robot Actions), a model-agnostic framework that integrates Collaborative Filtering (CF) with learnable scene representations.

A. Problem Formulation

Instead of predicting an aggregated score $y_{i,a}$ for a scene $i$ and action $a$ , StARS predicts a personalized score $\hat{y}_{u,i,a}$ for a specific user $u$ . The system learns from a dataset $D$ of annotations $(u, i, a, y_{u,i,a})$ .

B. Architecture and Components

StARS augments a differentiable scene encoder $\phi_\theta$ with a Collaborative Filtering module based on Matrix Factorization (MF). The prediction function $f(u, i, x_i, a)$ combines:

Latent Factors: Shared user embeddings ( $p_u$ ) and shared scene embeddings ( $q_i$ ) learned via MF to capture systematic rating tendencies.
Action Embeddings: Action-specific embeddings ( $r_a$ ) to handle multiple candidate actions.
Residual Fusion: To mitigate data sparsity and incorporate content information, the framework fuses collaborative signals with scene content features ( $h_i = \phi_\theta(x_i)$ $h_{i} = ϕ_{θ} (x_{i})$ ):
- Scene Residual Fusion: A hybrid scene embedding $\bar{q}_i = q_i + h_i$ enriches the collaborative item embedding with content features (e.g., spatial layout, object relations).
- Interaction Residual Fusion: The final prediction incorporates the user-scene interaction ( $p_u \odot \bar{q}_i$ ) and adds the content feature $h_i$ as a residual term. This provides a user-independent content shortcut, stabilizing predictions when collaborative factors are weakly estimated (sparse regime).

The final prediction equation is:
$\hat{y}_{u,i,a} = \left( (p_u \odot \bar{q}_i) + h_i \right)^\top r_a + b_u + b_i + b$
where $b$ terms represent global, user, and scene offsets.

C. Training Strategy

StARS employs a two-step training protocol:

Scene Encoder Pre-training: The scene encoder $\phi_\theta$ is pre-trained with a temporary linear head to predict action appropriateness from scene features alone.
End-to-End Joint Optimization: The pre-trained encoder is frozen (or fine-tuned) and jointly optimized with the CF parameters ( $p_u, q_i, r_a, b$ ) using the full personalized prediction objective.

3. Key Contributions

Reformulation of HRI: The paper reframes socially appropriate action generation as a preference estimation problem using recommender system techniques, explicitly modeling inter-user disagreement rather than aggregating it.
Model-Agnostic Framework: StARS is designed to be integrated with various scene encoders (e.g., MLPs, Transformers, Graph Neural Networks) and backbones without redesigning the underlying model.
Sparse Feedback Robustness: The framework specifically addresses the challenge of sparse preference feedback by leveraging collaborative filtering to infer user-specific patterns from limited data.
Empirical Validation: The approach is evaluated on two distinct datasets: MannersDB+ (domestic task-level actions) and SocNav1 (socially compliant navigation).

4. Experimental Results

The authors evaluated StARS against various backbone models (MLP, ResMLP, DCNv2, GraceAE, FT-Transformer for MannersDB+; GCN, GIN, GraphSAGE, GGNN, RGCN, GAT for SocNav1) using five-fold cross-validation.

Performance Improvement (RQ1): Across all backbones and datasets, integrating the CF component (StARS) consistently outperformed the base models.
- On MannersDB+, StARS variants showed substantial reductions in RMSE (e.g., from ~1.27 to ~1.13 for MLP) and significant increases in Pearson correlation ( $r$ ) and Lin's Concordance Correlation Coefficient (CCC).
- On SocNav1, StARS variants achieved lower MSE and higher correlation metrics across all Graph Neural Network architectures.
Consistency and Systematicity (RQ2): Statistical analysis (paired t-tests with Holm correction) confirmed that the improvements were systematic and not artifacts of specific folds or models. The gains were statistically significant ( $p < 0.05$ ) across all tested configurations.
Robustness to Sparsity (RQ3): Learning curve experiments demonstrated that StARS provides substantial performance gains even with limited training data (low-data regimes). Performance improved rapidly with early annotations, suggesting the method is data-efficient and suitable for scenarios where collecting personalized feedback is costly.

5. Significance and Claims

The paper claims that StARS demonstrates the viability of recommender system methods as a scalable approach to personalized decision-making in robotics.

Personalization without Redesign: The framework enables user-specific action selection without requiring a redesign of the underlying perception or decision-making backbone, supporting more adaptive behavior in real-world HRI.
Structured Variability: The results suggest that variation in social appropriateness judgments is structured and can be effectively captured through user-scene interaction modeling, rather than being random noise.
Data Efficiency: The approach offers a practical path for personalization in HRI where data is scarce, implying that a small number of early interactions can yield meaningful user-specific preference estimates.

The authors conclude that while StARS focuses on social appropriateness, the RS formulation could extend to other personalization-centric HRI tasks, such as collaborative assembly or adaptive assistance, provided that appropriate feedback mechanisms are in place.