💻 computer science

Mind the Context: Continual Learning of Socially Appropriate Robot Actions via Environmental-Social Disentanglement

This paper introduces the Explicit Disentanglement Dual-Branch (EDD) framework, which enables social robots to continually learn context-appropriate actions across diverse environments by explicitly separating environmental and social cues to mitigate catastrophic forgetting.

Original authors: Rafal Robert Karpinski, Fethiye Irmak Dogan, Nikhil Churamani, Yiming Luo, Maartje M. A. de Graaf, Davide Dell'Anna, Hatice Gunes

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rafal Robert Karpinski, Fethiye Irmak Dogan, Nikhil Churamani, Yiming Luo, Maartje M. A. de Graaf, Davide Dell'Anna, Hatice Gunes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot walking into a room. To a human, the room tells a story: is it a chaotic living room where a party is happening, or a quiet office where a serious meeting is underway? The same action, like starting a loud conversation or vacuuming the floor, might be perfectly polite in the first room but a total disaster in the second. This is the world of social robotics, where machines try to learn the unwritten rules of human behavior. But here's the tricky part: robots can't be programmed with every single rule for every possible room they might ever visit. Instead, they need Continual Learning. Think of this like a student who keeps going to new schools. They need to learn the new rules of the cafeteria or the gym without forgetting the rules they learned in their old school. If they forget, it's called "catastrophic forgetting," and the robot becomes confused and rude.

The big question researchers are asking is: how do we teach a robot to understand that where it is (the environment) and who is there (the social crowd) work together to decide what is polite? Usually, robots just look at a whole picture of a room and try to guess the rules. But this paper suggests that's like trying to solve a puzzle while wearing foggy glasses. The authors propose a smarter way: teach the robot to separate the "background" (the furniture, the walls) from the "people" (the crowd, their positions) and learn from them separately before putting the pieces back together.

The "Mind the Context" Experiment

In this study, the researchers built a new learning system called EDD (Explicit Disentanglement Dual-Branch). Imagine a robot with two different pairs of glasses. One pair, the "Environmental Lens," blurs out all the people so the robot can focus purely on the room's layout—like seeing if there's a table to serve food on or a messy floor to clean. The other pair, the "Social Lens," blurs out the furniture and walls, leaving only the silhouettes of people so the robot can see who is standing where and how close they are to each other.

The robot uses these two separate views to learn. It has two "brains" (or branches) that process these views independently and then combine their thoughts to decide: "Is it okay to start a conversation here?" or "Should I vacuum now?" To make sure the robot doesn't forget what it learned in the living room when it moves to the office, the team used a "replay buffer." This is like a digital scrapbook where the robot keeps a few snapshots of old rooms to practice on while learning new ones, preventing it from getting confused.

What They Found

The team tested this system in six different indoor settings, ranging from a cozy home to various office spaces like meeting rooms, hallways, and big open offices. They compared their "two-lens" robot against other methods, including robots that just looked at the whole picture at once, and even some very advanced AI models that try to guess social rules without any training (called "zero-shot" models).

The results were quite clear. The EDD robot, with its separated lenses and scrapbook, was the best at predicting what humans would consider polite. It made fewer mistakes (a lower error rate of 0.783 compared to over 1.2 or 2.0 for other methods) and its guesses matched human opinions much more closely.

The researchers also tested different ways to split the "people" from the "room." They found that using bounding boxes (drawing a box around people and hiding everything outside it) worked slightly better than just using their outlines (silhouettes) or just zooming in on the robot. This suggests that clearly hiding the people from the "environment" view helps the robot understand the room's rules better.

Interestingly, the order in which the robot visited these rooms didn't matter too much. Whether it started in a simple hallway and moved to a complex living room, or vice versa, the robot learned well either way. This suggests the system is quite robust and doesn't get easily confused by the sequence of new experiences.

The Takeaway

The paper suggests that for robots to be truly polite in our changing world, they need to stop looking at a scene as one big, messy blob. Instead, they need to actively separate the "where" from the "who." By explicitly teaching the robot to understand the environment and the social crowd as two different things, and then combining that knowledge, the robot learns faster and forgets less. While this was tested on synthetic images (computer-generated scenes) rather than real-world video, the results indicate that this "disentanglement" strategy is a promising path for building robots that can adapt to new social situations without losing their manners.

Technical Summary: Mind the Context – Continual Learning of Socially Appropriate Robot Actions via Environmental–Social Disentanglement

1. Problem Formulation

The paper addresses the challenge of enabling social robots to operate across diverse, changing indoor environments (e.g., homes, offices, meeting rooms) where social appropriateness is context-dependent. A specific action (e.g., starting a conversation or cleaning) may be appropriate in one setting (a crowded living room) but disruptive in another (a meeting room), even if the social arrangement of people appears similar.

The core problem is framed as Domain-Incremental Continual Learning (DIL). Unlike task-incremental settings where the task changes but the environment is static, DIL requires the robot to sequentially encounter new domains (environments) with distinct data distributions. The robot must learn to predict the appropriateness of multiple actions (e.g., vacuuming, serving, carrying objects) in these new domains while retaining knowledge from previous ones, thereby mitigating catastrophic forgetting.

A critical gap identified in prior work is that existing methods for learning socially appropriate actions typically treat the scene as a monolithic input, implicitly mixing environmental cues (layout, furniture, openness) and social cues (positions of agents, group configurations). The authors argue that for domain-incremental learning, explicitly separating these cues is necessary to reduce spurious correlations and improve knowledge transfer.

2. Methodology: The EDD Framework

The authors propose the Explicit Disentanglement Dual-Branch (EDD) framework, which consists of three main components:

A. Semantic Context Decomposition

To explicitly separate environmental and social information, the authors employ Panoptic Segmentation to decompose each scene image $x$ into two distinct views:

Environmental View ( $x_E$ ): This view highlights the background and spatial layout while obscuring social agents. It is constructed by masking out detected agents and replacing them with a fixed color, while also introducing "decoy" bounding box masks sampled from a domain pool to further obscure agent locations. This forces the network to rely on affordances and spatial constraints.
Social View ( $x_S$ ): This view isolates the social agents (humans, pets, other robots) against a uniform background. It is constructed using the agents' silhouette masks, preserving their relative positions and groupings while removing environmental context.

B. Dual-Branch Network

The decomposed views are processed by a Dual-Branch Architecture:

Environmental Encoder ( $\phi_E$ ): Processes $x_E$ to extract features related to the setting (e.g., room type, furniture density).
Social Encoder ( $\phi_S$ ): Processes $x_S$ to extract features related to agent configurations.
Fusion and Regression: The feature vectors from both branches are concatenated and passed through a shared Regression Head ( $g$ ) to predict a $K$ -dimensional vector of appropriateness scores for $K$ possible actions. This allows the model to learn how environmental and social factors jointly influence action appropriateness.

C. Continual Learning with Replay

To handle the sequential arrival of domains without access to past data, EDD utilizes an Experience Replay strategy:

A memory buffer stores a fixed number of samples from each encountered domain.
During training on a new domain, the model is trained on a mix of current domain data and replayed samples from the buffer.
This approach approximates joint training across all domains, minimizing the cumulative loss and mitigating catastrophic forgetting.

3. Key Contributions

Explicit Disentanglement: The paper introduces a novel approach to domain-incremental CL for social robotics by explicitly separating environmental and social cues at the input level using panoptic segmentation, rather than relying on implicit feature learning from full-scene images.
EDD Framework: The proposal of the Explicit Disentanglement Dual-Branch framework, which combines semantic decomposition with a dual-branch architecture and replay-based rehearsal.
Comprehensive Evaluation: The authors evaluate the framework across six indoor domains (Home and five office variants) and conduct ablation studies on:
- Disentanglement Strategies: Comparing bounding-box masks, silhouettes, spatial close-ups, and no decomposition.
- Domain Ordering: Testing various sequences (complexity-based, contrast-based) to assess sensitivity to the order of domain exposure.
- Baselines: Comparing against robotics-specific CL methods, domain-incremental vision baselines, and zero-shot Vision-Language Models (VLMs).

4. Experimental Results

The experiments were conducted on a merged dataset of OfficeDB and MannersDB+, focusing on the Pepper robot in nine action categories.

Performance vs. Baselines (RQ1): EDD outperformed all baselines, including state-of-the-art robotics CL methods (FedLGR), domain-incremental vision models (DUCA, DARE++), and zero-shot VLMs (Qwen, LLaVA, DeepSeek). EDD achieved the lowest Root Mean Squared Error (RMSE: 0.783) and the highest Pearson (PCC: 0.552) and Concordance (CCC: 0.375) correlation coefficients with human ratings. Notably, VLMs showed competitive RMSE but failed to correlate well with human judgments, suggesting foundation models alone are insufficient for this specific task.
Disentanglement Strategies (RQ2): The Bounding-Box decomposition (the proposed method) yielded the best overall performance and lowest forgetting (Backwards Transfer, BWT). While the "No Decomposition" control performed similarly in error metrics, the explicit separation provided better alignment with human judgments (higher PCC/CCC). The "Robot Close-up" strategy showed the best retention (lowest BWT) but lower correlation, suggesting it preserves domain-specific features well but may miss broader social context.
Domain Ordering (RQ3): The order of domain exposure had a measurable but modest effect on performance. While an "increasing complexity" curriculum showed slightly better performance on the Home domain, the overall impact on forgetting and general accuracy was limited compared to the architectural choices. The model remained relatively robust across different sequencing strategies.

5. Significance and Claims

The paper claims that explicitly disentangling environmental and social cues is a viable and effective strategy for continual learning in social robotics. The authors argue that:

Social appropriateness is not solely a function of agent arrangement but is jointly determined by the environment (affordances) and social configuration.
Treating these cues as separate inputs allows the model to better adapt to domain shifts without forgetting previous knowledge, as the environmental branch can adapt to new layouts while the social branch retains generalizable agent-interaction patterns.
Current zero-shot VLMs, while powerful, lack the specific structural inductive biases required to capture the nuanced, context-dependent nature of social norms in embodied settings.

The work suggests that future long-term deployments of social robots will benefit from architectures that structurally separate context types, enabling more robust adaptation to diverse and changing real-world environments. The authors acknowledge limitations, such as the use of synthetic data and the reliance on panoptic segmentation, and propose future work in real-world validation, explainability of branch contributions, and adaptive fusion mechanisms.