Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning

1. Problem Statement

Social robots require the ability to infer a user's internal states (latent variables like intent and attitude) from observable behaviors (actions) to achieve social intelligence. Current approaches often treat these perceptual tasks as isolated or assume static relationships between them. However, human social interaction is a dynamic, socio-cognitive process where internal states and observable actions influence each other continuously over time.

The core challenges addressed are:

• Dynamic Coupling: The relationship between tasks (e.g., how "intent" influences "future action") changes depending on the interaction stage (e.g., early approach vs. active engagement).
• Data Scarcity & Generalization: Learning new social tasks without catastrophic forgetting of previously learned ones.
• Interpretability: Understanding how different social cues interact to form a decision, rather than treating the model as a black box.

2. Methodology: SocialLDG

The authors propose SocialLDG, a novel Multi-Task Learning (MTL) framework inspired by Parallel Constraint Satisfaction (PCS) theory from cognitive psychology. The framework models decision-making as a network where constraints (tasks) interact simultaneously.

A. Input and Representation Learning

• Input: Egocentric video streams processed into 2D whole-body pose sequences (face, hands, body) using AlphaPose.
• Spatio-Temporal Encoder: A self-supervised autoencoder backbone is used.
  • Spatial: Graph Attention Network (GAT) processes pose keypoints frame-by-frame.
  • Temporal: Transformer encoder models the sequence.
  • Loss Function: A confidence-aware MSE loss is used to down-weight noisy keypoints (common in egocentric views) based on their confidence scores.
  • Output: A compact, task-agnostic social signal representation ($Z$).

B. The SocialLDG Architecture

The core innovation is a Lexically-guided Dynamic Graph Network that jointly predicts six tasks:

1. Physical Contact: Current ($y_{con\_cur}$) and Future ($y_{con\_fut}$).
2. Internal States: Intent ($y_{int}$) and Attitude ($y_{att}$).
3. Actions: Current ($y_{act\_cur}$) and Future ($y_{act\_fut}$).

The framework consists of three key modules:

1. Lexical Task Tokens (Node Initialization):

   • To prevent graph over-smoothing and inject semantic meaning, each task node is initialized with a unique lexical token.
   • These tokens are generated by averaging embeddings from a frozen SciBERT model based on descriptive sentences (e.g., "Detecting if the user is currently making physical contact").
   • These tokens are combined with the social signal representation $Z$ via a task prompt injection module.

2. Semantic Edge Generator (Dynamic Affinity):

   • Instead of a fixed graph, the model learns a dynamic task affinity matrix ($A$).
   • A Semantic Edge Generator uses the social context ($Z$) and task tokens to compute a continuous edge bias matrix ($B$).
   • This bias acts as a semantic prior, guiding the graph transformer to learn how strongly tasks influence one another at any given time step.

3. Dynamic Graph Transformer:

   • The final task affinity matrix $A$ is computed by combining the learned edge bias ($B$) with self-attention scores ($S$) derived from node features: $A_{i,j} = \sigma(S_{i,j} + B_{i,j})$.
   • Sigmoid Activation: Unlike standard Softmax, Sigmoid is used to allow independent edge weights and prevent forced aggregation.
   • Causal Masking: Edges flowing from "future" states to "current" states are masked to prevent information leakage (ensuring the model predicts the future based on the present, not vice versa).
   • Residual Connection: Original social signals are concatenated with updated node features to preserve information.

3. Key Contributions

1. Lexically-guided Dynamic Graph MTL: A framework that integrates language model priors with dynamic graph learning to model the evolving relationships between latent internal states and observable actions.
2. Task Scalability: Demonstrates the ability to rapidly learn new tasks (fine-tuning) without catastrophic forgetting, leveraging the explicit modeling of inter-task relationships.
3. Interpretability: The learned Task Affinity Matrix provides insights into how social perception evolves over time, revealing which tasks influence each other during specific interaction phases.

4. Experimental Results

The framework was evaluated on two challenging Human-Robot Interaction (HRI) datasets: JPL-Social and HARPER.

• Performance: SocialLDG achieved State-of-the-Art (SOTA) results, outperforming existing MTL baselines (including MMoE, PLE, and AssociationGraph) with an average F1 score of 85.61% and accuracy of 85.23%.
• Ablation Studies:
  • Removing the learnable edge bias or lexical task tokens significantly degraded performance, proving both are essential for capturing task semantics and dynamic correlations.
  • SciBERT fine-tuning for task tokens yielded the best results, suggesting scientific terminology aligns well with the task descriptions.
  • The GAT + Transformer encoder outperformed other backbones (like ST-GCN or LSTM) despite slightly higher reconstruction error, indicating better global dependency capture.
• Scalability: In transfer learning experiments, SocialLDG could generalize to new tasks (e.g., predicting future actions) with only 1 epoch of fine-tuning, significantly outperforming independent fine-tuning baselines and avoiding catastrophic forgetting.
• Dynamic Visualization: The model successfully detected interaction stages (pre-interaction, interacting, post-interaction) by analyzing the sparsity and density of the learned affinity matrix. For example, during a "throwing" action, the matrix became sparse, focusing only on relevant links, whereas it was dense during ambiguous phases.

5. Significance and Conclusion

This work bridges the gap between cognitive science theories and robotic perception. By explicitly modeling the dynamic, time-evolving relationships between a robot's perception of human intent, attitude, and actions, SocialLDG moves beyond reactive systems to proactive, socially intelligent agents.

• Practical Impact: The framework offers a scalable solution for robots to adapt to new social scenarios without full retraining.
• Theoretical Insight: It provides a data-driven method to understand the "black box" of social decision-making, showing how internal states and observable actions co-evolve.
• Limitations: The current datasets consist of acted interactions; spontaneous interactions may present higher challenges. Additionally, the model does not yet enforce temporal smoothing constraints, which could be beneficial for certain tasks.

In summary, SocialLDG represents a significant step forward in making robots capable of nuanced, context-aware social interaction by treating social perception as a dynamic, interconnected graph rather than a set of isolated classification problems.