💻 computer science

ARIS: Agentic and Relationship Intelligence System for Social Robots

This paper introduces ARIS, a modular agentic AI framework that integrates multimodal reasoning, a graph-based Social World Model, and retrieval-augmented generation to enable social robots to maintain contextually grounded, long-term relationships with users, demonstrating significantly higher perceived intelligence and likeability compared to baseline models in a user study with the Pepper robot.

Original authors: Stavya Datta, Fucai Ke, Leimin Tian, Hamid Rezatofighi

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Stavya Datta, Fucai Ke, Leimin Tian, Hamid Rezatofighi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a social robot that doesn't just talk like a chatbot, but actually remembers who you are, knows how you relate to your friends, and can recall your past conversations without getting overwhelmed. That is the goal of ARIS (Agentic and Relationship Intelligence System), a new "brain" designed for social robots like the Pepper robot.

Here is a breakdown of how ARIS works, using simple analogies:

1. The Problem: The "Goldfish" Robot

Current social robots are a bit like goldfish with a very short memory span. They can chat with you for a few minutes, but if you talk to them for hours, or if you bring a friend along, they get confused. They forget who your friend is, they don't remember what you talked about yesterday, and they struggle to understand that you and your friend are a "team." They also get slow and sluggish if the conversation gets too long because they try to remember everything at once.

2. The Solution: ARIS

The researchers built ARIS to fix these issues. Think of ARIS as a super-organized personal assistant that sits inside the robot's head. It has three main superpowers:

A. The "Social Map" (The Social World Model)

Imagine the robot has a giant, living mind map (a graph) drawn on a wall.

How it works: Every time you meet the robot, it draws a dot for you. If you mention your friend "Sarah," it draws a dot for Sarah and connects the two dots with a line labeled "Friend."
Why it matters: If you come back next week, the robot doesn't just see a stranger; it looks at its map, sees the line connecting you to Sarah, and instantly knows, "Ah, this is the person who hangs out with Sarah." This allows the robot to understand relationships (like "husband," "boss," or "best friend") and keep track of people even after many different conversations.

B. The "Smart Librarian" (Retrieval-Augmented Generation)

Imagine the robot has a library of every conversation it has ever had.

The Old Way (Non-RAG): If you ask a question, the old robot tries to read every single book in the library from cover to cover to find the answer. If the library has 10,000 books, this takes forever, and the robot gets tired (slow response time).
The ARIS Way (RAG): ARIS acts like a smart librarian. When you ask a question, the librarian doesn't read the whole library. Instead, they quickly scan the titles and pull out only the top 20 most relevant books (and the 20 most recent ones) to give you the answer.
The Result: The robot stays fast and snappy, no matter how many years of conversations it has stored. It doesn't get overwhelmed by the sheer volume of data.

C. The "Body and Eyes" (Multimodal Reasoning)

ARIS isn't just a voice in a box; it connects the robot's eyes and body to its brain.

Eyes: If you are wearing a red hat, the robot's vision system sees it and tells the brain, "Hey, they have a red hat." The robot can then say, "I like your red hat!"
Body: If you ask for a high-five, the robot doesn't just say "Okay"; it actually raises its arm and gives you a high-five.
The Brain: The "Reasoner" decides when to use the eyes, when to move the body, and when to just talk, based on what is happening in the moment.

3. The Test: Did It Work?

The researchers tested this system with 23 people using a Pepper robot. They compared the ARIS robot against a standard robot (which only used a basic AI chatbot without the memory map or the smart librarian).

The Results:

Smarter Feel: People thought the ARIS robot was significantly smarter and more alive (animacy).
More Human: People felt the ARIS robot was more human-like (anthropomorphism). It felt less like a machine and more like a social companion.
More Likable: People simply liked the ARIS robot more.
Memory Wins: When participants introduced a partner, the ARIS robot remembered the partner's name and relationship much better than the standard robot.
Speed Wins: Even when the conversation history was artificially stretched to thousands of messages, the ARIS robot stayed fast, while the standard robot slowed down to a crawl.

Summary

In short, ARIS gives a robot a long-term memory, a map of human relationships, and a smart way to search its own history. This makes the robot feel less like a tool that forgets everything every time you walk away, and more like a friend who remembers your stories, knows your friends, and can keep up with a long, deep conversation without getting tired.

Technical Summary: ARIS – Agentic and Relationship Intelligence System for Social Robots

1. Problem Statement

While foundational models (LLMs and VLMs) have enhanced social robotics by enabling richer perception and communication, current systems face critical limitations in multi-turn engagement, social-relationship reasoning, and contextually grounded dialogue at scale. Existing architectures primarily target task execution and physical manipulation rather than social intelligence. Consequently, robots struggle to:

Represent or reason about social relationships between individuals (e.g., family, colleagues).
Maintain historical consistency and responsiveness in dense, long-term interactions due to finite context windows and increasing inference latency.
Integrate social cognition with physical embodiment in a unified framework.

2. Methodology

The authors propose ARIS (Agentic and Relationship Intelligence System), a modular agentic AI framework designed to unify multimodal reasoning, a graph-based Social World Model, and Retrieval-Augmented Generation (RAG). The system is deployed on a Pepper Robot (v1.8) using an edge-client-server architecture.

2.1 System Architecture

Edge (Robot): The Pepper robot serves as the user interface, capturing audio-visual streams and executing physical actions (gestures, dances).
Client: A Raspberry Pi 5 that encapsulates audio-visual streams into gRPC packets for transmission.
Server: An NVIDIA RTX 4090 server that processes inputs. It utilizes Whisper-large-v3 for speech transcription and a Grok 2 LLM for reasoning.
Modular Reasoning Engine: A "Reasoner" module consults an API library to select appropriate actions (Speech, Vision, or Physical Embodiment) based on context.

2.2 Core Components

A. Social World Model (SWM)

Structure: A graph-based knowledge graph (Neo4j) where nodes represent Person Nodes (users) and edges represent relationships (e.g., friend, family).
Function: The system extracts relational cues from dialogue using an LLM to infer connections (e.g., "X is Y's supervisor"). It updates the graph via Cypher queries, creating new nodes for unknown users or merging existing ones using Levenshtein distance for name matching.
Attributes: The model stores user attributes (interests, occupation) linked to specific Person Nodes, enabling personalized interaction.

B. RAG-Based Speech Pipeline

Design: To address context window limits and computational costs, ARIS employs a RAG pipeline. Conversational history is stored as Message Nodes (user input + robot response) linked to Person Nodes.
Retrieval Strategy: Upon receiving input, the system retrieves:
1. The 20 most recent messages for immediate continuity.
2. The 20 semantically closest messages (using 1536-dimensional OpenAI embeddings and HNSW indexing) to provide context.
3. Immediate predecessors and successors of the retrieved messages are included, capping the total context at 80 messages.
Execution: The retrieved context is fed to the LLM (Grok 2) to generate a response, which is then appended to the graph.

C. Multimodal Integration

Vision: A Vision Language Model (Grok-2-Vision) acts as a captioner when visual context is required.
Re-Identification (ReID): A module processes 15 frames via majority voting to assign a unique face ID to individuals, linking them to their Person Node in the SWM.

3. Key Contributions

The paper outlines three primary contributions:

Social World Model for HRI: A graph-based representation that explicitly maps and updates human-human relationships, enabling robots to track social connections, re-identify users across encounters, and generate context-aware dialogue grounded in relational data.
Efficient RAG Pipeline: A novel retrieval mechanism using hybrid semantic and recency-aware retrieval. This allows the robot to sustain contextually grounded dialogue over thousands of exchanges while maintaining bounded latency and response relevance.
System Integration and Validation: The unification of these components into a modular agentic architecture that coordinates speech, vision, and physical action. The system was empirically validated against an LLM-only baseline.

4. Experimental Results

The authors conducted two experiments: a user study ( $N=23$ ) and a scalability simulation.

4.1 User Study (Hypothesis H1)

Participants interacted with the robot in dyadic pairs, comparing the ARIS system against an LLM-only baseline (which lacked SWM, ReID, and physical embodiment).

Metrics: Perceived Intelligence, Animacy, Anthropomorphism, and Likeability (Godspeed Questionnaire).
Findings: ARIS significantly outperformed the baseline across all metrics:
- Anthropomorphism: $d = 1.05$ (Large effect, $p < 0.001$ ).
- Intelligence: $d = 0.74$ (Medium effect, $p = 0.0018$ ).
- Animacy: $d = 0.70$ (Medium effect, $p = 0.003$ ).
- Likeability: $d = 0.46$ (Small-Medium effect, $p = 0.0398$ ).
SWM Performance: 7 out of 12 "second-to-interact" participants were successfully recognized by ARIS, and 9 reported the robot "knew about my partner," compared to significantly lower recognition rates in the baseline.

4.2 RAG vs. Non-RAG Scalability (Hypothesis H2)

The authors simulated dense conversations by duplicating message histories up to 14,000 entries.

Latency: The Non-RAG pipeline showed linear-to-superlinear growth in inference time, exceeding 10,000 ms at 14,000 messages. The RAG pipeline maintained a bounded latency below 4,000 ms regardless of history size.
Output Quality: Cosine similarity between RAG and Non-RAG outputs was 83.54%, suggesting that selective retrieval preserves response relevance comparable to full-history access.

5. Significance and Claims

The paper claims that ARIS demonstrates the value of integrating embodied reasoning and structured memory to advance social interaction in robotics. By unifying social reasoning, multimodal perception, and context retrieval, the system achieves measurable improvements in user engagement and perceived intelligence over standard LLM-only baselines.

The authors emphasize that their approach is cost-effective, invoking the Vision Language Model only when contextually required and capping RAG retrieval to reduce token consumption. They position ARIS as a step toward robots capable of "intense conversational depth" and sustained, socially aware engagement in real-world settings.

Limitations Acknowledged:

The evaluation lacks a formal ablation study to isolate individual component contributions.
The scalability test used synthetically duplicated messages rather than organic long-term dialogues, which may not fully capture semantic drift.
The sample size ( $N=23$ ) is modest, and the system was tested on a single robotic platform (Pepper).

The implementation is intended to be released as open source upon publication.