💻 computer science

M2HRI: An LLM-Driven Multimodal Multi-Agent Framework for Personalized Human-Robot Interaction

The paper introduces M2HRI, an LLM-driven multimodal multi-agent framework that enhances personalized human-robot interaction by equipping robots with distinct personalities and long-term memory while employing a coordination mechanism to manage their collective behavior, a design validated by a user study showing significant improvements in interaction quality and personalization.

Original authors: Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shaid Hasan, Breenice Lee, Sujan Sarker, Tariq Iqbal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine walking into a room where two robots are waiting to chat with you. In the past, if you had two robots, they would act like identical twins wearing the same uniform, speaking in the same monotone voice, and forgetting who you were the moment you left the room. They would often talk over each other, creating a chaotic mess.

This paper introduces M2HRI, a new system that changes the game. Think of M2HRI not as a factory line of identical machines, but as a well-rehearsed improv comedy troupe.

Here is how it works, broken down into three simple ingredients:

1. The Personalities (The "Characters")

In the old days, robots were like blank slates. M2HRI gives each robot a distinct personality, like characters in a movie.

The Analogy: Imagine one robot is the "Chatty, Emotional Friend" (who gets excited and talks about feelings), and the other is the "Calm, Organized Librarian" (who is quiet and loves facts).
How it works: The researchers used advanced AI (called Large Language Models) to give these robots "Big Five" personality traits. They found that humans can easily tell the difference between these robots. You don't just hear two machines; you hear two people with different vibes.
The Catch: Some personalities are easier to spot than others. It's easy to spot an "excitable" robot, but a "responsible" robot might be harder to distinguish in a short chat because being responsible is often shown through long-term actions, not just immediate words.

2. The Memory (The "Diary")

Before this system, robots had no long-term memory. If you told a robot your favorite color was blue today, and you came back tomorrow, the robot would act like it had never met you.

The Analogy: Without memory, a robot is like a goldfish that forgets everything after 10 seconds. With M2HRI, the robot has a digital diary.
How it works: The system lets robots remember your preferences (like your favorite food or hobbies) and past conversations. When you return, the robot says, "Hey! You mentioned you love sci-fi movies last time."
The Result: This makes the interaction feel personalized. It doesn't necessarily make the conversation flow smoother (that's the next part), but it makes the robot feel like a friend who actually knows you, rather than a stranger.

3. The Coordinator (The "Director")

This is the most critical part. If you give two robots personalities and memories but don't tell them how to work together, they will crash into each other. They might both try to answer your question at the exact same time, or one might interrupt the other.

The Analogy: Without a coordinator, it's like a jam session where everyone plays at once. With a coordinator, it's like a movie director on set.
How it works: M2HRI has a "central brain" (the Coordinator) that watches the conversation. It looks at the robots' personalities and the current situation to decide: "Okay, the 'Librarian' robot is the best one to answer this math question, and the 'Chatty' robot should wait its turn."
The Result: This stops the robots from talking over each other. It ensures the conversation flows naturally, like a polite human group discussion, rather than a chaotic shouting match.

What Did They Find?

The researchers tested this with 105 people (watching videos of these robot interactions) and found three big things:

Personalities Work: People could easily tell the robots apart and felt more engaged when the robots had distinct "voices."
Memory Matters: Robots that remembered your past preferences felt much more helpful and personal.
Coordination is Key: The "Director" was essential. Without it, the interaction felt broken and annoying. With it, the robots felt like a cohesive team.

The Big Takeaway

To build a future where robots hang out in our homes or hospitals, we can't just build smart machines. We need to build teams of unique individuals who remember us and know how to take turns.

M2HRI proves that for robots to feel truly social, they need to be more than just code; they need to have identity, history, and teamwork.

1. Problem Statement

While Human-Robot Interaction (HRI) has advanced significantly in single-agent scenarios, multi-agent HRI remains underdeveloped. Existing multi-robot systems typically treat robots as functionally identical, interchangeable units. This approach overlooks two critical factors in social environments:

Agent Individuality: In shared spaces, robots need distinct identities (personality and memory) to shape user perception and engagement. Without individuality, a multi-robot system feels like a collection of identical machines rather than a team of distinct social agents.
Coordination Complexity: Introducing individuality complicates coordination. Simple turn-taking rules are insufficient when robots have different personalities and histories. The system must decide which robot should respond based on identity, context, and social appropriateness to avoid chaotic overlaps and ensure coherent group behavior.

The paper addresses the gap in modeling agent individuality (personality and long-term memory) alongside structured coordination in a multimodal multi-agent setting.

2. Methodology: The M2HRI Framework

The authors propose M2HRI, a framework built on Large Language Models (LLMs) and Vision-Language Models (VLMs) that integrates perception, memory, personality, and centralized coordination.

A. Agent Architecture

Each robot agent ( $a_i$ ) is modeled as a composition of an Identity Profile ( $\phi_i$ ) and a Cognitive Loop ( $\Lambda_i$ ):

Identity Profile ( $\phi_i$ ):
- Memory ( $M_i$ ): Divided into Working Memory (short-term context), Semantic Memory (factual knowledge), and Episodic Memory (user preferences and past interactions). Each agent maintains a private long-term store.
- Personality ( $B_i$ ): Modeled using the Big Five Factor Model (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism). Traits are discretized (1–5) and mapped to natural language descriptors that condition the LLM's reasoning and response generation.
- Action Capability ( $\Gamma_i$ ): The set of executable primitives (speech, gestures, locomotion).
Cognitive Loop ( $\Lambda_i$ ):
- Perception: Uses a VLM to process multimodal input (speech transcription + visual frames) into a unified semantic observation.
- Planning: An LLM-based planner generates an embodied interaction policy (ordered list of actions) based on the observation, memory, and personality.
- Execution: Maps the policy to physical robot behaviors (e.g., NAO robots).

B. Centralized Coordination Mechanism

To manage interaction among multiple agents, M2HRI employs a centralized coordinator:

Input: Aggregates local observations, interaction context, and personality vectors from all agents.
Process: An LLM evaluates the "response likelihood score" ( $r_{i,t}$ ) for each agent based on who is most socially and contextually appropriate to speak.
Output: Selects agents exceeding a threshold to respond sequentially, preventing overlap and ensuring coherent turn-taking.

C. System Implementation

Hardware: Two NAO humanoid robots equipped with cameras and microphones.
Software: Cognitive processing is offloaded to an external workstation.
- VLM: For scene grounding and observation generation.
- Memory: Implemented via ChromaDB (semantic/episodic) with text embeddings, managed by a LangGraph agent.
- Planning/Coordination: Powered by GPT-4o-mini (via AutoGen) and an LLM-based coordinator.
- Communication: Structured JSON tool calls serialized over ZMQ to NaoQi proxies.

3. Key Contributions

Framework Design: Introduction of M2HRI, the first framework to jointly model distinct agent identities (personality + long-term memory) and centralized coordination in a multimodal multi-robot setting.
Theoretical Insight: Demonstrates that treating robots as interchangeable is a limitation; distinct identities are necessary for users to perceive robots as unique social partners.
Empirical Validation: Conducted a controlled user study ( $n=105$ ) to rigorously evaluate the impact of personality, memory, and coordination on interaction quality.

4. Experimental Results

The study evaluated three Research Questions (RQs) using a video-based observer approach with 105 participants across seven conditions.

RQ1: Effect of Personality (Distinguishability & Engagement)

Distinguishability: Users could reliably distinguish between robots with different personalities. Traits like Openness and Neuroticism were highly distinguishable (Mean $\approx$ 4.8/5), while Conscientiousness was less salient in short interactions.
Consistency: Robots maintained consistent personalities throughout interactions (all $p < .001$ ).
Engagement: High expressiveness traits (Openness, Neuroticism) significantly boosted engagement. Less expressive traits (Conscientiousness, Agreeableness) had a weaker effect on engagement.
Recognition: Participants correctly identified intended traits at rates significantly above chance (up to 93.3% for Openness/Extraversion).

RQ2: Effect of Long-Term Memory

Personalization: The "with-memory" condition significantly outperformed the "without-memory" condition in recall accuracy ( $p < .001$ ) and preference awareness ( $p < .001$ ).
Naturalness: Memory improved interaction naturalness, though the effect was smaller than for personalization. This suggests that while short-term context (working memory) maintains basic flow, long-term memory is critical for personalized continuity.

RQ3: Effect of Coordination

Coherence: Centralized coordination significantly improved conversational flow and response appropriateness.
Overlap Avoidance: The most significant impact was on preventing interruptions. The "without-coordination" condition suffered from frequent overlaps, whereas the coordinated condition maintained strict turn-taking.
Conclusion: Coordination is essential to prevent the "chaos" of simultaneous responses and to ensure the interaction feels socially organized.

5. Significance and Conclusion

The paper establishes that effective multi-agent HRI requires a dual approach:

Individuality: Robots must possess distinct personalities and memories to be perceived as unique social agents, which enhances user engagement and personalization.
Coordination: A structured coordination mechanism is necessary to manage these individual agents, ensuring that their interactions are coherent, non-overlapping, and socially appropriate.

Implications:

Design: Future social robot teams should not be homogeneous; they should be designed with distinct "personalities" and shared memory architectures.
Architecture: Centralized coordination is superior to decentralized, local-context-only decision-making for maintaining social norms in multi-robot dialogue.
Future Work: The authors plan to explore adaptive personality modeling and real-world long-term deployments to further refine these dynamics.

In summary, M2HRI demonstrates that combining LLM-driven individuality with structured coordination creates a more coherent, personalized, and socially acceptable multi-robot interaction experience.