🤖 AI

Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

This paper introduces POLAR, a multimodal memory-augmented framework that organizes long-term user interactions into a knowledge graph to enable embodied agents to effectively retrieve personalized context for improved task execution and reasoning over time.

Original authors: Jeongeun Lee, Chanyoung Park, Dongha Lee

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeongeun Lee, Chanyoung Park, Dongha Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot butler. This robot is great at following instructions like, "Go get the red cup" or "Find the shoes." It can see the world and understand language. But here's the problem: in the real world, people don't always speak in clear, technical terms.

If you ask your robot, "Bring me my shoes," and you have three different pairs of shoes in the house, the robot gets stuck. It knows what a "shoe" is, but it doesn't know which pair you mean. Maybe you always wear the blue sneakers when it rains, or the high heels for your Tuesday meetings. Without knowing your personal history, the robot is just guessing.

This paper introduces a new system called POLAR to solve this problem. Think of POLAR as giving the robot a special, organized diary that helps it remember your specific habits and preferences over a long time.

Here is how POLAR works, broken down into simple parts:

1. The Problem: The Robot's Short Memory

Current robots are like people with very short attention spans. If you ask them a question based on something you said weeks ago, they often forget the details. They might remember the general idea ("Oh, you like shoes"), but they can't remember the specific story ("You bought those blue sneakers on a snowy day last winter"). Because they can't connect the dots, they often grab the wrong item.

2. The Solution: POLAR's "Smart Diary"

POLAR doesn't just save a video recording of every time the robot moved around (which would be a messy, unorganized pile of data). Instead, it organizes your history into a structured memory graph with two specific types of notes:

The "Who and What" Notes (Semantic Memory):
Imagine a card for every object in your house. On the card for your "blue sneakers," the robot writes down specific facts: "Bought last winter," "Used on snowy days," "Worn for morning runs." These are short, clear facts about you and your relationship with that object.
The "How I Found It" Notes (Episodic Memory):
This is like a travel log. It remembers the robot's past journeys. For example: "Last time I looked for these sneakers, I checked the living room first, but they weren't there. I found them in the garage." This helps the robot know where to look next time, saving it from wasting time in the wrong rooms.

3. How It Works in Real Life

When you give a vague command like, "Bring me my snow-day shoes," here is what POLAR does:

It Reads the Clues: It looks at your new request and searches its organized diary.
It Finds the Match: It sees the note on the "blue sneakers" card that says "Used on snowy days." It realizes, "Ah, the user means these shoes, not the red ones!"
It Plans the Route: It checks its travel log and remembers, "Last time, I found them in the garage, not the bedroom." So, it skips the bedroom and heads straight to the garage.

4. What the Experiments Showed

The researchers tested this system with different robot "brains" (AI models) in a simulated house. They compared three approaches:

The "No Memory" Robot: Only looks at what it sees right now. (It fails often because it doesn't know which shoe you mean).
The "Raw Data" Robot: Tries to read through all its past video logs and conversations at once. (It gets confused by too much messy information).
The POLAR Robot: Uses the organized diary.

The Results:
The POLAR robot was much better at finding the exact item you wanted, especially when there were many similar items (like three pairs of shoes) or when your request was very vague. It also figured out the fastest path to the item because it remembered where it had looked before.

The Big Takeaway

The paper concludes that for a robot to be truly helpful in the long run, it can't just have a "long memory" (storing everything). It needs a smart memory (organizing information into useful facts and experiences). POLAR turns messy past interactions into a clear guide, allowing the robot to understand not just what you are asking for, but who you are asking for it.

Technical Summary: POLAR – Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

1. Problem Statement

Multimodal Large Language Model (MLLM)-based embodied agents have demonstrated potential in solving complex physical tasks. However, existing approaches primarily focus on generic instructions and category-level object recognition. In real-world scenarios, personalized assistance often requires instance-level grounding based on implicit context accumulated over long-term interactions.

When a user issues a request like "bring me my shoes" in a home with multiple pairs, the agent must determine the specific intended instance based on prior history (e.g., "the shoes I wore on my last run"). Current MLLM-based agents face two primary challenges in this setting:

Insufficiency of Current Inputs: Relying solely on the current instruction and visual observation is often inadequate for resolving ambiguous, personalized references.
Ineffectiveness of Raw Context: While prior interactions contain necessary user-specific information, raw interaction logs are unstructured, noisy, and difficult for MLLMs to reason over directly, often leading to performance degradation in long-context settings.

2. Methodology: The POLAR Framework

The authors propose POLAR (Personalized Long-term Agent with Retrieval), a multimodal memory-augmented framework designed to organize prior interactions into structured, task-relevant memory. POLAR operates in two distinct stages: Memorization and Utilization.

A. Memory Representation (Multimodal Knowledge Graph)
POLAR structures memory as a graph $M = (V, E)$ containing three node types:

Object Nodes: Unique identities for objects (e.g., Object ID).
Semantic Memory Nodes: These capture personalized context associated with an object. Instead of storing raw logs, POLAR reformulates interactions into concise, fine-grained textual statements describing single user-specific attributes (e.g., "used on snow days"). This prevents entanglement of information and aids retrieval accuracy.
Episodic Memory Nodes: These store the agent's trajectory, raw user instruction, and reference images. Crucially, raw trajectories are converted into compact textual summaries that capture planning-relevant experience (e.g., success/failure status, unproductive rooms, informative observations) rather than preserving every low-level step.

Edges in the graph are timestamped to support temporal updates and recency-aware retrieval. If a new interaction matches an existing object or semantic node, the graph is updated incrementally rather than duplicating entries.

B. Utilization and Planning
During task execution, POLAR retrieves relevant memories to guide the agent:

Retrieval: Given a current instruction $I_k$ , the system retrieves top- $k$ semantic memory nodes using embedding similarity (via BGE-M3 encoder).
Grounding: The agent traverses graph connections from retrieved semantic nodes to corresponding object nodes, gathering associated instructions and episodic memories.
Reasoning & Planning: The MLLM policy reasons over the retrieved candidate objects to ground the intended target instance. It then leverages the associated episodic memory (past search experiences) to guide high-level path planning and low-level action selection.

The embodied agent itself follows a hierarchical planning framework:

High-level Planning: Generates a coarse path (sequence of rooms/waypoints) based on a scene graph and global reasoning.
Low-level Planning: Selects executable actions (Move, Turn, Stop) based on egocentric visual observations.

3. Experimental Setup

Dataset: The evaluation is built upon PinNED, a personalized instance grounding dataset in Habitat-Matterport3D (HM3D) scenes.
Scenarios: Three evaluation scenarios were designed:
- Compositional: Requires combining evidence from multiple prior episodes (Single vs. Joint).
- Distractor: Requires distinguishing the target from semantically/visually similar objects of the same category.
- Temporal: Tests the ability to handle evolving contexts or changes in object associations over time.
Baselines:
- No-prior: Uses only current instruction and observations.
- Raw-interaction: Provides the agent with 15 randomly sampled raw prior interaction logs.
Backbones: Evaluated across multiple MLLMs, including open-source models (Qwen3-VL, Qwen2.5-VL) and proprietary models (GPT-5, GPT-4o-mini, Gemini-2.5-Flash).

4. Key Results

Performance Gains: POLAR consistently outperforms both the no-prior and raw-interaction baselines across all MLLM backbones and scenarios.
- The no-prior baseline often fails, confirming that current observations alone are insufficient for personalized grounding.
- The raw-interaction baseline shows inconsistent improvements; in some cases, unstructured context introduces noise that hinders performance.
Instance Grounding: POLAR significantly reduces Category Match (CM) errors (reaching the correct category but wrong instance), demonstrating superior ability to resolve user intent among same-category candidates.
Memory Design Analysis:
- Semantic Memory: Provides finer-grained retrieval compared to sparse or dense retrieval over raw interactions, isolating request-relevant information more effectively.
- Episodic Memory: Structured summaries of past trajectories yield higher success rates than raw trajectories or simple instruction-only retrieval, proving that extracting planning-relevant experience is critical.
Temporal Robustness: POLAR performs notably better in the Joint and Temporal scenarios, indicating its object-centric graph effectively connects distributed information and handles context evolution.

5. Significance and Claims

The paper claims that effective long-term personalization for embodied agents requires more than simply extending context windows or storing raw interaction logs. The core contribution is the demonstration that structuring memory is essential.

Specifically, the authors assert that:

Structured Memory is Necessary: Converting unstructured prior interactions into task-relevant memory (semantic and episodic) is crucial for reliable instance grounding and planning.
Dual Memory Roles: Personalized assistance requires distinct memory types: semantic memory for fine-grained user-specific attribute retrieval and episodic memory for reusing past search experiences to guide future planning.
Implicit Context Resolution: Agents must be capable of interpreting implicit user intent accumulated over time, a capability that current MLLMs lack without structured memory augmentation.

The work concludes that POLAR bridges the gap between generic instruction following and true personalized assistance by organizing multimodal history into a retrievable, reasoning-friendly knowledge graph.