🤖 AI

Unified Agent: Managing Interactions across Devices

The paper introduces "Unified Agent," a stateful system that effectively manages cross-device and cross-time interactions by maintaining a compact, action-ready state, demonstrating superior and robust performance over existing agent designs through a newly constructed benchmark.

Original authors: Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei, Xin Lin, Truong Nguyen

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei, Xin Lin, Truong Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the conductor of a very busy, very smart orchestra. In this orchestra, the musicians aren't just playing instruments; they are your phone, your laptop, your tablet, and even a robot vacuum. This is the world of AI agents—computer programs that don't just answer questions but actually do things for you, like booking a table or sending a message.

Usually, these agents are like musicians who only hear the person standing right in front of them. If you whisper a request to your phone, the agent on your laptop has no idea what you said. But what if you want an agent that acts like a true conductor, listening to the whole room and remembering what you did on your phone ten minutes ago, even if you are now holding your tablet? This is the challenge of cross-device interaction. The big question is: how do we teach an AI to remember the "story" of your day across all your gadgets without getting overwhelmed by too much information? If the AI forgets which device you were using, it might ask, "Which phone was that?" instead of just doing the task. This paper explores how to build an agent that keeps a perfect, compact memory of your interactions so it can act seamlessly, no matter which device you pick up next.

The Problem: The "Amnesiac" Agent

Picture this: You are shopping for a new restaurant. You browse menus on your phone, then switch to your laptop to read reviews, and finally check your tablet for the location. Later, you sit down with your phone and say, "Book a table at the restaurant I was just looking at."

If you talk to a standard AI agent today, it might panic. It only sees your phone right now. It doesn't know you were looking at a restaurant on your laptop five minutes ago. It might ask, "Which device were you on?" or "Which restaurant?" It's like a waiter who only remembers the order you just placed but has forgotten everything you said while walking into the restaurant.

The authors of this paper argue that current AI systems are missing a crucial ingredient: a compact, carried state. Most agents either try to remember everything (which is too heavy and slow) or they forget everything once the moment passes. They need a way to carry a "backpack" of just the right information—enough to remember what you were doing, but light enough to move quickly.

The Solution: The "Unified Agent"

The researchers built a new kind of AI called the Unified Agent. Think of this agent as a super-organized personal assistant who carries a special notebook.

Instead of trying to remember every single word you've ever said or every screen you've ever looked at, this notebook only writes down three specific things:

Engagement Evidence: "Who was the user touching? Which device did they stare at the longest?" (e.g., "The user spent 5 minutes on the phone.")
Stated Facts: "What facts did the user say?" (e.g., "They mentioned a specific time for dinner.")
The Standing Request: "What is the user currently trying to do?" (e.g., "They want to book a table.")

Every time you look at a screen or say something, the agent updates this notebook. When you later ask, "Book the restaurant I was looking at," the agent doesn't need to guess. It opens its notebook, sees that you were most engaged with your phone, and knows exactly which restaurant you meant. It doesn't need to ask, "Which device?" because the answer is already in its pocket.

The Experiment: A 3D Video Game World

To test if this idea actually works, the researchers didn't just guess; they built a video game-like world called UA-BENCH. Imagine a virtual 3D house with a computer, a laptop, a phone, and even a robot. They created 100 different scenarios where a virtual person interacts with these devices in different orders.

In these scenarios, the "user" might look at a laptop, then switch to a tablet, and finally ask the agent to do something without saying which device they used. The researchers then pitted their Unified Agent against four other types of AI designs:

Stateless agents: Who only look at the current moment.
Multi-agent systems: Where different AIs try to vote on what to do.
Memory-heavy systems: Who try to remember every single detail.

The Results: The Compact Backpack Wins

The results were clear. The Unified Agent significantly outperformed all the other systems. In the default test, it scored an overall 0.668, while the next best system (a "Full Context" system that tried to remember everything) only scored 0.613.

Here is the magic: The Unified Agent didn't just win because it was smarter; it won because it was efficient.

The "Full Context" system's memory grew like a snowball rolling down a hill, getting 11 times bigger as the conversation went on.
The Unified Agent's memory stayed small and bounded, no matter how long the interaction lasted. It only kept the facts that mattered.

The researchers tested this with different "brains" (different AI models) and different levels of thinking power. In every single case, the Unified Agent stayed ahead. It proved that having a well-organized, compact state is more important than just having a bigger, more complex memory.

Why This Matters

This paper suggests that the future of AI isn't about building bigger, heavier brains that try to remember everything. Instead, it's about building better organizers.

By keeping a "compact, action-ready" summary of what you've done and where you've done it, an AI can finally act like a true partner across all your devices. It stops asking, "Which phone was that?" and starts saying, "Got it, booking the table on your phone." It turns a collection of disconnected gadgets into a single, seamless experience, all thanks to a little bit of smart organization.

Technical Summary: Unified Agent – Managing Interactions across Devices

1. Problem Formulation

The paper addresses the challenge of enabling a single AI agent to manage tasks for a user across multiple heterogeneous devices over time. Current mainstream agent systems face a critical limitation in this scenario: observations are scattered across devices and moments, yet existing architectures are not designed to handle this dispersion effectively.

The Core Challenge: When a user makes a request later (e.g., "Book a table at the restaurant I've been looking at") without specifying a device, the deciding cues (e.g., gaze, pointing, screen engagement) from earlier interactions may no longer be visible.
Limitations of Existing Approaches:
- Stateless/Single-Agent Systems: Treat devices as tools but lack effective cross-time state management. Once a cue passes, the relevant evidence is lost, forcing the agent to ask clarifying questions (e.g., "Which device was it on?").
- Multi-Agent Systems: Coordinate across agents but often fail to maintain the "compact carried state" required for cross-device, cross-time requests. They typically resolve targets based on the present request or static endpoints rather than accumulated engagement evidence.
The Goal: The agent must maintain a well-organized state that integrates engagement evidence, stated facts, and standing requests across devices and time, allowing it to act on device-unspecified requests even when original cues are gone.

2. Methodology: Unified Agent

The authors propose Unified Agent, a stateful agent architecture designed to carry interaction evidence across devices and moments.

2.1 The Carried State ( $S_t$ )

Unlike systems that retain raw history (which grows linearly and competes for context window space), Unified Agent maintains a compact, action-ready state ( $S_t$ ) updated at every moment $t$ . This state is organized into three distinct streams:

Engagement Evidence: Accumulates device-associated interaction cues (e.g., pointing, gaze, hand contact) over time to support the identification of the relevant device.
Stated Facts: Preserves user-provided information by topic and, when identifiable, by device. This ensures facts remain distinguishable across different devices.
Standing Request: Retains the latest requested action.

2.2 The Perceive–Fold–Act Pipeline

The agent operates through a continuous loop (Figure 3):

Observe: The agent receives a local observation ( $O_t$ ) from a device camera and the user's current utterance.
Update (Fold): The agent folds the current observation into the previous state ( $S_{t-1}$ ) to produce an updated state ( $S_t = U(S_{t-1}, O_t)$ ). This process accumulates evidence, records facts, and updates the standing request without storing the full transcript.
Act: The agent decides the next action ( $a_t$ $a_{t}$ ) by reading both the updated state ( $S_t$ $S_{t}$ ) and the current observation ( $O_t$ $O_{t}$ ) ( $a_t = D(S_t, O_t)$ $a_{t} = D (S_{t}, O_{t})$ ).
- The state provides historical context (e.g., which device had the strongest engagement).
- The observation provides immediate context.
- The agent uses this combined input to infer intent, recall relevant facts, and route the request to the appropriate device(s) via a shared language interface.

3. Benchmark: UA-BENCH

To evaluate state designs, the authors constructed UA-BENCH, a reproducible benchmark for cross-device interaction.

Environment: A rendered 3D home scene containing screen devices (computer, laptop, phone) and potentially a fetch robot.
Structure: Episodes follow a designed temporal progression:
1. Initial observation establishes the scene.
2. An earlier interaction establishes user-device engagement via visual/linguistic cues.
3. A later device-unspecified request and recall questions test if the system can retrieve information when immediate visual evidence has changed.
Ground Truth: Evaluation targets are derived directly from the controlled episode specification (not model-generated annotations), ensuring exact ground truth for object identity, visibility, and screen state.
Tasks: The benchmark evaluates six tasks: Detect user/devices, Identify engaged device (Eng), Infer intent (Int), Recall information (Inf), Pick responding device (Rsp), and Decide next action (Nxt).

4. Experimental Results

The authors evaluated Unified Agent against state controls (e.g., Full Context, Self-notes) and published baselines (e.g., Mixture-of-Agents, Mem0, MM-DST) across four Multimodal Large Language Model (MLLM) settings.

4.1 Performance Comparison

Overall Superiority: Unified Agent significantly outperformed all compared systems in the default setting (GPT-5.6-Luna, low reasoning effort), achieving an overall score of 0.668 compared to the next best (Full Context at 0.613).
Specific Gains: The largest advantages were observed in Next Action (Nxt) and Recall (Inf), where Unified Agent maintained performance while other systems dropped significantly when cues were removed.
Efficiency: While "Full Context" (retaining all raw history) performed reasonably well, its state size grew linearly, reaching 11 times the size of Unified Agent's state. Unified Agent's state remained bounded, scaling with observed devices and facts rather than interaction length.

4.2 Ablation Studies

The authors tested key design elements to understand their contributions:

Engagement Evidence: Removing counting or pointing cues significantly degraded the ability to identify the engaged device (Eng).
Prior State: Removing prior state significantly degraded the ability to recall information (Inf).
Device Abilities: Removing per-device ability clauses degraded routing decisions (Rsp).
Conclusion: Different tasks require different information streams; the unified state effectively integrates these complementary roles.

4.3 Generalization and Real-World Validation

MLLM Robustness: The performance advantage of Unified Agent persisted across different MLLM families (Gemini 3.1 Flash-Lite, GPT-5.6-Luna, GPT-5.6-Sol) and varying reasoning efforts.
Real Photographs: In a case study using real photographs of a user interacting with a laptop, tablet, and phone, Unified Agent successfully identified the device with the strongest accumulated engagement and recalled the correct time for a "restaurant I was working on," outperforming baselines that failed to recall the correct device-specific fact.

5. Key Contributions

Problem Formulation: Defined the specific challenge of user-agent interaction across time and devices where cues disappear.
Unified Agent: Introduced a stateful agent that maintains a compact state of engagement evidence, stated facts, and standing requests, acting on this state plus current observations.
UA-BENCH: Constructed a benchmark with exact ground truth for evaluating cross-device, cross-time interaction.
Empirical Validation: Demonstrated that a well-designed state architecture significantly outperforms adaptations of published designs and memory baselines, with robustness across MLLM settings.

6. Significance and Claims

The paper claims that the primary advantage of Unified Agent stems from how the state is represented and used, rather than added system complexity or learned memory mechanisms.

State Design as a Complement: The findings highlight that effective state design is a crucial complement to model strength for cross-device agents.
Action-Ready Summary: By organizing evidence into a compact, action-ready form, the agent can resolve references (e.g., "the restaurant I was looking at") even when the original visual cue is gone, without needing to query the user for clarification.
Privacy and Efficiency: The design offers a privacy-preserving alternative to raw history retention. By keeping an explicit, compact record of engagement and facts rather than the full interaction transcript, the system minimizes data retention while remaining inspectable and revisable.

The authors conclude that Unified Agent demonstrates that maintaining a bounded, structured state allows agents to effectively manage interactions across devices and time, solving a gap in current mainstream agent systems.