Technical Summary: Engagement Process (EP)

Problem Statement

Recent advances in large language models (LLMs), embodied intelligence, and tool-augmented agents have enabled operation in increasingly realistic digital and physical environments. However, a fundamental mismatch exists between the temporal structure of actions and observations in these settings. Traditional interaction formalisms, such as Partially Observable Markov Decision Processes (POMDPs), rely on fixed decision epochs where an observation is paired with a single instantaneous action. In reality, agents often face:

• Action-side timing issues: Deliberation latency (e.g., LLM reasoning time), persistent actions (e.g., navigation commands), and tool calls with delayed results.
• Observation-side timing issues: Asynchronous or sparse arrivals of messages, alarms, and tool responses, alongside continuous or multi-rate sensor feedback.

Current approaches often treat these timing phenomena as implementation details or handle them via external patches (e.g., fixed polling intervals or hand-crafted interruption rules). This obscures the temporal structure necessary for agents to adapt their behavior to explicit time costs, leading to suboptimal policies that may over-deliberate, miss urgent events, or fail to interrupt ongoing processes effectively.

Methodology: The Engagement Process (EP)

The authors propose the Engagement Process (EP), a new interaction formalism that inherits the decision-theoretic structure of POMDPs but fundamentally rethinks the temporal interface.

Core Definition

An EP is defined over a discrete sequence of ticks $T = \{0, 1, 2, \dots\}$ (extendable to continuous time) and specified by $E = (S, A, Y, F, O, U)$:

• State ($S$): Summarizes the environment, ongoing processes, and interface conditions. It can encode persistent action effects and state-dependent observation access.
• Actions as Interventions: Unlike standard POMDPs where an action is a default control at every tick, EP treats actions as timed interventions. The agent selects a set of interventions $A_t \subseteq A$ (which can be empty or contain multiple simultaneous atomic actions).
• Observations as Events: Observations are generated from the state process as information events $Y_t \subseteq Y$, independent of decision steps. They can arrive asynchronously, be empty, or contain multiple events.
• Dynamics: The state evolves as $s_{t+1} \sim F(\cdot | s_t, A_t)$. Crucially, the system evolves even when $A_t = \emptyset$, allowing for the modeling of persistent processes.
• Utility: Instantaneous utility $u_t$ can represent task rewards as well as costs for action execution, sensing, interruptions, and deliberation time.

Key Distinctions

• Decoupling: EP separates the agent's internal decision process from the external timing of observations and interventions. The agent maintains an information state $w_t$ based on the history of intervention sets and observation event sets, allowing it to track pending computations, tool calls, and deadlines.
• Compositional Reuse: The interface supports hierarchical organization. Subsystems within an agent can be modeled as EP-like processes themselves, where module calls are interventions and returned messages are observations, all sharing the same abstract interface.
• Relation to Existing Models:
  • POMDPs: Recovered as a synchronized special case where every tick has exactly one action and one observation.
  • SMDPs/Options: Represented as restricted EPs where interventions create "active" components in the state, but EP allows observations to enter the interface during the execution of these options, not just at termination.

Key Contributions

1. Explicit Temporal Interface: EP provides a unified language for timing phenomena (deliberation latency, delayed feedback, persistent actions, interruptions, multi-rate observations) that are often treated separately or hidden in state design.
2. Separation of Concerns: It decouples the external action-observation interface from the agent's internal organization, allowing components operating at different frequencies (e.g., high-level planners vs. low-level controllers) to coordinate on a shared timeline.
3. Unified Formalism: It offers a homogeneous interface for both agent-environment interaction and internal module-module interaction, facilitating compositional design without requiring separate interfaces for each boundary.

Experimental Results

The authors evaluate EP across three settings:

1. Toy Experiments

• Urgency-Aware Deliberation: In tasks where reasoning accuracy improves with time but consumes a deadline, EP-trained policies learn to balance computation depth against urgency. In contrast, step-trained policies (which treat deliberation as instantaneous) overuse the slowest, most accurate mode, leading to high timeout rates (79.0% vs. 0.0% for EP in single-task settings).
• Hierarchical Patrol with Interruptions: In a patrol task with asynchronous alarms, EP allows the agent to interrupt ongoing navigation or handling routines immediately upon alarm arrival.
  • Module-level: EP resolves 90.9% of alarms compared to 51.5% for a baseline that only decides at module boundaries.
  • State-level: EP outperforms hand-written patches and boundary-based baselines by learning trade-offs between urgency, distance, phase progress, and interruption costs, achieving significantly higher returns and lower expiration rates.

2. LLM-Based Experiments

• Digital Assistant (Asynchronous Emails): An agent must complete a main task while handling time-sensitive emails.
  • EP treats main-task generation as an interruptible action, allowing the agent to triage emails as they arrive.
  • Baselines: AgentLoop (synchronized) misses many deadlines; Periodic Poll improves but is limited by fixed intervals.
  • Result: EP achieves the highest utility and balanced scores, reducing email timeout rates by ~40% relative to AgentLoop and reducing first-response latency by nearly half.
• Embodied Scheduling: An agent coordinates tutoring and cooking processes.
  • EP exposes cooking signals immediately during tutoring, allowing the agent to decide whether to interrupt, defer, or continue.
  • Result: EP achieves the highest utility and critical dish success rates, outperforming polling-based and engine-side reflex baselines, particularly in multi-dish scenarios with overlapping deadlines.

3. Time-Aware LLM Training

• The authors trained reasoning models (Qwen3-8B) on math problems with explicit token budgets and urgency instructions.
• EP-aware Training: Rewards are given only if the answer is correct and within the token budget.
• Result: EP-aware models learn to adapt generation length to the budget (e.g., generating ~420 tokens for a 1k budget vs. ~1400 for an 8k budget). In contrast, correctness-only RL models (RLVR) improve accuracy but fail to respect budgets, often generating far more tokens than allowed.

Significance and Claims

The paper claims that the Engagement Process does not seek greater state-space expressivity than augmented POMDPs. Instead, its primary contribution is an explicit temporal interface that decouples actions and observations on a shared timeline.

• Modest Scope: EP is presented as an interaction formalism, not a specific agent architecture or learning algorithm. It leaves memory management, scheduling, and policy learning to the systems built upon it.
• Core Value: By making time explicit at the interface, EP exposes temporal structures (like deliberation latency and asynchronous feedback) that are often hidden in standard step-based formulations. This enables agents to adapt to explicit time costs, handle interruptions more flexibly, and compose subsystems with heterogeneous temporal dynamics without relying on external patches or rigid polling schedules.
• Future Work: The authors suggest future directions include EP-aware architectures, richer continuous-time formulations, and larger-scale evaluations in embodied, hierarchical, and multi-agent settings.