Technical Summary: FocusMem

Problem Statement

GUI agents require memory at two complementary temporal scales: episodic memory (reusable evidence from completed tasks) and working memory (constraints and progress within the current interaction). Existing approaches face a trade-off: replaying full multimodal trajectories is computationally expensive and expands the policy context, while text-based memory loses critical visual or procedural details.

Latent memory offers a solution by compressing multimodal trajectories into a few continuous tokens. However, current latent memory methods suffer from three specific failure modes due to their "one-shot" mapping of a trajectory to a single, fixed latent block:

1. Content Recoverability: Important details may be lost during compression because a single block must serve both reusable experience and current task progress, which have different informational requirements.
2. Conditional Readout: A fixed memory block cannot adapt to different decision stages. The same compressed evidence might need to highlight different parts of a trajectory depending on whether the agent is searching, filtering, or verifying.
3. Evidence Trust: Retrieved trajectories may be irrelevant or mismatched to the current decision, yet fixed latent blocks still influence the policy, potentially misleading the agent.

Methodology: FocusMem

FocusMem addresses these issues by factorizing the latent memory interface into three distinct responsibilities: Content Formation, State-Conditioned Readout, and Evidence Trust. The framework operates with a frozen GUI policy, optimizing only the memory pathway.

1. Evidence Construction

• Episodic Evidence: Retrieved complete trajectories from an episodic bank.
• Working Evidence: Segmented chunks of the expired prefix of the current interaction history.
• Both are mapped to a $K$-token latent block.

2. Factorized Latent Memory

The architecture consists of three stages:

• Stage A: Role-Aware Content Basis (What remains recoverable?)

  • Instead of shared queries, FocusMem introduces role-specific base queries ($Q_{0}^{ri}$) for episodic and working memory.
  • Semantic Supervision: A KL-divergence loss ensures the latent block preserves broadly verbalizable trajectory content.
  • Functional Supervision: A role-specific pseudo-target loss ($\hat{v}^{ri}_i$) ensures the block retains control-relevant information (e.g., reusable procedures for episodic, state changes for working).
  • This stage establishes a stable, role-specific foundation independent of the current decision state.

• Stage B: State-Conditioned Readout (What is exposed now?)

  • A lightweight state–item adapter generates a residual query ($\Delta Q_{t,i}$) based on the current decision state ($u_t$) and the encoded item.
  • The final decision-facing block ($Z^{dynamic}_{t,i}$) is generated by cross-attending the dynamic query to the evidence.
  • This allows the same stored evidence to produce different latent representations depending on the current interaction stage, surfacing only the relevant parts of the trajectory.

• Stage B: Evidence Trust (Should it be used?)

  • A lightweight trust gate scores each decision-facing memory block.
  • During inference, blocks with scores below a threshold $\gamma$ are dropped and removed from the policy's attention.
  • Training uses a straight-through HardConcrete mask and injects high-confidence irrelevant episodic items to teach the gate to suppress mismatched evidence.

3. Optimization

Training proceeds in two stages:

• Stage A: Trains the role-aware content basis using fixed queries and semantic/functional supervision.
• Stage B: Initializes from Stage A, enabling the dynamic query residual and the trust gate. The shared compressor, conditional readout, and trust module are jointly optimized while the policy remains frozen.

Key Contributions

1. Problem Formulation: The authors formulate compact latent GUI memory around three distinct responsibilities: forming recoverable content, adapting its policy-facing view to the current decision, and controlling its influence on the policy.
2. FocusMem Framework: A frozen-policy latent-memory framework that realizes these responsibilities through a role-aware content basis, state-conditioned readout, and an independent block-level trust gate.
3. Comprehensive Evaluation: Matched evaluations across five GUI-agent benchmarks (MMInA-Wikipedia, MMInA-Shopping, DeepShop, WebVoyager, Online-Mind2Web) and targeted diagnostics that isolate the source of performance gains.

Experimental Results

FocusMem was evaluated against agents with no memory, raw trajectory replay, and prior latent memory adaptations (CoMEM-style, HyMEM-style, Mem-W-style).

• Performance: FocusMem consistently outperforms all baselines. It improves task success rates (SR) by +10.8 to +18.0 points over a fully matched "Action-only Fixed" baseline across the five benchmarks.
• Efficiency: It achieves these gains while keeping the memory interface compact (at most 48 latent tokens before gating) and reducing the mean number of executed steps per task.
• Ablation Studies:
  • Content Basis: Combining semantic and functional supervision yields the best recoverability for both general content and role-specific information.
  • Dynamic Readout: While fixed and dynamic readouts perform similarly with concise evidence, dynamic readout is significantly more robust when relevant content must compete with additional trajectory context under a compact bottleneck.
  • Trust Gate: The gate effectively suppresses irrelevant injected evidence, preventing performance degradation that plagues non-gated baselines.

Significance and Claims

The paper claims that effective latent memory depends not merely on compressing past interactions, but on what is retained, what is exposed, and what is allowed to influence the policy. By separating these responsibilities, FocusMem demonstrates that:

• Episodic and working memory serve distinct roles requiring different content retention strategies.
• A single trajectory can provide different information at different interaction stages, rather than relying on a fixed summary.
• Mismatched evidence can be excluded under a bounded context budget, preventing the agent from being misled by retrieved but irrelevant history.

The authors conclude that separating content formation, state-conditioned readout, and evidence trust yields a compact yet stronger memory interface, suggesting that future latent memory systems should explicitly model these distinct mechanisms rather than treating memory as a monolithic compressed block.