CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

Technical Summary: CoAdapt-GUI

Problem Statement
Mobile GUI agents currently exhibit brittleness when deployed to applications absent from their source training data. While recent advances have improved performance on established benchmarks, these evaluations often assume a predefined collection of applications and workflows. In real-world deployment, agents encounter "unseen" applications where both interface elements and task procedures are novel. Existing approaches to generalization, such as online reinforcement learning (RL) on target interactions, often focus solely on policy adaptation. However, success on new tasks within a familiar interface does not guarantee effective behavior in an unfamiliar application. Agents may fail due to interface-specific grounding issues or a lack of procedural workflow knowledge (e.g., completion conditions, failure recovery). Updating only the policy or only the workflow context leaves critical sources of failure unresolved, particularly under cross-application shifts where source experience may contain negative transfer (app-specific details) that hinders adaptation.

Methodology: CoAdapt-GUI
The paper introduces CoAdapt-GUI, a Test-Time Adaptation (TTA) framework designed to jointly adapt a structured workflow context and a policy using only the agent's own target-app rollouts and task-level rewards, without access to target demonstrations or held-out evaluation signals.

The framework operates on two complementary adaptive states:

1. Transfer-Constrained Workflow Context ($M_t$):

   • Separation of Concerns: The system distinguishes between app-bound source state (screens, coordinates, specific identifiers) and transferable procedural knowledge (abstract procedures, failure modes, verification rules).
   • FSM-Grounded Representation: Source workflows are represented as Finite-State Machines (FSMs). A "frozen synthesizer" constructs a transferable state ($M_{tr}$) by filtering out app-bound details via eligibility predicates and schema validators.
   • Target-Grounded Revision: During adaptation, the agent maintains a separate target-grounded state ($M_{t}^{tgt}$). A "frozen reflector" contrasts successful and failed traces from matched rollouts to propose revisions to the workflow context. These revisions are validated against a typed schema (excluding app names, package IDs, etc.) before entering the population.
   • Retrieval: For a target task, the system retrieves bounded, eligibility-constrained source workflow context ($M_0$) and combines it with the evolving target-grounded state.

2. Task–Context-Matched Policy Adaptation:

   • Architecture: The policy is a Vision-Language Model (VLM) with a frozen backbone and a trainable LoRA adapter ($\theta_t$).
   • Group-Relative Optimization: To prevent interference from context variations, policy updates are computed only among rollouts sharing the same task and the same context condition ($G(q, \kappa)$).
   • Objective: The framework uses a group-relative objective (Shao et al., 2024) to update the LoRA parameters based on advantages ($A_j$) calculated within these matched groups. This ensures that credit assignment reflects policy improvements rather than differences in task difficulty or context guidance.

3. Joint Adaptation Loop:

   • The framework executes matched rollouts under controlled reset conditions.
   • The same rollout stream drives two channels:
     • Context Channel: Updates ratings of context variants and proposes validated child revisions.
     • Policy Channel: Updates the LoRA adapter using the accumulated trajectories, but only when valid task–context comparison groups exist.
   • Updates are interaction-coupled but not jointly differentiable; the current context shapes trajectories for policy learning, while the current policy generates the success/failure signals used to revise the workflow.

Key Contributions

1. Joint Target-Side Adaptation: A TTA framework that updates separate workflow-context and policy states using only autonomous target interactions and rewards, eliminating the need for target demonstrations.
2. Transfer-Constrained Adaptation: A mechanism that separates transferable workflow knowledge from app-bound source state. It coordinates reward-guided updates where context revisions are validated by schema checks, and policy credit is strictly computed within matched task–context groups.
3. Comprehensive Evaluation: The framework is evaluated across two levels of generalization:
   • Instance Generalization: New instances of known task templates (AndroidWorld-Generalization).
   • Task-Type Generalization: Entirely new task templates within unseen apps (AndroidWorld Plus).

Experimental Results
The authors evaluated CoAdapt-GUI against baselines including a Base Policy, Static Context Transfer, Policy-Only TTA, and Context-Only TTA.

• AndroidWorld-Generalization (New Task Instances):

  • CoAdapt-GUI achieved a 45.0% success rate.
  • This outperformed the reported Policy-Only TTA baseline (37.5%) by 7.5 percentage points and the Context-Only TTA (35.0%) by 10.0 percentage points.
  • Static Context Transfer achieved 28.75%.

• AndroidWorld Plus (New Task Templates):

  • This setting tested transfer to task types not seen during adaptation.
  • CoAdapt-GUI raised the overall success rate from a Base Policy of 38.6% to 52.9%.
  • Context-Only TTA improved the base to 48.1%, while Policy-Only TTA only reached 40.0%.
  • In Category-Shared Apps (where source workflows could be retrieved), CoAdapt-GUI reached 70.4%.
  • In Category-Novel Apps (no source workflows available), CoAdapt-GUI still improved success from 29.4% (Base) to 34.3%, demonstrating the efficacy of target-grounded adaptation even without source priors.

Significance and Claims
The paper claims that transfer-constrained workflow context provides substantial gains for novel-app generalization, and that jointly adapting the policy further improves held-out performance. The results suggest that:

1. Workflow Adaptation is Critical: Adapting the workflow context alone (Context-Only TTA) yields significant improvements over static transfer or policy-only updates, particularly in scenarios with new task types.
2. Synergy of Joint Adaptation: The combination of context and policy adaptation (CoAdapt-GUI) achieves the best overall performance, indicating that the two states are distinct but complementary channels for generalization.
3. Robustness to Source Shift: The framework effectively handles scenarios where source workflows are unavailable (Category-Novel Apps) by constructing target-grounded knowledge from scratch.

The authors emphasize that their approach avoids information leakage by using disjoint source and target pools, restricting updates to the adaptation split, and freezing states before evaluation. They conclude that CoAdapt-GUI offers a viable path for deploying GUI agents that can autonomously adapt to unseen applications using limited interaction budgets.