FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems

1. Problem Statement

Traditional training of Graphical User Interface (GUI) agents relies on centralized data collection, which faces significant bottlenecks:

• Cost & Scalability: Manual labeling and data aggregation are expensive and difficult to scale.
• Privacy: Real-world user interaction data is sensitive and cannot be publicly shared due to privacy concerns.
• Heterogeneity Gap: Existing Federated Learning (FL) benchmarks for GUI agents (e.g., FedMABench) are limited to single-platform scenarios (typically Android). They fail to address the complex, real-world heterogeneity arising from diverse platforms (Mobile, Web, Desktop), devices (different screen sizes/resolutions), Operating Systems (iOS, Android, Windows, macOS, Linux), and data sources.

The core challenge is how to train a unified, generalizable GUI agent using distributed, privacy-preserving data across these highly heterogeneous environments without degrading performance due to non-IID (non-Independent and Identically Distributed) data shifts.

2. Methodology: The FedGUI Framework

The authors introduce FedGUI, the first comprehensive benchmark designed to simulate and evaluate federated training of GUI agents across diverse real-world conditions.

A. System Architecture

• Framework: Built upon the ms-swift framework, extended to support FL.
• Components: A central server coordinates heterogeneous clients (Mobile, Web, Desktop).
• Algorithms: Integrates 7 representative FL algorithms (FedAvg, FedProx, SCAFFOLD, FedAvgM, FedAdam, FedYogi, FedAdagrad).
• Models: Supports 20+ base models, including state-of-the-art open-source Vision-Language Models (VLMs) like Qwen, InternVL, Gemma, and proprietary models (GPT-4o, GPT-5).
• Action Space: Defines a unified action space of 17 discrete action types (e.g., CLICK, TYPE, SCROLL, HOTKEY) to standardize interactions across different platforms, mapping platform-specific nuances into a common format.

B. Dataset Construction

FedGUI constructs six curated datasets derived from nine source datasets (e.g., AndroidControl, AitW, Mind2Web, OmniAct) to model four specific types of heterogeneity:

1. Cross-Platform Heterogeneity (FedGUI-Platform): Data from Mobile, Web, and Desktop. Includes subsets with varying degrees of platform mixing (IID, Partial, Skew).
2. Cross-Device Heterogeneity (FedGUI-Device): Data from five different Android devices (e.g., Pixel 7 Pro, Tablets) to study variations in resolution and screen size.
3. Cross-OS Heterogeneity (FedGUI-OS): Data from Ubuntu, macOS, and Windows to capture OS-level UI design and interaction differences.
4. Cross-Source Heterogeneity (FedGUI-Mobile, FedGUI-Web): Data from different collection pipelines (e.g., human-collected vs. synthetic) within the same platform.
5. Comprehensive Setting (FedGUI-Full): Combines all sources and platforms to simulate the most realistic, complex federated scenario.

C. Evaluation Metrics

Performance is measured at the action level using three metrics:

• Action Type Accuracy: Correctness of the predicted action category.
• Grounding Accuracy: Spatial correctness for coordinate-based actions (e.g., clicking the right element).
• Success Rate (SR): End-to-end task completion requiring both correct action type and correct grounding.

3. Key Contributions

1. First Unified Benchmark: Introduced FedGUI, the first benchmark enabling cross-platform (Mobile/Web/Desktop) federated learning for GUI agents.
2. Systematic Heterogeneity Modeling: Constructed six datasets specifically designed to isolate and quantify four dimensions of real-world heterogeneity (Platform, Device, OS, Source).
3. Extensive Empirical Study: Conducted experiments across 7 algorithms and 20+ models, providing a deep analysis of how different heterogeneity types impact model convergence and generalization.
4. Open Source Release: Publicly released code, data, and a scalable infrastructure for the community.

4. Key Results & Insights

Extensive experiments yielded several critical findings:

• Cross-Platform Collaboration is Essential:

  • Single-platform models suffer catastrophic failure when tested on unseen platforms.
  • Federated learning significantly restores utility, proving that collaboration across platforms is necessary to bridge domain isolation.
  • Platform Sensitivity Hierarchy: Desktop environments exhibit the highest sensitivity to heterogeneity, followed by Web, then Mobile. Mobile interactions transfer more effectively to other platforms than desktop interactions do.

• Impact of Heterogeneity:

  • Performance degrades consistently as data distribution shifts from IID to highly skewed (Non-IID) settings.
  • Cross-OS heterogeneity poses a more significant challenge than cross-device or cross-source heterogeneity.
  • Cross-application shifts have a smaller impact compared to cross-platform shifts, suggesting platform-level GUI differences are the primary bottleneck.

• Algorithm Performance:

  • Adaptive Optimizers (e.g., FedYogi, FedAdam) outperform standard baselines (FedAvg) and correction-based methods (FedProx), demonstrating superior robustness in cross-platform settings.
  • FL acts as a "salvage effect," elevating unusable local models to a functional level, though a gap with centralized training remains.

• Model Efficiency & Scalability:

  • Compact Models: After federated fine-tuning, compact open-source models (e.g., Qwen3-2B) can outperform large proprietary models in specific GUI tasks.
  • Edge Deployment: Sub-2B models are feasible for deployment on high-end mobile devices with 16GB RAM.
  • Communication Efficiency: Using LoRA (Low-Rank Adaptation) reduces communication overhead significantly (from ~16GB/round for full fine-tuning to ~84MB/round).

5. Significance

FedGUI bridges the gap between theoretical federated learning research and practical GUI agent deployment.

• Privacy-Preserving: It offers a pathway to train powerful agents using real-world user data without compromising privacy.
• Scalability: It demonstrates that distributed training can scale to diverse environments, overcoming the limitations of centralized data collection.
• Foundation for Future Work: By identifying that platform and OS are the most influential factors, FedGUI guides future research toward developing platform-aware aggregation mechanisms and hierarchical FL architectures.

In summary, FedGUI establishes that while heterogeneity is a major challenge, federated collaboration across diverse platforms is not only feasible but essential for building robust, generalizable, and privacy-preserving GUI agents for the real world.