Technical Summary: OmegaUse-OfficeVal

Problem Statement

Large Language Model (LLM) agents are increasingly expected to transition from natural language interaction to delivering concrete work products, specifically in the realm of "vibe working"—completing long-horizon office-suite tasks. However, existing benchmarks fail to adequately evaluate whether agents can perform these workflows at a reasonable economic cost. Current productivity benchmarks (e.g., GDPVal, RLI) often lack task-level economic annotations or restrict access to reference outputs. Office-automation benchmarks (e.g., OfficeBench, OdysseyBench) typically focus on short-horizon, synthetic, or bounded operations without measuring human labor time or market value. Furthermore, GUI-agent benchmarks often prioritize reaching a specific environment state over the quality and usability of the final deliverable. Consequently, there is a lack of rigorous evaluation frameworks that determine if an agent can produce a correct, usable artifact for a long-horizon office task while quantifying the associated human labor and economic value.

Methodology

Benchmark Construction: OmegaUse-OfficeVal

The authors introduce OmegaUse-OfficeVal, a benchmark comprising 100 long-horizon office-suite tasks derived from authentic practitioner requests. The construction pipeline involves four key stages:

1. Task Collection & Screening: Starting with 1,715 practitioner-proposed tasks, a funnel-based screening process filters for real-world relevance, clarity, and non-triviality. A final set of 100 tasks is selected based on agreement among senior experts, ensuring they represent workflows typically assigned to junior workers or interns but requiring sustained execution.
2. Privacy-Preserving Adaptation: Task instructions are rewritten to remove sensitive information while preserving user intent. Input files (DOCX, PPTX, XLSX, PDF, images, videos) are reconstructed using LLM assistance and manually revised to ensure realism and fidelity to the original task requirements.
3. Economic Grounding: Each task is annotated with two economic signals:
   • Human Labor Time: Recorded time for qualified human annotators to complete the task without LLM assistance. The final value is the average of the two shortest valid completion times per task (collected from 20 annotators).
   • Task Price Proxy: An estimated market price. For ~20% of tasks, explicit prices from practitioners are used. For the remainder, three domain experts provide independent estimates, which are aggregated using a consistency-based outlier rejection method to derive a final price.
4. Code-Based Verification: To ensure reproducible and stable evaluation, the authors avoid human or LLM-as-judge scoring. Instead, they develop code-based verifiers derived from fine-grained rubrics.
   • Rubrics: Each task has a usability rubric (checking if the artifact is valid, openable, and editable) and a task-completion rubric (rewarding completed requirements and penalizing unintended damage).
   • Scoring: A task must pass all usability checks to receive a non-zero score. The final score is a normalized weighted sum of task completion items, clipped at zero to prevent negative scores.

Experimental Setup

The benchmark evaluates several frontier LLMs (GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, Qwen3.7-Plus) against a human baseline. The human baseline uses the highest-scoring submission from the annotation process to represent a "best junior-worker" reference. Evaluation metrics include:

• Score: Aggregate task completion quality.
• Time-Weighted Score: Performance weighted by human labor time.
• Price-Weighted Score: Performance weighted by task price proxy.
• Efficiency: Cost and runtime per task.

Key Results

Performance vs. Human Baseline

• Quality Gap: The human baseline achieves a score of 27.79, substantially outperforming all evaluated LLMs. The best-performing model, GLM-5.2, achieves a score of 17.91.
• Failure Rates: Humans produce the fewest zero-score tasks (29%). Among LLMs, GLM-5.2 has the highest share of high-score tasks (>50), while DeepSeek-V4-Pro and MiniMax M3 exhibit high zero-score rates (50% and 51%, respectively), indicating frequent complete failures on complex tasks.
• Efficiency: All evaluated LLMs are significantly cheaper and faster than human workers. For instance, Qwen3.7-Plus costs only $0.22 per task and takes 0.19 hours, compared to the human average of $6.86 and 2.32 hours.

Economic and Value-Weighted Insights

• Value-Weighted Discrepancy: The model with the highest average score (GLM-5.2) is not necessarily the one that captures the most economically important tasks. Qwen3.7-Plus achieves the highest time-weighted and price-weighted scores, suggesting it performs relatively better on high-value or labor-intensive tasks despite a slightly lower raw average score.
• Difficulty Correlation: There is a negative correlation between human labor time and task scores. Tasks requiring more human effort tend to receive lower scores from both humans and LLMs, with the decline being more pronounced for LLMs. This indicates that current agents struggle to maintain state and quality over extended, complex workflows.

Contributions

1. OmegaUse-OfficeVal Benchmark: A fully open-sourced dataset of 100 authentic, long-horizon office-suite tasks with multimodal inputs and outputs, covering diverse domains and operation intents.
2. Task-Level Economic Grounding: The introduction of human labor time and task price proxy annotations for every task, enabling direct cost-benefit analysis and value-weighted evaluation of agent performance.
3. Code-Based Verification Protocol: A deterministic evaluation framework using fine-grained rubrics and executable code to ensure stable, reproducible, and objective scoring of final deliverables, avoiding the subjectivity of human or LLM-as-judge evaluation.
4. Empirical Evaluation: A comprehensive assessment showing that while current LLM agents offer significant efficiency gains (lower cost and time), they have not yet reached human-level deliverable quality in long-horizon office workflows.

Significance

The paper positions OmegaUse-OfficeVal as a rigorous testbed for tracking progress toward reliable "vibe working." By shifting the evaluation focus from environment state reachability to the quality and usability of final office artifacts, and by grounding tasks in economic reality, the benchmark addresses critical gaps in current productivity-agent research. The authors claim that the benchmark and its released assets (code, dataset, rubrics) provide a necessary foundation for developing agents that can not only execute office tasks but do so with a level of quality and economic viability suitable for real-world deployment. The results suggest that while efficiency advantages are already present, the primary challenge for future agents lies in improving deliverable quality to match human standards in complex, long-horizon scenarios.