AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios

Technical Summary of "AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios"

Problem
While Large Language Model (LLM) driven AI agents have demonstrated exceptional performance in specialized vertical domains such as coding, deep research, and complex problem-solving, their utility in daily, general-purpose scenarios remains limited and difficult for ordinary users to perceive. Current evaluation benchmarks often prioritize increasing task difficulty in isolation or focus on specific verticals, failing to capture the diversity of agentic tasks required for daily work, life, and learning. Furthermore, existing evaluations frequently lack the necessary diversity to reflect real-world requirements, such as handling various file attachments, inferring implicit instructions, and engaging in iterative refinement. There is a critical gap in assessing whether general users can effectively utilize natural language instructions to complete diverse, tangible, file-based daily tasks.

Methodology
To address these gaps, the authors propose AgentIF-OneDay, a task-level instruction-following benchmark designed to evaluate general AI agents in daily scenarios. The methodology encompasses three core components:

1. Task Categorization: The benchmark structures 104 tasks into three user-centric categories:

   • Open Workflow Execution: Tasks requiring precise, exhaustive execution of explicit, multi-step operational procedures (e.g., travel planning with specific verification steps).
   • Latent Instruction Inference: Tasks where agents must infer implicit rules or constraints from provided attachments (e.g., PDFs, spreadsheets) and apply them to new scenarios without explicit textual instructions.
   • Iterative Refinement: Tasks simulating multi-turn collaboration where agents must modify or expand existing outputs based on user feedback while maintaining state consistency.

2. Data Construction: The dataset was built using a hybrid approach:

   • Human Collection: Experts submitted difficult, objective, and search-resistant questions within their domains.
   • Automated Synthesis: A "File-centered Automated Agentic Pipeline" was developed to expand the dataset. This pipeline extracts workflows from human-authored seeds, searches for and analyzes relevant domain-specific attachments (using tools like ChatGPT), and synthesizes new tasks and rubrics that maintain the original logical structure but vary in content and context.
   • Quality Control: A rigorous multi-stage review process, including expert validation and filtering for spatio-temporal independence and answer verifiability, ensured high data quality.

3. Evaluation Pipeline: The benchmark employs Instance-level rubrics with a refined evaluation pipeline:

   • Scoring: A binary scoring system distinguishes between "Bonus" items (successful completion of key requirements) and "Penalty" items (critical errors). Scores are normalized based on the maximum achievable points for each task.
   • Verification: The pipeline utilizes LLM-as-a-judge (specifically Gemini-3-Pro) enhanced with tools for factual verification (web search), visual parsing (for PPT/HTML files), and file content alignment. This approach achieved an 80.1% agreement rate with human judgment.

Key Contributions

• AgentIF-OneDay Benchmark: A comprehensive dataset comprising 104 tasks and 767 scoring points, covering 767 scoring points across Work, Life, and Study domains, with diverse file formats (PDF, Excel, SVG, etc.) and attachment counts.
• Novel Evaluation Framework: The introduction of a three-dimensional categorization (Open Workflow, Latent Inference, Iterative Refinement) that specifically targets the nuances of daily agent usage, moving beyond simple text generation to file-based, multi-modal task completion.
• Automated Data Synthesis Pipeline: A scalable methodology for generating high-quality, logically coherent agentic tasks by combining human seeds with automated workflow extraction and attachment synthesis.
• Robust Judge Consistency: Demonstration that advanced multimodal foundation models (Gemini-3-Pro) can serve as reliable judges for complex, file-based tasks when paired with specific verification tools, achieving high consistency with human evaluators.

Results
The authors benchmarked four leading general AI agents: Manus, Genspark, ChatGPT-Agent, and Minimax-Agent.

• Overall Performance: Manus achieved the highest overall score (0.645), followed closely by Genspark (0.635) and ChatGPT-Agent (0.626). Minimax-Agent scored lower (0.562).
• Performance Parity: A significant finding is that agents built on leading LLM APIs (relying on prompt engineering and external tools) achieved success rates comparable to custom Agent RL-based systems. This suggests that baseline agentic competence has become a commodity integrated into modern foundational models.
• Domain Specialization:
  • ChatGPT-Agent excelled in Work scenarios (72.18), positioning it as a premier productivity tool.
  • Manus performed best in Life scenarios (73.40), acting as an effective life assistant.
  • Genspark led in Study scenarios (71.19), serving as an ideal learning partner.
• Capability Dimensions:
  • Latent Instruction Inference was identified as the weakest capability across all agents, with even top performers struggling to fully extract and apply implicit formatting rules from attachments.
  • Genspark showed superior Instruction Following (0.766) and handling of Negative Constraints (0.824).
  • Manus demonstrated the highest Factuality (0.731) and robustness across input modalities (performing similarly with and without attachments).
  • Minimax-Agent showed the strongest Logic/Functionality (0.755) but suffered from high latency (1416.2s).

Significance
The paper claims that AgentIF-OneDay provides an empirically valuable reference for understanding the current strengths and bottlenecks of AI agents in daily scenarios. It highlights that while agents are becoming capable of handling complex workflows, significant challenges remain in latent instruction inference and state maintenance during iterative refinement. The benchmark underscores a shift in the competitive landscape: the future advantage lies not in building core agentic skills from scratch, but in optimizing products for specific user needs and leveraging user data. Furthermore, the high-quality, finely annotated instruction data generated by this benchmark holds significant potential as training data for reinforcement learning, aiding in the development of more robust general-purpose agents. The work also validates the feasibility of using advanced multimodal models as automated judges for complex, file-based evaluations, potentially setting a new standard for future agentic benchmarks.