🤖 AI

InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction

The paper introduces InfantAgent-Next, a highly modular multimodal generalist agent that integrates tool-based and pure vision capabilities to enable collaborative, step-by-step computer interaction, achieving state-of-the-art performance on diverse benchmarks including OSWorld, GAIA, and SWE-Bench.

Original authors: Bin Lei, Weitai Kang, Zijian Zhang, Winson Chen, Xi Xie, Shan Zuo, Mimi Xie, Ali Payani, Mingyi Hong, Yan Yan, Caiwen Ding

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bin Lei, Weitai Kang, Zijian Zhang, Winson Chen, Xi Xie, Shan Zuo, Mimi Xie, Ali Payani, Mingyi Hong, Yan Yan, Caiwen Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly clumsy, personal assistant named InfantAgent-Next.

In the world of computer automation, most assistants today are like specialists who only know how to do one thing really well, or generalists who try to do everything but often get confused.

Some assistants are like Librarians: They are great at using a specific list of tools (like "search the web" or "open a file"), but if you ask them to click a button on a screen they've never seen before, they freeze because they don't have a tool for that specific button.
Other assistants are like Artists: They can look at a screen and "see" what's there, but they struggle to do complex math or write code because they are trying to do it all with their eyes alone, without a calculator or a text editor.

InfantAgent-Next is different. It's like hiring a Swiss Army Knife Team instead of a single person.

How It Works: The "Brain and Hands" Team

Instead of one giant brain trying to do everything, InfantAgent-Next breaks every job down into small steps and sends each step to the expert best suited for it.

The Manager (The Planner): When you ask, "Save this webpage to my bookmarks," a high-level "Manager" model reads your request. It doesn't try to click the mouse itself. Instead, it says, "Okay, we need to open the browser, find the bookmark button, and click it."
The Specialists: The Manager then calls in the right expert for the job:
- The Eye (Visual Grounding): If the task is to "click the red button," a specialized vision model looks at the screen, finds the exact pixel coordinates of that red button, and tells the system where to click. It's like a sharp-eyed spotter guiding a blindfolded archer.
- The Calculator (Code Execution): If the task is "analyze this Excel file," the Manager hands the file to a coding expert who writes a script to do the math instantly, rather than trying to count the rows by looking at the screen.
- The Ear (Audio Analysis): If you upload a recording and ask, "What page number is mentioned?", a specialized audio model listens to the file and extracts the answer.
The Memory: The system keeps a shared notebook. Every time a specialist finishes a step, they write it down. The next specialist picks up the notebook, reads what happened, and continues the story. This ensures the team doesn't lose track of the goal.

Why This Matters (The "Magic" Tricks)

The paper highlights two main problems this system solves:

The "Clicking" Problem: Many AI agents are bad at clicking the right spot on a screen. They might click 5 pixels to the left and miss the button. InfantAgent-Next uses a "Zoom-In" technique. If it needs to click a button, it doesn't just guess. It takes a picture of the screen, finds the general area, crops that area to make it bigger, finds the button again, crops it again, and then clicks. It's like using a magnifying glass to find a needle in a haystack, ensuring the click is precise.
The "Editing" Problem: When editing a text file, AI often gets the line numbers wrong (e.g., "Change line 5" when it should be line 6). InfantAgent-Next has a "Double-Check" system. It proposes a change, then looks at the actual text to verify, "Does line 5 actually say what I think it says?" If not, it adjusts its plan before making the edit, preventing messy mistakes.

What It Can Do (The Proof)

The researchers tested this "Team of Specialists" against other top-tier AI agents on three types of challenges:

Real-World Desktop Tasks (OSWorld): They asked the agent to do things like "download a file," "edit a document," and "navigate a website." InfantAgent-Next beat the famous "Claude Computer Use" agent by 7.27%. It was better at actually getting the job done without human help.
Coding and Bug Fixes (SWE-Bench): They asked the agent to fix broken code in real software projects. It performed as well as, or better than, many expensive, closed-source commercial agents.
General Knowledge & Logic (GAIA): They asked complex questions that required searching the web, reading files, and reasoning. InfantAgent-Next ranked second among all open-source agents, proving it can handle tricky, multi-step logic.

The Bottom Line

InfantAgent-Next isn't just one big AI model trying to be everything. It's a modular conductor that knows exactly when to call the "Eye," the "Ear," the "Coder," or the "Mouse-Clicker." By letting each specialist do what they are best at, the whole system becomes much smarter, more accurate, and capable of handling the messy, real-world tasks we face on our computers every day.

The researchers have made the code for this "Team of Specialists" available for anyone to use and study, hoping to help build even better computer assistants in the future.

1. Problem Statement

Current automated AI agents for computer interaction face a fundamental trade-off between generality and accuracy:

Tool-Based Agents: (e.g., OpenHands, AutoGPT) rely on predefined tools (code, search) to boost accuracy on specific tasks. However, they require manual integration of tools for every new scenario, making them brittle and limited in scope. They often fail when encountering software not covered by their toolset.
Pure Vision Agents: (e.g., UI-TARS, Aguvis) use Vision-Language Models (vLLMs) to control GUIs directly without tools. While highly general, they struggle with high-resolution reasoning, often mislocalizing UI elements (clicking the wrong coordinates) or failing at precise tasks like file editing and code manipulation where simple tool calls would suffice.
The Gap: Existing solutions either lack the flexibility to handle diverse, unstructured tasks or lack the precision to execute complex, tool-heavy operations reliably. There is a need for a hybrid paradigm that unifies the precision of tool-based execution with the adaptability of pure vision control.

2. Methodology: InfantAgent-Next Architecture

InfantAgent-Next introduces a highly modular, multimodal generalist agent that integrates tool-based and pure-vision approaches within a unified dialogue context. Instead of relying on a single monolithic model, it routes subtasks to specialized models based on their strengths.

Core Architecture

The system operates in an iterative loop consisting of three distinct roles, each potentially powered by a different model:

Planner: Analyzes the user request and current state to generate a high-level task plan.
Tool Selector: Dynamically selects the most appropriate toolkit (e.g., File Editing, Web Browsing, Computer Use) from a curated library.
Executor: Invokes the selected tools and gathers feedback from the environment.

Key Technical Components:

Modular Workflow & Memory: The agent maintains a unified dialogue history but parses responses using special tags (<thought>, <task>, <toolkit>, <execute_bash>, etc.). This allows the system to reconstruct context specifically for the current subtask (planning vs. execution), preventing context window bloat and irrelevant content generation.
Dynamic Toolset: The system categorizes tools into seven toolkits (File Reading, Searching, Editing, Web Browsing, Computer Use, Code Execution, Advanced Tools). It dynamically selects a subset of relevant tools for each step to reduce inference costs and cognitive load on the LLM.
Multimodal Integration: The agent supports text, images, audio, and video. It can route audio analysis to specialized audio models, visual grounding to vision models, and logical reasoning to advanced LLMs.

Novel Improvements

The paper introduces specific algorithms to address known failure points in SOTA agents:

Iterative Region Cropping for Mouse Clicks (Algorithm 1): To solve the "visual grounding" problem where models misidentify coordinates:
1. The agent takes a full-screen screenshot.
2. A visual grounding model predicts a candidate coordinate.
3. The system crops a smaller region around that coordinate.
4. This process repeats $n$ times (iteratively narrowing the search area).
5. The final refined coordinate is used for the click. This significantly reduces distraction from irrelevant pixels.
Robust File Editing (Algorithm 2): To handle errors in line-number-based file edits:
1. The agent generates a plan with start/end lines and expected content.
2. It verifies if the actual file content matches the expected boundaries.
3. If a mismatch occurs (e.g., off-by-one error), it switches to a fallback strategy: it fuzzifies the expected content strings and searches the file for the best matching span, updating the edit request dynamically before applying the change.

3. Key Contributions

InfantAgent-Next Framework: An open-source, multimodal generalist agent that bridges tool-based and pure-vision paradigms, supporting text, image, audio, and video inputs.
Detailed Modularization: A novel architecture that decouples planning, tool selection, and execution, allowing different models (reasoning, visual grounding, audio analysis) to collaborate seamlessly within a unified context.
Enhanced Execution Mechanisms: The introduction of Iterative Region Cropping for high-precision GUI interaction and a Fuzzy-Match Fallback for reliable file editing, addressing the two main failure modes of current agents.
Open-Source Toolkit: Release of a suite of optimized agent tools and an isolated execution environment (VM-based) to support future research.

4. Experimental Results

The agent was evaluated on three major benchmarks covering vision, logic, and general tasks:

OSWorld (Visual Reasoning):
- Metric: Accuracy on 369 open-ended desktop tasks.
- Result: InfantAgent-Next achieved 35.3% accuracy (50 steps), outperforming Claude-Computer-Use (26.0%) by 7.27% and surpassing other open-source frameworks like Agent-S2. It demonstrated superior ability to ground natural language instructions in GUI elements.
SWE-Bench (Logical Reasoning & Code):
- Metric: Resolving real-world GitHub issues.
- Result: On the SWE-Bench-Verified (50 cases) subset, it achieved 66% accuracy, outperforming several closed-source agents (e.g., Amazon Q Developer Agent at 54%). On SWE-Bench-Lite, it achieved 31.67%, ranking among the top open-source agents.
GAIA (General Task Performance):
- Metric: Open-ended questions across three difficulty levels.
- Result: InfantAgent-Next ranked second among open-source agents (56.97% average), trailing only OWL, and achieved state-of-the-art performance on Level 2 (intermediate) difficulty questions.
Ablation Studies:
- The Iterative Region Cropping mechanism was shown to converge in accuracy by the 3rd iteration, with 2 iterations offering the best trade-off between performance and cost.
- The File Editing logic achieved a 90.4% total success rate on SWE-Bench tasks, successfully repairing 84.3% of failed attempts (e.g., fixing off-by-one line errors).

5. Significance

InfantAgent-Next represents a significant step forward in Generalist Computer Agents. By moving away from the "single model does everything" or "rigid toolchain" approaches, it demonstrates that a modular, multi-model architecture can achieve both high precision in execution (via specialized grounding and editing logic) and broad adaptability (via dynamic tool selection and multimodal understanding).

The work proves that:

Hybridization is key: Combining the reasoning power of LLMs with the precision of specialized vision and tool modules outperforms monolithic models.
Iterative refinement works: Simple algorithmic improvements (like iterative cropping) can drastically solve the "hallucinated coordinate" problem that plagues visual agents.
Scalability: The framework is robust enough to handle complex, multi-step workflows involving code, web browsing, and file manipulation, making it a viable candidate for real-world automation tasks.

The open-sourcing of the code, models, and evaluation scripts provides a strong foundation for the community to further develop multimodal agent systems.