ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
The paper introduces ISE, a three-stage synthesis paradigm that generates high-quality, execution-grounded multi-turn OS-agent trajectories to significantly improve tool-use performance in fine-tuned models, outperforming larger zero-shot baselines.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a very smart but inexperienced robot how to use a computer. You want it to be able to handle complex tasks, like "organize my files, write a report based on them, and email it to my boss."
The problem, according to this paper, is that the "textbooks" (training data) we've been using to teach these robots are flawed in three specific ways:
- They start with the tools, not the user: Most data is built by looking at a list of available computer commands (like "delete file" or "send email") and making up a reason to use them. It's like teaching a chef by saying, "Here is a knife, now imagine a reason to chop something," rather than starting with "I'm hungry, make me a sandwich."
- They are too short: Real life involves back-and-forth. If a robot makes a mistake, you correct it. If it succeeds, you ask for the next step. Most training data is just one question and one answer, like a vending machine transaction, rather than a real conversation.
- They are fake: The robots are usually trained on "simulated" results. The computer pretends to delete a file or guesses what the error message would be. It never actually touches a real computer, so it never learns what to do when things go wrong for real.
The Solution: ISE (Intent → Simulate → Execute)
The authors propose a new three-step recipe called ISE to fix these problems. Think of it as a high-end culinary school for robots.
Step 1: The Menu (Intent)
Instead of starting with a list of tools, they start with real human needs.
- The Analogy: Imagine a chef who creates a menu based on what different types of people actually want to eat, not just what ingredients are in the pantry.
- How they did it: They created 50,000 unique "personas" (like a busy finance manager, a creative writer, or a student) and mixed them with different types of tasks and difficulty levels. This ensures the robot learns to handle a wide variety of real-world requests, not just the easy, common ones.
Step 2: The Role-Play (Simulate)
This is where they fix the "too short" problem. They use a special AI to act as the human user.
- The Analogy: Imagine a method actor playing the role of a frustrated customer. This actor doesn't just say "Make a sandwich" and stop. If the robot drops the bread, the actor says, "Hey, you dropped the bread, pick it up!" If the robot does a good job, the actor says, "Great, now put the cheese on."
- The Secret Sauce: The authors made sure this "actor" AI stays in character. It doesn't accidentally turn into a helpful assistant (which would ruin the training). It stays strictly in the role of the user, reacting to what the robot actually does, not what the robot thinks it did.
Step 3: The Real Kitchen (Execute)
This is the most critical step. The robot doesn't just guess what happens; it actually does the work on a real, isolated computer.
- The Analogy: Instead of a chef pretending to chop vegetables on a cutting board that isn't real, the chef actually chops real vegetables. If the knife slips and cuts the table, the system records that real mistake.
- Why it matters: When the robot tries to run a command and it fails (e.g., "File not found"), the "actor" user sees the real error message and reacts to it. This teaches the robot how to recover from real-world disasters, something fake simulations can't do.
The Results
They took this new "textbook" (called ISETrace) and used it to train a robot (specifically a model called Qwen3-8B).
- The Before: Without this new training, the robot could only solve about 19% of complex, multi-step computer tasks.
- The After: With the new training, the robot solved 37% of the tasks.
- The Comparison: This trained 8-billion-parameter robot actually performed better than:
- A much larger robot (32 billion parameters) that wasn't trained on this data.
- A very famous, expensive commercial robot (GPT-4o) that wasn't trained on this data.
The Takeaway
The paper argues that how you create training data is just as important as what data you create. By starting with real human needs, forcing long conversations, and making the robot practice on real computers (including the failures), they created a much smarter agent.
They also proved that the "long conversation" part was the key. When they cut the training data down to just single questions (removing the back-and-forth), the robot's performance dropped significantly. This shows that learning to handle a conversation is essential for a robot to be truly useful.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.