← Latest papers
🤖 machine learning

DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

The paper introduces DocsChisel, an adaptive framework that iteratively optimizes tool documentation by analyzing agent failure traces to dynamically refine information fields, resulting in significant improvements in LLM agent task success rates compared to existing baselines.

Original authors: You Lu, Kun Zhang, Bihuan Chen, Xin Peng

Published 2026-08-12
📖 8 min read🧠 Deep dive

Original authors: You Lu, Kun Zhang, Bihuan Chen, Xin Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your favorite super-smart robot assistant is trying to build a house, cook a gourmet meal, or solve a mystery. To do these things, the robot can't just think; it needs to grab tools. It might need a digital hammer, a virtual wrench, or a high-tech oven. But here's the catch: the robot doesn't know how to use these tools unless someone gives it a manual. In the world of artificial intelligence, these "robots" are called Large Language Model (LLM) agents, and the "manuals" are the tool documentation. For a long time, scientists thought the best way to make these agents smarter was to just give them better brains or teach them how to find tools faster. They treated the manuals as fixed, unchangeable rulebooks. But what if the problem isn't the robot's brain, but the fact that the manual is written in a language the robot doesn't quite understand, or is missing the one crucial step it needs to succeed?

This is the puzzle that a team of researchers from Fudan University decided to crack. They asked a simple but profound question: Does the way we write these tool manuals actually matter, and does a "one-size-fits-all" manual work for every robot? To find out, they didn't just guess; they ran a massive experiment. They looked at thousands of tools and tried different versions of their manuals on different types of AI agents. What they discovered was surprising: there is no single perfect manual. A manual that helps one robot succeed might confuse another. A detail that is helpful in a "cooking" task might be a distraction in a "coding" task. Because of this, they built a new system called DOCSCHISEL. Think of it as a super-smart editor that watches the robot try and fail, then chisels away the confusing parts of the manual and adds the missing pieces, tailoring the instructions specifically for that robot and that specific job. Their results show that by doing this, the robots became much, much better at finishing their tasks.

The Problem: The "One-Size-Fits-All" Manual Doesn't Fit

Imagine you are teaching a new friend how to use a complex video game controller. You might write a guide that says, "Press A to jump." But if your friend is playing a racing game, they might need to know that "Press A to accelerate." If you give them the same generic guide for both games, they will crash the car or fall off the cliff.

For a long time, researchers building AI agents assumed that tool documentation was like a static dictionary. They thought, "If we just make the dictionary shorter or cleaner, the AI will understand better." Some previous attempts tried to compress these manuals into tiny, neat summaries, while others tried to rewrite them to be more standard. But the researchers behind this paper realized something was missing: they were treating the manuals as if they were perfect, just needing a little polish. They suspected that the content of the manuals—the specific fields of information like "how to use this," "what happens if I mess up," or "what the output looks like"—was actually the real key, and that different AI agents needed different keys.

The Detective Work: What's in the Manuals?

To test their theory, the team acted like digital detectives. They gathered a massive collection of 24,955 tools from 14 different datasets. These tools covered everything from managing emails and calendars to analyzing data and even fixing software bugs. They wanted to see what kind of information was actually included in these manuals.

They found a chaotic mess of inconsistency. Some manuals had a section called "Usage Guidance," while others didn't. Some listed the "Input Parameter Name" (what you type in), while others skipped it entirely. They identified 17 different types of information fields that could be in a manual. The only two fields that appeared in every single dataset were the "Tool Name" and a "Functionality Description" (what the tool does). Everything else was hit or miss.

But the real surprise came when they tested how these different fields affected the AI's performance. They took a standard set of tools and started playing "add or remove" with the information fields. They would take a manual, remove the "Usage Guidance," and see if the AI failed. Then they would add it back and see if it succeeded.

The results were wild. The same piece of information could be a hero in one situation and a villain in another.

  • For one AI model, removing a specific field might cause it to fail 10% more often.
  • For a different AI model, removing that exact same field might actually help it succeed because the extra text was confusing it.
  • In a "Data Analysis" task, a specific field might be crucial.
  • In an "Email Management" task, that same field might be useless noise.

The researchers found that, on average, just adding or removing a single piece of information changed the AI's success rate by 6.34 percentage points. This proved that a fixed, unchangeable manual is a bad idea. What works for a "ReAct" agent (a robot that thinks step-by-step) might not work for a "Multi-Agent" system (a team of robots working together).

The Solution: DOCSCHISEL, the Adaptive Editor

Since a fixed manual doesn't work, the team built DOCSCHISEL (Adaptive Tool Documentation Optimization Framework). You can think of DOCSCHISEL as a tireless, hyper-observant editor that sits next to the AI agent while it works.

Here is how it works, step-by-step:

  1. The Watchful Eye: First, DOCSCHISEL lets the AI agent try to do its job using the original, unedited manuals. It watches carefully and records every time the agent fails.
  2. The Diagnosis: When the agent fails, DOCSCHISEL looks at the "failed trace"—the digital footprint of what went wrong. It asks an AI (a "diagnosis" model) to figure out why it failed. Was it because the manual didn't say the tool required a specific input? Was it because the example was confusing?
  3. The Memory: This is the secret sauce. DOCSCHISEL keeps a "memory" of what went wrong in the past. If it sees that "missing usage guidance" caused a failure in the "Email" domain, it remembers that for the next tool in the same domain. It learns from its mistakes.
  4. The Chisel: Based on the diagnosis and its memory, DOCSCHISEL decides to Add, Remove, or Refine specific parts of the manual.
    • Add: "The robot didn't know it needed a password, so let's add a 'Required Input' field."
    • Remove: "The robot got confused by a long code snippet, so let's delete it."
    • Refine: "The instruction was vague, so let's make it clearer."
  5. The Test: It then tests the new, edited manual. If the agent succeeds more often, great! If the new manual causes the agent to fail on tasks it used to do, DOCSCHISEL knows to back off. It keeps the best version of the manual and moves on to the next tool.

The Results: A Massive Leap Forward

The team tested DOCSCHISEL against two other top-tier methods (EASYTOOL and DRAFT) and the original, unedited manuals. The results were staggering.

  • Compared to the original manuals: DOCSCHISEL improved the AI's task success rate by 95.89%. That means the robots almost doubled their ability to get the job done.
  • Compared to the other smart methods: On average, DOCSCHISEL was 75.15% better than the next best approach.
  • Consistency: It didn't just work for one type of robot or one type of task. It worked across different AI models (like GPT-4o, GLM-5, and Claude Haiku 4.5) and different agent styles (single robot vs. team of robots).

The researchers also looked at the cost. Optimizing the manuals takes time—about 12.65 minutes per tool. However, they argued this is a small price to pay for such a huge improvement. The new manuals weren't much longer than the old ones, so they didn't overwhelm the AI with too much text.

Why This Matters

This paper changes the way we think about teaching AI. It shows that we can't just dump a generic instruction manual on a robot and hope for the best. The "manual" needs to be a living, breathing thing that adapts to the robot's brain and the specific job it's doing.

The researchers didn't just suggest this; they proved it with data. They showed that by carefully chiseling away the bad parts and adding the right parts of the documentation, we can make AI agents significantly more reliable. It's like realizing that to teach a student to play the piano, you don't just give them a book of sheet music; you have to tailor the lesson to their specific finger strength, their musical taste, and the specific song they are trying to learn. DOCSCHISEL is the tool that does exactly that for AI agents.

Drowning in papers in your field?

Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.

Try Digest →