ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation
ClawTrace introduces a cost-aware tracing platform that generates detailed TraceCards to enable the CostCraft distillation pipeline, which effectively reduces LLM agent costs by 32% through targeted pruning of expensive, redundant steps while preserving successful behaviors.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very smart, but expensive, robot assistant (an LLM agent) that tries to solve complex problems, like organizing a spreadsheet or writing code. Sometimes, this robot gets the job done perfectly. Other times, it fails or takes a wildly inefficient path, burning through money on every step it takes.
The paper introduces a new system called ClawTrace and a method called CostCraft to teach this robot how to be smarter and cheaper, without needing to retrain its brain.
Here is the breakdown using simple analogies:
1. The Problem: The "Blind" Teacher
Imagine a teacher trying to improve a student's study habits by looking at their test papers.
- Old Method: The teacher only looks at whether the student got an "A" or an "F."
- If the student got an "A," the teacher says, "Great job, keep doing exactly what you did!"
- If the student got an "F," the teacher says, "Fix this mistake."
- The Flaw: This is dangerous.
- Scenario A: The student got an "A" but spent 10 hours studying for a test that only took 1 hour. The old teacher says, "Keep doing it!" because the result was good. The student keeps wasting time.
- Scenario B: The student got an "F" because they forgot to write their name. The teacher says, "Fix the name." But maybe the student also wasted 5 hours on a wrong math problem. The teacher misses that waste because they only looked at the final grade.
The paper argues that existing AI training tools are like this "blind" teacher. They know if the AI succeeded or failed, but they don't know how much each step cost (in money and time). Without that cost signal, the AI can't learn to cut out expensive, useless steps.
2. The Solution: The "Receipt" (ClawTrace)
The authors built a tool called ClawTrace. Think of this as a super-detailed receipt printer for the robot's brain.
- Every time the robot talks to its brain (LLM call), uses a tool (like opening a file), or spawns a helper robot, ClawTrace records it.
- It doesn't just say "Task Done." It prints a TraceCard (a small summary) that says:
- "Step 1: Read file. Cost: $0.02."
- "Step 2: Read the same file again. Cost: $0.02. Redundant!"
- "Step 3: Write answer. Cost: $0.01."
- This receipt is compact and easy to read, allowing other systems to analyze the "receipt" without needing the whole messy conversation history.
3. The Teacher: CostCraft (The Three-Action System)
Using these "receipts," a new distillation pipeline called CostCraft creates a set of rules (a "Skill") for the robot. It acts like a coach who gives three specific types of advice:
- Preserve (The "Keep Doing" Rule):
- Analogy: "You got an A, and you did this specific thing right. Keep doing it."
- Source: Taken from successful runs.
- Prune (The "Cut the Fat" Rule):
- Analogy: "You got an A, but you wasted $5 reading the same file twice. Next time, read it once and save the rest. You won't lose the grade, but you'll save money."
- Source: Taken from successful runs that had expensive, unnecessary steps. This is the paper's big innovation.
- Repair (The "Fix the Mistake" Rule):
- Analogy: "You failed because you forgot to check the math. Next time, double-check the math before submitting."
- Source: Taken from failed runs, using a "cheat sheet" (oracle) to see what the right answer was.
4. The Results: What Happened?
The researchers tested this on 30 spreadsheet tasks (like a math exam for robots).
- Cost Matters: When they removed the "cost" info from the receipts, the robot started making expensive mistakes. It would stop working entirely or produce garbage because it didn't know which steps were wasteful.
- The "Prune" Surprise: The most interesting finding was about the Prune rules.
- The researchers thought these rules would just save money.
- Reality: They actually saved the robot's performance. When they removed the "Prune" rules, the robot failed much more often.
- Why? It turns out that when a robot is allowed to be wasteful (reading files twice, checking things it doesn't need), it often gets confused and forgets to write the final answer. The "Prune" rules act like a guardrail, keeping the robot focused so it doesn't crash.
- Cross-Benchmark Test: They tried using the rules learned from spreadsheets on totally different tasks (like writing code or analyzing documents).
- The "Prune" rules (cutting waste) worked great everywhere. They saved money on unrelated tasks too.
- The "Preserve" rules (keeping specific habits) actually made things worse on the new tasks. Why? Because the robot learned habits specific to spreadsheets (like a certain formatting style) that didn't make sense for coding.
Summary
The paper claims that to teach an AI agent to be efficient, you can't just look at the final grade (Success/Fail). You need a receipt (ClawTrace) that shows exactly how much every step cost.
By using this receipt, you can teach the AI three things:
- Keep what works.
- Cut what is expensive but useless (which surprisingly helps the AI stay on track).
- Fix what broke.
Without knowing the cost, the AI learns to be "expensive" and "clumsy," often failing tasks it could have easily solved if it just stopped wasting time.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.