It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
This paper refutes the assumption that optimal harness complexity monotonically decreases with model capability, revealing through a 432-run experiment that harness sensitivity is non-monotone and critically depends on model type, with verbose structures hindering frontier chat models while strict guidance benefits frontier reasoning models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a team of different workers to fix a messy room (the "workspace"). You have a manager who gives them instructions (the "harness").
The common belief in the AI world has been: "The smarter the worker, the less detailed the instructions they need." In other words, a genius needs a simple note, while a beginner needs a strict, step-by-step manual. This paper argues that this belief is wrong.
Here is the breakdown of what the researchers found, using simple analogies:
1. The Experiment: A Test Kitchen for AI
The researchers set up a "test kitchen" called HEAT-24.
- The Room: A digital workspace with 12 files (like code, notes, and data).
- The Jobs: 24 specific tasks, like "fix this broken code," "find a specific file," or "write a report in a strict format."
- The Workers: They tested 6 different AI models, ranging from "Frontier" (super-smart, expensive) to "Constrained" (smaller, cheaper).
- The Instructions: They gave the workers three types of instruction manuals:
- Light: A simple two-sentence note.
- Balanced: A four-step guide with a list of allowed files.
- Strict: A massive, 6-stage manual with strict rules on how to verify the work and exactly how to format the output.
They ran this experiment 432 times to see which combination worked best.
2. The Big Surprise: The "One-Size-Fits-All" Rule is Broken
The researchers expected a smooth curve: Smarter AI = Simpler instructions. Instead, they found a chaotic, non-linear mess. The "best" instruction style depends entirely on what kind of AI you are using, not just how smart it is.
The "Over-Thinker" (Frontier Chat Model)
- Who it is: A very smart model designed for chatting (like a conversationalist).
- What happened: When given the Strict manual, it crashed. Its success rate dropped by nearly 30%.
- The Analogy: Imagine asking a brilliant poet to fill out a complex tax form. If you give them a simple prompt ("Write a poem"), they are perfect. But if you give them a 10-page legal contract on how to write the poem, they get confused, start rambling, and forget to actually write the poem.
- The Result: For this type of AI, less is more. Simple instructions work best.
The "Architect" (Frontier Reasoning Model)
- Who it is: A super-smart model designed for deep logic and problem-solving (with "extended thinking" enabled).
- What happened: When given the Strict manual, it performed better than with simple notes. It also finished the job faster.
- The Analogy: Imagine a master architect. If you say, "Build a house," they might wander off thinking about 100 different possibilities. But if you give them a strict blueprint with exact measurements and a checklist, they get to work immediately, ignore distractions, and build the house perfectly.
- The Result: For this type of AI, more structure is better. It uses the strict rules as a scaffold to focus its thinking.
The "Small but Mighty" (Constrained Model)
- Who it is: A tiny model (only 2 billion parameters) that was surprisingly well-trained.
- What happened: It performed just as well as the huge, expensive models across all instruction types.
- The Analogy: This is like a small, highly disciplined apprentice who, despite having a small brain, was trained so well on how to follow orders that they can do the job as well as a PhD professor.
- The Result: You can't judge a model's "harness needs" just by its size (parameter count). Training quality matters more.
The "Lost Beginner" (Low-Capability Models)
- Who it is: Smaller models with less training.
- What happened: They struggled with everything. Simple notes made them pick the wrong files; strict notes overwhelmed them.
- The Result: They need a "Goldilocks" approach—just enough structure to guide them, but not so much that they get confused.
3. The Two Main Mistakes
The researchers categorized why the AI failed:
- The "Chatterbox" Error (Format Violation): The smart models understood the task but couldn't stop talking. Instead of giving the required data (like a JSON file), they wrote a long story explaining why they did it. This happened mostly when the instructions were too complex.
- The "Wrong Door" Error (Wrong File): The smaller models often didn't know which file to touch. Without a clear list of "allowed files," they would edit the wrong document.
4. The Takeaway: Stop Guessing, Start Matching
The paper concludes that there is no single "best" way to prompt an AI.
- If you have a Chat AI, keep instructions light and simple. Don't over-explain.
- If you have a Reasoning AI, give it strict, detailed rules. It thrives on structure.
- If you have a small, well-trained AI, it might be just as good as the big ones, regardless of the instructions.
In short: You wouldn't give a toddler a complex legal contract, and you wouldn't give a Supreme Court Justice a sticky note. You have to match the instruction style to the type of worker, not just their intelligence level.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.