The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task
This study demonstrates that the choice of agent scaffolding has a far greater impact on the cost and reliability of AI coding agents than the underlying tool interface (MCP vs. CLI), revealing that MCP support is often unnecessary and can be significantly more expensive than direct CLI execution.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Digital Workshop: Setting the Stage
Imagine you have a brilliant but very literal robot assistant. This robot can write stories, solve math problems, and chat about anything, but it lives in a sealed room. It can't see your computer screen, open your files, or click buttons on a website. To get any actual work done, you need a "foreman" to stand between the robot and the real world. This foreman is called an agent scaffolding. Its job is to listen to the robot's ideas, translate them into actions (like "open this file" or "send this email"), and then bring the results back so the robot can keep thinking.
For a long time, people have been arguing about the best way to build this foreman. One popular method, called MCP (Model Context Protocol), is like giving the robot a giant, detailed encyclopedia of every single tool it might ever need. Every time the robot asks for help, the foreman hands it the whole encyclopedia again, just in case. Another method, the CLI (Command-Line Interface), is simpler: the foreman just gives the robot a basic toolbox and says, "You know how to use a hammer and a screwdriver, right? Go figure out how to use the specific tools you need."
The big question everyone is asking is: Which method costs less? Since every time the robot talks to the computer, it has to pay for the words (tokens) it sends and receives, people assumed the "giant encyclopedia" method (MCP) would be incredibly expensive because it sends so much extra text. Some experts even guessed it could cost 35 times more than the simple toolbox method. But nobody had actually measured this fairly, and the numbers were all over the place.
The Great Tool-Box Showdown
In this study, a team of researchers decided to settle the debate by running a massive, controlled experiment. They didn't just guess; they built a digital obstacle course—a specific task involving six steps like finding a bug in code, fixing it, and sending it to a team. They then sent this same task through seven different "foremen" (agent scaffoldings) and five different robot brains (AI models) to see how much it actually cost to finish the job.
Here is the twist: The researchers found that the foreman itself mattered way more than the toolbox.
The most surprising discovery was that two of the seven foremen they tested didn't even have the "giant encyclopedia" (MCP) capability. They only used the simple toolbox (CLI). Yet, they finished every single task perfectly. Not only did they finish, but they were 5 to 28 times cheaper than the fancy foremen that did have the encyclopedia. In fact, for one specific small robot brain, the cost jumped 139 times just because it was paired with a different foreman, even though the robot and the task were exactly the same.
So, what about the "giant encyclopedia" (MCP) being expensive?
The researchers tried to compare the two methods directly within the same foreman, but the results were messy and inconsistent. Sometimes the encyclopedia cost more, sometimes it cost less, and sometimes it was the same. The data was too shaky to say one was definitively cheaper than the other in a fair fight. However, they did find one clear difference: When things went wrong, the encyclopedia method was much more expensive. About 12.9% of the money spent on the MCP runs bought nothing (because the task failed), compared to only 2.2% for the simple toolbox runs. But the robots failed just as often with both methods; it's just that when the MCP robots failed, they wasted more money doing it.
The "Magic" of the Foreman
The study also uncovered a sneaky trick. The researchers told the robots exactly which toolbox to use. But when they checked the logs, they found that the robots often ignored the instructions! Some robots were given the "giant encyclopedia" but quietly used the simple toolbox instead. Others did the opposite. This means that if you just ask a robot "how much did this cost?" without checking what it actually did, you might be measuring a mix of both methods, which makes the numbers useless.
The Bottom Line
The main takeaway isn't that one tool is magic and the other is trash. Instead, the study suggests that how you build your robot's foreman is the most important factor in how much it costs.
- For simple, repeatable jobs: A small, custom-made foreman that only uses the basic toolbox (CLI) is vastly cheaper and more reliable than a giant, general-purpose assistant carrying a massive encyclopedia.
- For small robot brains: If you are running a smaller, cheaper AI model on your own computer, the "general-purpose" foremen can make the task cost nearly 140 times more just by being too chatty and carrying too much unnecessary baggage.
The researchers concluded that for specific, well-defined tasks, we don't need the heavy, expensive "giant encyclopedia" approach. We can build leaner, cheaper systems that get the job done without the extra weight. And if you are going to use the fancy encyclopedia method, you need to make sure your robot actually uses it, or you might be paying for a tool you never touched.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.