Signal or Noise? A Benchmark Study of Agent Skills in Web Development

In the modern landscape of software creation, artificial intelligence has evolved from a simple tool that answers questions into a persistent partner that writes code. To make these digital assistants more reliable, developers have begun attaching "skills" to them. Think of a skill not as a one-time instruction, but as a permanent rulebook or a set of habits that the AI carries with it for an entire work session. These rulebooks contain guidelines on how to write code for specific technologies, warnings about common mistakes to avoid, and examples of how to solve problems. The hope is that by giving the AI this extra context, it will act more like an experienced human engineer and less like a novice guessing at every step. However, there is a hidden cost to this approach. Every time a skill is added, the amount of text the AI must read before answering a question grows larger. This raises a critical, unanswered question: does the extra knowledge actually help the AI do its job better, or does the sheer volume of text simply confuse it, slowing it down and making more mistakes?

A team of researchers at Baidu set out to answer this question by treating the addition of skills as a scientific experiment rather than a default setting. They focused on web development, a massive field where AI is frequently asked to build websites and applications using languages like JavaScript and HTML. They gathered thirty-one different public skill guides, ranging from general coding advice to specific instructions for popular software frameworks. To test them, they used a rigorous testing ground consisting of fifty real-world web projects, each containing twenty sequential tasks that build upon one another. This created a total of one thousand distinct challenges for the AI to solve. The researchers then ran the same tasks under four different conditions. In the first scenario, the AI received no extra skill guide at all. In the second, it received the specific skill guide intended for that project. In the third, to isolate the effect of text length, the AI received a guide of the exact same size but filled with irrelevant, nonsensical content. Finally, they broke down the helpful guides to see which specific parts—such as rules, warnings, or code examples—were actually doing the work. They tested these setups across four different large language models, ranging from widely used commercial systems to specialized coding models.

The results challenged the common assumption that more context always leads to better performance. Across all four models tested, adding the intended skill guide actually lowered the success rate of the AI. On average, the AI completed fewer tasks correctly when the skill was present compared to when it was absent. The drop in performance was not trivial; it ranged from a small decline to a significant loss of nearly four percentage points in success rates. Furthermore, the AI became less efficient, using significantly more computing resources to complete the same work. In some cases, the cost of processing the extra text jumped by nearly four hundred percent. The researchers found that the AI only improved its performance in a small minority of cases, roughly one in five to one in three pairings of a specific skill and a specific project. This suggests that for the vast majority of situations, the injection of these skills was counterproductive, introducing noise rather than signal.

The study also revealed that the reason for this failure depends entirely on which AI model is being used. For two of the models, the problem was simply the length of the text. When these models were given a guide of the same length but with irrelevant content, they performed just as poorly as they did with the real skill guide. This indicates that their attention was being diluted by the sheer volume of words, regardless of what those words meant. For the other two models, the length of the text was not the issue; they handled the irrelevant text just fine. Instead, the specific content of the skill guide actively misled them, steering them away from the correct solution. This distinction is crucial because it means there is no single fix for the problem. For some systems, the solution is to shorten the text; for others, the content itself must be rewritten or removed entirely.

Perhaps the most surprising finding was that a skill that works well for one AI model often fails or even harms a different model. The researchers found almost no connection between how a skill performed on one system versus another. A guide that helped one model solve a problem might cause a different model to fail at the same task. This lack of consistency means that developers cannot simply look at a popular skill guide and assume it will work for their specific setup. Instead, the decision to use a skill must be made on a case-by-case basis, considering the specific model, the specific project, and the specific task at hand. The study also showed that these skills were most harmful on the easiest tasks. When the AI already knew how to solve a problem, the extra rules seemed to lock it into a rigid way of thinking, preventing it from correcting small mistakes that it would have otherwise fixed easily.

Finally, the researchers dissected the helpful skills to see what made them work when they did. They found that the most valuable parts were often the brief warnings about what not to do, known as anti-patterns. These short rules were effective across the board. In contrast, the sections containing long blocks of example code were a mixed bag. While they sometimes helped weaker models, they tended to confuse the most advanced ones, suggesting that showing an AI too many examples can be detrimental. The study concludes that the era of blindly attaching skill guides to AI agents is over. Instead, the injection of these tools should be treated as a careful routing decision, where the potential benefit is weighed against the cost of the extra text for every single deployment. The findings suggest that without this careful, per-project auditing, the very tools meant to make AI smarter are often making it slower and less reliable.