The Generalist Strikes Back:\\A General LLM Agent Outperforms Specialist Bioinformatics Agents on Free-Form, End-to-End Benchmark Analysis
This paper demonstrates that a general-purpose frontier LLM agent (Claude Opus 5.5) using only off-the-shelf coding tools outperforms specialized bioinformatics agents on a comprehensive benchmark, revealing that current limitations in measuring progress stem primarily from benchmark reliability issues rather than agent capability.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast and intricate world of biology, scientists are constantly trying to make sense of massive amounts of data. They study the blueprints of life, the proteins that build our bodies, and the complex ways cells communicate. To do this, they rely on specialized computer programs that act like digital assistants, guiding researchers through a maze of calculations to find answers hidden in the numbers. For years, the prevailing idea was that to handle these difficult tasks, you needed a highly trained, custom-built assistant. These digital specialists were designed with a specific set of tools and rules, much like a master carpenter who carries only the hammers and chisels needed for woodworking, leaving behind everything else. The assumption was that a general-purpose computer program, one that could do many things but knew nothing specific about biology, would be too clumsy to compete with these experts.
However, the landscape of artificial intelligence has shifted rapidly. Newer, more powerful computer models have emerged that can read, reason, and write code with a level of skill that rivals human experts in many fields. This raises a practical question for the scientific community: do we still need those custom-built, specialized assistants, or has the general-purpose model become smart enough to handle the job on its own? A recent study set out to answer this by pitting a general artificial intelligence against the best specialized bioinformatics agents currently available. The researchers wanted to see if a smart, all-around computer program could solve real-world biology problems just as well as, or even better than, the machines built specifically for the job.
The study focused on a rigorous test called PromptBio-Bench, which contains 244 different challenges ranging from simple data checks to complex biological analyses. These tasks are not multiple-choice questions with a single right answer; instead, they require the computer to produce actual files, such as tables of numbers, charts, and lists of genetic variations, that must match the quality and format of what a human expert would produce. The researchers used a state-of-the-art general model, running it in a standard coding environment, and asked it to solve every single task. They tested it in two ways: once with a massive library of 667 specialized biology instructions available to it, and once without any of those instructions, relying entirely on its own knowledge. They then compared its performance against three leading specialized agents that had been designed specifically for bioinformatics.
The results were surprising. The general model, running without any special biology tools, managed to complete every single one of the 244 tasks. When its answers were graded against expert references, it achieved a success rate of nearly 87 percent. This was significantly higher than the specialized agents, which scored between 72 and 76 percent. The general model performed particularly well on the most difficult tasks, where the specialized agents tended to struggle. Even more striking was the finding regarding the specialized instructions. The researchers discovered that the general model never actually used a single one of the 667 biology skills it was given access to. It solved the problems using its own internal reasoning. The small difference in scores between the version with skills and the version without was not due to the skills themselves, but rather to minor, random variations in how the model made a few specific decisions during the process.
The study also revealed that the test itself had some flaws. When the researchers closely examined the tasks where the general model failed, they found that many of the failures were not mistakes by the computer, but rather errors in the test's own grading system or in the "correct" answers provided by the test creators. For instance, some of the reference answers were based on outdated data or made assumptions that were not clearly stated in the original question. After fixing these issues in the grading system, the general model's success rate climbed even higher, reaching over 92 percent. This suggests that the benchmark was not measuring the computer's ability to do biology as well as it was measuring the computer's ability to guess what the test creators wanted.
Ultimately, the research suggests that for well-defined biological analyses, a powerful, general-purpose artificial intelligence is now capable of matching or exceeding the performance of systems built specifically for the field. The specialized agents, with their curated tools and complex structures, are no longer the only path forward. The study indicates that the main limitation in measuring progress is no longer the capability of the agents themselves, but rather the reliability of the tests used to judge them. As these general models continue to improve, the focus for the scientific community may need to shift from building more specialized tools to creating more robust and fair ways to evaluate how well these intelligent systems are truly working.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.