Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
This paper presents a human-aligned LLM-as-a-Judge evaluation framework for the ChatInvent agentic drug discovery system, which defines specific quality dimensions, validates and optimizes a judge model through expert annotation to achieve high alignment, and reveals that informal phrasing does not hinder performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the high-stakes world of drug discovery, scientists spend years designing molecules, building them in the lab, and testing them to see if they can cure disease. This process is slow, expensive, and filled with complex decisions. Recently, a new kind of artificial intelligence has entered the scene: systems that do not just answer questions but take action. These are called agentic systems. Unlike a standard chatbot that simply generates text, an agent can plan a multi-step experiment, call upon specialized software to calculate chemical properties, and retrieve data from vast scientific databases. It acts more like a digital research assistant that can actually do the work, not just talk about it. However, as these digital assistants become more capable, a critical problem emerges: how do we know if they are doing a good job? Traditional ways of grading computer answers, which often rely on matching words to a correct answer, fail when the task is open-ended and creative. Furthermore, asking human experts to grade every single output is too slow to keep up with the speed of these new machines. The scientific community needed a way to evaluate these agents that was both fast and trustworthy.
A team of researchers at AstraZeneca and the University of Gothenburg tackled this challenge by building a system where one artificial intelligence acts as a judge for another. They focused on a specific tool called ChatInvent, an agentic assistant designed to help scientists navigate the stages of drug development. The researchers knew that simply letting an AI grade itself was risky, as these models can be biased or inconsistent. To solve this, they designed a rigorous testing framework to ensure the "judge" AI agreed with human experts. They defined four specific qualities that a good answer should have: whether the response was complete, whether the information was relevant to the question, how clear and well-organized the text was, and whether the agent stayed within its assigned role without trying to do things it was not supposed to do. They also included a strict check to see if the agent used the correct computer tools to get the job done.
To find the best judge, the team tested four different powerful AI models, including options from major technology companies and open-source projects. They asked five human experts in chemistry and artificial intelligence to grade a set of 35 questions and answers. These experts were given the same instructions and information as the AI judges. The researchers then compared the scores given by the humans against the scores given by each AI model. They found that while the human experts did not always agree with each other perfectly, the AI models were surprisingly consistent. One model, in particular, aligned very closely with the human majority opinion. The researchers then used a small set of examples, annotated by the human experts, to fine-tune this best-performing AI judge. This process, known as optimization, improved the judge's ability to match human thinking, raising its agreement score from 80% to 86%.
With a reliable judge in place, the team evaluated 70 new questions that the ChatInvent system had never seen before. The results revealed both strengths and weaknesses in the agent's performance. The system was excellent at staying on topic, organizing its answers clearly, and using the right tools to gather information. However, it struggled with completeness. In nearly 40% of the cases, the agent provided a partial answer, often forgetting to include specific details like where to find a generated file or a requested chemical property. The researchers discovered that this was not because the agent misunderstood the question, but because it sometimes failed to execute a final step in its workflow. Interestingly, the way the question was asked made a difference. The team tested questions ranging from very formal to very casual. They found that overly formal phrasing did not help the agent perform better; in fact, slightly more informal or neutral phrasing sometimes led to better results. This suggests that the agent might benefit from having its own internal system rewrite a user's stiff or complex prompt into a clearer version before it begins working.
The study concludes that while these agentic systems hold great promise for speeding up drug discovery, they are not yet perfect. The researchers demonstrated that it is possible to build a robust, automated evaluation system that mirrors human judgment, provided that the judge is carefully aligned with expert opinions. This approach allows developers to quickly identify where their agents are failing—such as missing file paths or incomplete data—without needing to hire a team of human graders for every test. The work serves as a blueprint for the future, showing that for artificial intelligence to be truly useful in science, we must first teach it how to grade its own work with the same care and precision that a human expert would use.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.