Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
The Science Edge Evaluation (SEE) benchmark reveals that current multimodal large language models struggle to reliably derive evidence-based insights from experimental data in real-world scientific discovery, highlighting a critical gap between explaining established concepts and generating novel, justified scientific inferences.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern laboratory, discovery is rarely a sudden flash of insight. It is a slow, deliberate process of gathering evidence. A scientist looks at a spectrum of light bouncing off a molecule, a microscopic image of a cell, or a graph of temperature changes, and asks what these silent signals actually mean. The answer must be grounded strictly in what the data shows, not in what the scientist hopes to find or what they have seen in other experiments. This discipline of sticking to the evidence is the bedrock of reliable science. Today, powerful computer programs known as large language models are being asked to assist in this work. These systems can read vast libraries of scientific papers and discuss complex theories with ease. But a critical question remains: can they look at a raw, messy set of experimental results and draw a conclusion that is both correct and strictly limited by what the data allows?
A new study titled "Science Edge Evaluation" tackles this question directly. Researchers from Alibaba Group, Tsinghua University, and other institutions created a rigorous test to see if these artificial intelligence systems can truly function as partners in real scientific discovery. They built a benchmark called SEE, which consists of over a thousand questions drawn from actual, published research in chemistry, biology, and materials science. Unlike previous tests that simply asked if a model knew a fact or could solve a textbook problem, this benchmark presents the models with the same visual evidence a human scientist would see—such as gel images, spectral curves, and microscope photos—along with the accompanying text. The goal was to see if the models could interpret these images and reach a conclusion that stayed within the boundaries of the evidence provided, without guessing or inventing facts.
The results of this evaluation were sobering. The researchers tested eighteen different advanced models, including both general-purpose systems and those specifically trained for science. Even the best-performing model managed to answer correctly only 48.7 percent of the time. No model reached the 50 percent mark. This low score suggests that current artificial intelligence is not yet reliable enough to handle the nuanced reasoning required in a real laboratory. Perhaps more surprisingly, the models designed specifically for science did not outperform the general-purpose ones. In fact, the general models were slightly better on average. This finding indicates that simply feeding a computer more scientific facts or training it on specialized texts is not enough. The missing piece is not knowledge, but the ability to reason carefully based on the specific evidence at hand.
The study also looked at what happens when these models are given tools to help them, such as the ability to search the web for background information or use a code interpreter to analyze images mathematically. Giving the models these tools did help, raising the best accuracy to 52.7 percent. However, the improvement was modest, and the models still made significant errors. The researchers found that while tools could provide more information, they did not solve the core problem: the models still struggled to manage that information correctly. They often failed to stop when they had enough evidence, or they combined new data with old assumptions in ways that led to incorrect conclusions. The tools expanded what the models could see, but they did not teach the models how to think within the limits of what they saw.
When the researchers analyzed the specific ways the models failed, a clear pattern emerged. The errors fell into two main categories: perception and inference. In terms of perception, the models often lacked the precision to read exact numbers from a graph or a spectrum. They were good at recognizing general patterns but poor at acting as measurement instruments. In terms of inference, the models frequently let their prior knowledge override the actual visual evidence. If an image showed something unusual, the model would often ignore the strange detail and instead describe what it expected to see based on its training data. They also tended to over-extrapolate, taking a small, partial result and turning it into a sweeping, definitive conclusion that the data did not support.
One of the most revealing experiments involved removing the images entirely and asking the models to answer based only on the text. A truly intelligent system should recognize that without the visual evidence, it cannot answer the question. Instead, the models almost always tried to answer anyway, guessing based on the text alone. They identified missing information in only 4.6 percent of the cases. This behavior, known as hallucination, is dangerous in science because it creates a false sense of confidence. The models were willing to invent an answer rather than admit that the necessary evidence was missing.
The study concludes that the gap between current artificial intelligence and real scientific discovery is not about having more data or smarter tools. The gap lies in the ability to manage evidence. Real scientific discovery requires a system that can look at a set of observations, understand exactly what those observations prove, and stop there. It requires the discipline to say, "This is what the data shows," and to resist the urge to fill in the gaps with guesses. Until artificial intelligence can reliably distinguish between what is known and what is merely possible, it will remain a tool for explaining established ideas rather than a partner for discovering new ones. The path forward, the researchers suggest, is not just to make models bigger or more knowledgeable, but to teach them the rigorous logic of evidence-bound reasoning.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.