Do LLMs Know a Good Hypothesis When They See One? Logit-Based Energy Scoring Outperforms Prompted LLM-as-Judge for Scientific Hypothesis Ranking
This paper demonstrates that a logit-based energy scoring method, which leverages a language model's intrinsic confidence, significantly outperforms prompted LLM-as-judge approaches in ranking scientific hypotheses across multiple disciplines.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Science has always been a conversation between what we know and what we might discover. When a researcher stands before a complex problem, they must first imagine a possible explanation, a hypothesis, that fits the facts they have gathered. For decades, this step has relied entirely on human intuition, but a new tool has entered the laboratory: the large language model. These are powerful computer programs trained on vast amounts of text that can now write sentences, summarize research, and even propose new scientific ideas. Yet, a critical question remains unanswered. If a computer can generate a list of possible explanations for a phenomenon, how do we know which one is actually correct? The current standard for answering this is to ask another, even larger computer program to read the list and pick the best one, much like a teacher grading a test. However, this method has a flaw: it often favors ideas that sound familiar or well-written, rather than those that are genuinely new or scientifically sound.
A team of researchers at Oak Ridge National Laboratory decided to test a different approach. Instead of asking a model to compare ideas and give a verbal opinion, they asked the model to simply "feel" the weight of each idea. They treated the computer's internal confidence as a signal. When a model reads a sentence it expects to see, it feels a sense of certainty; when it reads something strange or unlikely, it feels surprise. The researchers wondered if this raw, internal feeling could be a better judge of scientific truth than a formal, spoken ranking. They set out to see if a computer could recognize a good hypothesis simply by how naturally it fit into the story of a scientific paper, without ever being told to make a choice.
To test this, the researchers gathered a massive collection of 1,323 real scientific papers from twelve different fields, ranging from astronomy and physics to law and business. For each paper, they took the background information and the specific question the scientists were trying to answer. Then, they created a challenge for the computer models: a list of sixteen possible hypotheses. Only one of these was the real hypothesis that the original scientists had used; the other fifteen were cleverly written but incorrect alternatives. The goal was to see if the computer could identify the single correct hypothesis from the crowd of fifteen wrong ones.
The team ran this test using seven different computer models, ranging from very small, open-source programs to massive, proprietary systems. They compared two ways of judging the hypotheses. The first method was the traditional approach: they prompted a large, commercial model to read all sixteen options and explicitly rank them from best to worst. The second method was the new approach: they fed each hypothesis into a model one by one, along with the background context, and measured the model's internal confidence score. This score was based on how likely the model thought the specific words of the hypothesis were to appear next, given the context. They did this twice, once using a standard probability calculation and once using a raw measure of the model's internal energy, which captures the strength of the prediction before it is smoothed out.
The results were surprising and shifted the understanding of how these tools work. The traditional method, where a large model was asked to act as a judge and rank the list, performed poorly. It correctly identified the top hypothesis in only about 16.6% of the cases. In contrast, the method that relied on the model's internal confidence, without any explicit ranking instructions, performed significantly better. When the researchers looked at the best combination of model and scoring method, they found that a relatively small, open-source model using the raw energy score correctly identified the right hypothesis in 53.1% of the cases. This means that a modest computer program, simply "feeling" the text, was more than three times as effective at finding the correct scientific idea as a much larger, sophisticated system that was explicitly told to compare and rank the options.
The study also revealed that the size of the model did not guarantee better performance. The most successful configuration was a model with just one billion parameters, which is tiny compared to the massive systems often used for complex reasoning. In fact, the largest models tested did not always perform the best when asked to rank hypotheses. The researchers noted that the internal confidence signal was not uniform across all models; for some, the raw energy score was a powerful tool, while for others, it was less effective. This suggests that the way a model is built and trained matters more than its sheer size when it comes to this specific task. The study also found that the method worked consistently well across most scientific disciplines, though the differences between fields were small enough that they could not be considered definitive rules.
One of the most important takeaways from this work is that the way we currently ask computers to evaluate scientific ideas might be flawed. By forcing a model to articulate a preference and rank options, we may be introducing errors that hide its true understanding. The researchers suggest that the model's own internal likelihoods, its quiet confidence in a specific sequence of words, might be a more honest and reliable indicator of scientific plausibility than its spoken judgments. This does not mean the problem is solved. The researchers acknowledge that their test used hypotheses from papers that had already been published, so the models might have been recognizing familiar phrasing rather than truly understanding the science. They propose that the next step is to test these methods on brand-new, unpublished ideas where the answer is not known, requiring human experts to judge the results.
Ultimately, this research offers a new path for trustworthy artificial intelligence in science. It suggests that we do not need to rely on complex, verbal debates between computer programs to find the right answer. Instead, we might be able to trust the quiet, internal signals of a model, allowing even smaller, more accessible tools to play a vital role in the scientific discovery process. The study concludes that while the current methods have limitations, the potential of using a model's intrinsic confidence to evaluate scientific hypotheses is real and warrants further exploration.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.