Chemical Chain-of-Thought Functions as a Hallucination-Prone Molecular Scratchpad
This study reveals that chemical chain-of-thought reasoning in language models functions as a hallucination-prone molecular scratchpad where fabricated structural claims are common yet causally load-bearing for generating correct answers, necessitating process-level supervision over simple answer evaluation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast landscape of artificial intelligence, a new generation of models has learned to think before it speaks. Instead of jumping straight to an answer, these systems are trained to write out a step-by-step reasoning process, much like a student working through a math problem on a sheet of scratch paper. This method, known as chain-of-thought reasoning, has become a standard way to help computers solve complex puzzles. In the field of chemistry, where molecules are the building blocks of life and medicine, this ability is particularly promising. Scientists hope that if a computer can explain its logic for designing a new drug or identifying a chemical structure, human experts can check that logic for errors. The assumption has been that if the final answer is correct, the reasoning used to get there must also be trustworthy.
However, a recent study challenges this comforting assumption. Researchers from institutions including the National University of Singapore and the Hong Kong Polytechnic University investigated whether these AI models are truly reasoning through chemical problems or if they are simply making things up along the way. They focused on a specific type of error called a hallucination, where a model confidently states a fact that is simply not true. In the context of chemistry, this might look like a model claiming a molecule contains a specific chemical group, such as a carboxylic acid, when that group is actually missing from the structure. Because chemical structures are precise and can be checked against a known database, this field offers a rare opportunity to test if an AI's explanation matches reality. The researchers wanted to know: when a model gets the right answer, is it because it understood the chemistry, or did it just guess the right shape while inventing a fake story to justify it?
To find the answer, the team analyzed four different families of reasoning models across twelve different chemistry tasks, ranging from describing molecules to designing new ones. They built a system to read the models' written reasoning and check every single chemical claim against the actual molecular structures involved. What they discovered was startling. Hallucinations were widespread. In many cases, the models produced the perfectly correct molecular structure but included fabricated chemical details in their written explanation. For instance, on a task involving the generation of molecules from text descriptions, about 13 percent of the responses from one leading model were exactly correct in their final output, yet the reasoning text that accompanied them contained at least one made-up chemical claim. The study found that the presence of these lies had almost no connection to whether the final answer was right or wrong. A model could be completely accurate in its result while its reasoning was riddled with falsehoods, or it could be wrong in its result while its reasoning was clean.
This disconnect suggests that the written reasoning is not always a faithful record of how the model arrived at its conclusion. To understand why this happens, the researchers performed a series of experiments where they deliberately altered the models' outputs. They took the written reasoning text and swapped out the names of chemical groups for incorrect ones. Surprisingly, changing these verbal claims had very little effect on the final answer the model produced. The model seemed to ignore its own written explanation when it came time to generate the final structure. However, when they changed the original input or the internal structural sketches the model wrote down, the final answer changed dramatically. This revealed that the models were not relying on their own prose to solve the problem. Instead, they were using the reasoning space as a kind of molecular scratchpad.
The researchers found that the models were writing down fragments of chemical structures, often in a format called SMILES, which is a way of representing molecules as text strings. These fragments acted as intermediate notes that the model used to build the final answer. While the readable sentences describing the chemistry were often unreliable and prone to fabrication, these structural drafts were carrying the real weight of the computation. Different models used this scratchpad in different ways. Some relied heavily on these text-based structural fragments, while others seemed to focus more on broader chemical cues like the shape of the molecule or the position of specific parts. The study showed that the readable explanation is often just a post-hoc rationalization—a story told after the fact—while the actual work happens in these hidden structural drafts.
The implications of this finding are significant for how we trust and use artificial intelligence in science. If a model can produce a correct drug candidate while lying about the chemical groups it used, relying solely on the final answer is risky. A human chemist reading the explanation might be misled by the fabricated details, assuming the model understood the chemistry when it did not. The researchers demonstrated that it is possible to train models to be more honest. By creating a new training method that rewards the model only when its reasoning contains no fabricated claims, they produced a version of the model that made far fewer mistakes in its explanations without losing its ability to solve the problems. This new approach, which they called Chem-R-Faithful, reduced the rate of fabricated claims significantly while maintaining high performance on the tasks.
Ultimately, this work reveals that chain-of-thought reasoning in chemistry is neither a perfect explanation nor a useless byproduct. It is a complex, hallucination-prone workspace where the model drafts its solution. The readable text is often unreliable, but the structural notes hidden within that text are crucial for getting the right answer. This means that to truly trust an AI in science, we cannot just look at the final result or the written story. We must develop new ways to verify the process itself, checking the structural drafts and the logic behind the scenes. As these tools become more common in drug discovery and molecular design, understanding that the reasoning trace is a working draft rather than a final report will be essential for ensuring safety and accuracy in the chemicals of the future.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.