From Hallucination to Reliability: Generative Modeling and the Structure of Scientific Inference

Science has long relied on a simple, comforting idea: if you build a model of the world, you can trust it to tell you the truth, provided you feed it the right data. But a new generation of artificial intelligence is changing the rules. These are not the old-style computers that simply sort information into neat categories; they are generative models, systems designed to create entirely new, complex data from scratch. They can predict how a protein folds into a three-dimensional shape or forecast the weather across the entire globe. The problem is that these creative machines have a dark side. They are prone to "hallucinations," a term borrowed from human psychology to describe when the AI invents plausible-looking details that simply do not exist. In a medical scan, this might look like a fracture in a healthy bone; in a protein model, it might be a solid structure where nature has left a chaotic, disordered mess. For decades, scientists worried that these errors were fatal flaws, rendering the technology too dangerous for serious discovery. If the machine is making things up, how can we ever know what is real?

A new paper by Charles Rathkopf, a researcher at the Jülich Research Centre, argues that we have been looking at the problem the wrong way. The fear that hallucinations make generative AI unreliable is based on a misunderstanding of how science actually works. Rathkopf suggests that we should stop judging the AI model in isolation and start judging the entire scientific process, or "workflow," in which the model is embedded. The paper proposes that even if the AI continues to invent false details, scientists can build a safety net around it. By using their own deep, pre-existing knowledge of the natural world, researchers can filter out the lies and keep the truths. The result is that the final scientific conclusion can be reliable, even if the tool used to reach it is imperfect.

To understand why this matters, one must first grasp what these AI models are doing. Unlike traditional software that follows strict, pre-written rules, generative AI learns by studying vast amounts of existing data and then guessing what comes next. It is like a student who has read every book in a library and is now asked to write a new story. The student might write something beautiful and new, but they might also invent a character who never existed or a plot point that contradicts the laws of physics. In the world of science, this "invention" is the hallucination. The paper points out that these errors are not just random mistakes; they are inevitable. Because the AI is trying to create complex, high-dimensional structures—like the intricate folding of a protein or the swirling patterns of a storm—it must fill in gaps where the data is thin. In doing so, it sometimes creates a structure that looks perfect on the surface but is fundamentally wrong underneath. The author argues that we cannot simply train the AI harder or build a better version of the model to stop this. The very mechanism that allows the AI to be creative and useful is the same mechanism that forces it to hallucinate.

The paper challenges a common assumption in the field: that a hallucination is simply a piece of data that looks different from what the AI was trained on. If that were true, scientists could just throw away any output that didn't match their training data. But Rathkopf argues this is a trap. In science, the goal is not to repeat what we already know; it is to discover what we do not know. If a scientist throws away a strange new result just because it looks different from the past, they might accidentally throw away a genuine discovery. Instead, the paper defines a hallucination more precisely: it is an output that misrepresents the real-world object it is supposed to describe, and it does so by accident, not by design. It is a mistake the machine makes on its own, not a trick it was taught.

The author then turns to two real-world examples to show how scientists are already solving this problem without needing to fix the AI itself. The first case is AlphaFold, a famous AI system developed by Google DeepMind that predicts the 3D shapes of proteins. Proteins are the workhorses of life, and their shapes determine what they do. AlphaFold 3, the latest version, uses a generative approach that allows it to invent new structures. The developers knew this would lead to hallucinations. In fact, the system sometimes invents a solid, compact shape for a part of a protein that is actually supposed to be floppy and disordered. If a scientist took that output at face value, they would be misled. However, the team built a "workflow" around the AI to catch these errors. They added a second layer of analysis that acts like a quality control inspector. This system looks at the AI's output and checks it against known rules of chemistry and physics. For instance, it knows that disordered parts of a protein should be exposed to the surrounding water, while solid parts should be tucked inside. If the AI invents a solid shape for a disordered part, the workflow flags it as suspicious. The system also assigns a confidence score to every part of the predicted shape, telling the scientist exactly which parts are reliable and which parts are likely hallucinations. The AI still makes mistakes, but the workflow ensures that those mistakes never reach the final conclusion.

The second example comes from meteorology, specifically a system called SEEDS used for weather forecasting. Weather is chaotic; tiny changes in the starting conditions can lead to wildly different outcomes. To handle this, forecasters use "ensembles," which means running the same model many times to get a range of possible futures. SEEDS uses a generative AI to create these many possible futures quickly. The paper notes that individual forecasts from this AI might be physically impossible, such as a weather map where the air pressure and temperature contradict the laws of physics. These are hallucinations. Yet, the system remains reliable. How? Because the scientists do not look at a single forecast in isolation. They look at the whole group of 512 forecasts together. If one forecast is a hallucination, it is just one voice in a large crowd. As long as the hallucinations are rare and not all the same, they wash out when the scientists calculate the final probability of rain or heat. The workflow here is the ensemble itself. By treating the output as a collection of possibilities rather than a single truth, the system filters out the noise. The researchers also use statistical tools to check if the group of forecasts, as a whole, matches reality, ensuring that the final probability numbers are trustworthy.

The core finding of the paper is that reliability does not require the AI to be perfect. It requires the human process around the AI to be smart. Rathkopf argues that we should stop trying to build a "hallucination-free" AI, a goal that may be impossible, and start building "hallucination-resistant" workflows. This shifts the focus from the machine to the method. The machine can be a black box, a mysterious generator of ideas, as long as the scientists surrounding it have the knowledge to test those ideas. The paper suggests that this is how science has always worked: we use tools that are imperfect, but we surround them with checks and balances. The AI provides the raw material, and the scientist provides the judgment.

This approach also changes how we think about the role of AI in science. Some critics argue that AI is only good for finding new ideas (discovery) but not for proving they are true (justification). They say we should only trust AI if we can test its predictions in a lab. Rathkopf agrees that testing is crucial, but he adds that the workflow itself is part of the proof. The act of filtering, ranking, and qualifying the AI's output is a form of justification. It is not just about waiting to see if the AI is right; it is about building a system that ensures the AI is right before we even run the experiment. The paper suggests that as long as we have enough background knowledge about the system we are studying—whether it is a protein or a storm front—we can construct these safety nets.

The paper concludes with a sobering but hopeful note. The ability to build these reliable workflows depends on having a deep understanding of the target system. If scientists are trying to model something they know very little about, they cannot build the filters needed to catch the AI's lies. In those frontier areas, the risk remains high. But for the many fields where we already have a solid grasp of the rules—like protein folding or weather dynamics—generative AI can be a powerful partner. The machine will continue to hallucinate, just as a human artist might make a mistake while painting. But as long as the scientist knows what a real painting looks like, they can spot the error and keep the masterpiece. The paper does not claim that the AI problem is solved or that hallucinations will disappear. It claims something more practical: that we can learn to live with them, manage them, and still use these powerful tools to expand the boundaries of human knowledge. The reliability of science, the author argues, has never been about the perfection of our tools, but about the wisdom of how we use them.