The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines
This paper introduces the "hallucination snowball" model to demonstrate how errors in multi-agent LLM pipelines transform and become increasingly undetectable as they propagate through stages, proving that early-stage verification is significantly more effective than end-of-pipeline checking for mitigating hallucination survival.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a new architecture has emerged to tackle complex tasks: the multi-agent pipeline. Imagine a team of specialized workers, each with a distinct job, passing a single document down a line. One agent might research a topic, the next analyzes the findings, a third writes a narrative, and a fourth reviews the final draft. This approach, used by major frameworks to handle everything from financial reporting to medical summaries, relies on a simple premise: each worker trusts the output of the one before them. They receive only the text passed to them, without access to the original source documents or a way to verify the facts they are handed. While this chain of trust allows for sophisticated workflows, it creates a hidden vulnerability. If the first worker makes a mistake, that error does not simply sit there; it travels down the line, changing shape and becoming harder to spot with every handoff.
Researchers have long focused on building better detectors to catch these mistakes, assuming that if an artificial intelligence is smart enough, it can spot a lie in a story. However, a new study challenges this assumption, revealing that the problem is not a lack of intelligence, but a structural flaw in how the information moves. The researchers, working with a four-agent system designed for financial analysis, demonstrated that a false fact introduced at the very beginning of the chain undergoes a transformation that makes it nearly impossible to detect by the time it reaches the end. They call this phenomenon the "hallucination snowball." Just as a small snowball rolling down a hill gathers more snow and becomes harder to stop, a small error in a raw number gets wrapped in calculations, then woven into a story, and finally approved as a confident conclusion. With each step, the ability to prove the fact is wrong is systematically destroyed.
To understand how this happens, the team set up a controlled experiment using a pipeline of four agents: a researcher, an analyst, a writer, and a reviewer. They started with a set of real financial data and deliberately injected false numbers into the researcher's output, such as changing a company's cost of goods sold from a true value of $63.1 billion to a fabricated $71.2 billion. They then watched how this lie traveled through the system. At the first stage, the false number was a raw fact, easy to check against the original records. By the second stage, the analyst had used this false number to calculate a year-over-year percentage increase, turning the lie into a derived computation. By the third stage, the writer had woven this calculation into a paragraph of professional prose, explaining why the company faced "significant cost headwinds." Finally, the reviewer, seeing only the polished text and no access to the original documents, approved the report as well-reasoned and sound.
The results were stark. When the researchers tested how well a powerful AI model could spot the lie at each stage, the detection rate dropped dramatically. At the very beginning, the model caught 72 percent of the errors. But by the time the error reached the final reviewer, the detection rate had fallen to just over 50 percent. More than one in five of the injected lies survived the entire process completely undetected, ending up in the final report as if they were true. The study showed that this decay was not random; it happened most sharply when the raw number was turned into a calculation, and again when that calculation was turned into a story. By the time the error became part of a narrative, it was structurally invisible to any downstream check. Even the most advanced AI models tested, which could catch nearly 90 percent of errors at the start, could not overcome this structural ceiling. When projected through the pipeline, their ability to detect the final error dropped to a range where nearly half the lies would still slip through.
The researchers also tested a common solution: checking the final output for errors. They found that waiting until the end of the line to verify the work was almost useless. Because the original facts had been transformed and buried in text, the final check could only catch a tiny fraction of the remaining errors, offering almost no improvement over having no check at all. The study argues that the timing of the check matters far more than the quality of the tool used. When the researchers placed a simple, automated check at every handoff point between agents, the results changed completely. By catching the error while it was still a raw number, before it could be turned into a calculation or a story, they reduced the number of surviving lies from nearly 60 percent down to just 16 percent. This intervention was so effective that a single check at the very first handoff caught three-quarters of all errors, proving that the earlier the verification happens, the more powerful it is.
There is a cost to this approach, however. When the system aggressively filters out errors at the start, the final reports sometimes contain gaps where numbers should be, because the downstream agents were not always able to fill in the corrected values perfectly. This led to a slight drop in the internal consistency score of the reports, making them appear slightly less polished. Yet, the researchers emphasize that a report that is fluent but built on fabricated numbers is far less valuable than a report that is slightly rougher but factually accurate. The study concludes that the solution to this problem is not to build smarter detectors for the end of the line, but to install gates at the beginning. By verifying information before it is transformed, we can stop the snowball before it gains momentum, ensuring that the final output is built on a foundation of truth rather than a cascade of uncorrected errors.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.