Auditing the Auditor: Extraction Accuracy, Hallucination, and Abstention in Open-Weight LLMs for ESG Reporting A Longitudinal Empirical Benchmark Across 518 Corporate Disclosures (2010–2026)
This longitudinal benchmark of five open-weight LLMs across 518 ESG reports reveals that while models reliably summarize narrative policies, they struggle with precise numerical extraction and frequently hallucinate data without explicit reasoning or strict instructions, highlighting a critical need for improved abstention mechanisms in automated sustainability auditing.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern business world, a company's commitment to the environment, its treatment of workers, and its ethical governance are no longer just matters of public relations. These factors have become hard data, legally required in many places, and scrutinized by investors who need to know if a corporation is truly sustainable or merely pretending to be. This shift has created a massive challenge for auditors. A single sustainability report can stretch to hundreds of pages, filled with dense tables of numbers, footnotes, and shifting definitions. To make sense of this volume, financial institutions are turning to artificial intelligence, specifically a type of computer program known as a large language model. These programs are trained on vast amounts of text and can read documents to find specific answers, much like a very fast, very well-read librarian. However, for these tools to be trusted with official audits, they must do more than just summarize a story; they must extract exact numbers without making things up, and they must know when to admit that an answer simply does not exist in the document.
A team of researchers from the University of Ghana set out to test exactly how well these open-source artificial intelligence models perform in this high-stakes environment. They gathered a massive collection of 518 real corporate sustainability reports published between 2010 and 2026, covering a period that includes both older voluntary reports and the newest, strictly regulated filings required by European law. From this library of documents, they created a rigorous test consisting of 120 specific audit questions. These questions ranged from asking for simple policy summaries to demanding precise figures for greenhouse gas emissions or water usage. The researchers then fed these questions to five different leading artificial intelligence models, simulating a real-world scenario where the computer first searches the document for the relevant page and then attempts to read the answer. Crucially, to ensure fairness, they used a separate, independent artificial intelligence system to grade the answers, preventing the models from giving themselves high scores for their own work.
The results revealed a clear divide between how well these machines understand words versus how well they handle numbers. When asked to summarize a company's general policies or commitments, the models performed quite well, often capturing the essence of the text with high accuracy. However, when the task required pulling a specific number from a complex table—such as the exact amount of carbon dioxide emitted in a specific year—their performance dropped significantly. Even when the researchers gave the models the perfect page from the report, the best-performing model could only extract the correct number about 73 percent of the time. The difficulty often lay not in finding the right page, but in reading the table correctly, such as confusing a parent company's total with a subsidiary's figure or misreading the units of measurement.
Perhaps the most critical finding concerned how these models behave when the answer is missing. In a real audit, a company might fail to report a specific metric, or a search might accidentally pull a page from the wrong year. The researchers tested this by giving the models a question about 2024 hazardous waste but providing a document page that only discussed 2022 water recycling. Under standard instructions, many of the models tried to be helpful by inventing a number or substituting a similar figure from a different year, a behavior known as hallucination. This happened in nearly half of the cases for some models. However, the study showed that this dangerous tendency could be largely fixed. By giving the models a strict instruction to say "I cannot answer" if the exact data was not present, the rate of made-up numbers plummeted. Furthermore, one model designed to "think" through its reasoning before answering was naturally better at spotting these mismatches and refusing to guess, even without the strict instructions.
The study also looked at how the changing nature of corporate reporting affects these tools. The researchers found that the newest, mandatory reports required by European regulations, which use standardized tables and consistent labels, are actually easier for the artificial intelligence to read than the older, voluntary reports. Once the computer found the right section in these modern documents, its accuracy improved noticeably. This suggests that as regulations force companies to organize their data more clearly, these tools will become more reliable. Ultimately, the research concludes that while artificial intelligence is ready to help summarize the stories companies tell about their sustainability, it is not yet ready to replace human auditors for checking the hard numbers. The safest approach involves using these models to draft summaries and flag potential issues, but keeping a human in the loop to verify the final figures, ensuring that the numbers are real and not just a convincing guess.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.