Agentic Data Cleaning Without a Clean Reference: An Experimental Study of Capabilities and Trade-offs
This paper presents an experimental study evaluating seven agentic configurations for reference-free data cleaning, revealing that while advanced capabilities like evidence grounding and conservative repair enhance safety and reproducibility, they introduce significant trade-offs in detection performance and operational cost rather than delivering consistent improvements across all criteria.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of data, numbers and records are the raw material for decisions that shape our lives, from approving a loan to diagnosing an illness or tracking environmental safety. However, this raw material is rarely perfect. It often contains missing entries, impossible dates, or values that seem wrong at first glance. The central challenge for anyone working with data is distinguishing between a genuine mistake that needs fixing and a rare, unusual event that is actually true. If a sensor records a sudden spike in radiation, it could be a broken machine, or it could be a real, dangerous event that must be preserved. If a bank transaction looks suspicious, it might be fraud, or it could be a legitimate but unusual purchase. Traditionally, cleaning this data required a trusted, perfect version of the dataset to compare against, but in the real world, such a perfect reference rarely exists. This leaves data scientists in a difficult position: how do you fix a problem when you cannot be certain what the correct answer is?
Recent advances in artificial intelligence have introduced a new approach using "agents," which are computer programs capable of reasoning and using tools to solve problems. These agents can read data, spot patterns, and even write code to fix errors. The hope has been that these intelligent systems could clean data on their own, even without a perfect reference guide. A new study by Hadi Fadlallah explores exactly how well this works. The research asks a fundamental question: if we give an AI agent more and more powerful tools—such as the ability to search for external facts, check its own work with code, or follow strict safety rules—does it actually get better at cleaning data without making dangerous mistakes? The study does not assume that more power automatically means better results; instead, it treats the process as a careful experiment to see what happens when different capabilities are added or removed.
To find the answer, the researcher set up a controlled experiment using three very different types of real-world data: financial records used for credit decisions, clinical health records, and environmental radiation monitoring logs. The study created two distinct testing environments. In the first, the researcher intentionally injected known errors into the data, such as missing values or impossible dates, and also inserted rare but valid signals that should not be changed. This allowed for a precise measurement of how well the agents could find the errors and, crucially, how well they could avoid fixing the things that were actually correct. In the second environment, the agents were asked to clean the original, unmodified data. Since there was no perfect answer key for this real data, these runs were used to observe the agents' behavior, such as how often they hesitated, how much evidence they gathered, and how much time and computing power they consumed.
The experiment tested seven different versions of the cleaning agent, ranging from a simple system that only looked at the data structure to a highly complex system equipped with every available tool. The simplest version, which relied only on the artificial intelligence's own reasoning without any external checks, performed very poorly. It failed to identify most of the errors and often made incorrect assumptions. However, when the researchers added a step where the agent could use standard computer programs to check for obvious errors—like verifying if a date format was valid or if a number was within a reasonable range—the system's ability to spot problems improved significantly. This suggests that while the artificial intelligence is good at thinking, it is not as reliable as a simple, deterministic computer check for catching basic structural mistakes.
As the researchers continued to add more sophisticated capabilities, such as the ability to search a local library of documents for evidence or to rank sources by their reliability, the results became more nuanced. The most advanced system, which combined all these tools with a strict policy to avoid making changes unless it was absolutely certain, behaved very cautiously. In the tests where the correct answers were known, this cautious system never made an unsafe change or a useless modification. It successfully avoided the trap of "over-cleaning," where an agent might fix something that didn't need fixing. However, this extreme caution came with a cost: the system rarely made any direct repairs at all. It chose to flag uncertain cases for human review or leave them alone rather than risk making a mistake. While this is a safe strategy, it means the system did not actually fix the data in many cases where a correction might have been possible.
The study also measured the practical costs of these different approaches. The most cautious system (A6) was the slowest and required the most computing power, taking nearly two minutes to process a single run on original data and using a massive amount of digital resources. In contrast, the system that relied on simple, deterministic checks (A1) was almost instantaneous and used almost no computing power. The research found that adding more tools did not lead to a steady improvement in performance. Instead, each new capability introduced a trade-off. The system that was best at finding errors was not the same one that was best at avoiding mistakes, and the system that was safest was the most expensive to run. There was no single "perfect" agent that excelled at everything.
Ultimately, the research suggests that the idea of a fully autonomous agent that can clean any dataset without human oversight is not yet a reality. The study shows that while artificial intelligence can be a powerful partner in the data-cleaning process, it works best when its reasoning is grounded in hard evidence and strict safety rules. The most effective approach appears to be a combination of simple, reliable computer checks for obvious errors and a cautious, evidence-based strategy for handling complex or ambiguous cases. The researchers conclude that rather than trying to build a single, all-powerful machine, data scientists should carefully select which tools to use based on the specific risks and requirements of their task. If the cost of a mistake is high, a system that hesitates and asks for human help is preferable to one that acts quickly but might be wrong. The study provides a clear map of these trade-offs, showing that in the world of data cleaning, being too eager to fix things can be just as dangerous as doing nothing at all.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.