Towards Practical Benchmarking of Data Cleaning Techniques: On Generating Authentic Errors via Large Language Models

This paper introduces TableEG, a framework that leverages fine-tuned large language models to generate authentic, distribution-aligned synthetic errors in tabular data, thereby addressing the scarcity of real-world error datasets and establishing a robust benchmark for evaluating data cleaning techniques.

The Gist

Imagine you are a chef trying to teach a robot how to spot spoiled ingredients in a massive kitchen. To do this, you need to show the robot examples of bad food. But here's the problem: real spoiled food is rare, and making fake spoiled food is hard.

If you just throw random things at the robot (like putting a banana in a soup pot), the robot learns to spot "weird" things, not "spoiled" things. It's like training a security guard by having them chase a mannequin instead of a real thief. The guard learns to chase mannequins, but fails when a real criminal shows up.

This is exactly the problem data scientists face with Data Cleaning. They need to find errors in spreadsheets (tables) to fix them, but they don't have enough real-world examples of errors to train their AI.

Here is how the paper "Towards Practical Benchmarking of Data Cleaning Techniques" solves this using a new tool called TableEG.

1. The Old Way: The "Robot Chef" with a Rulebook

Previously, researchers used a tool called BART to create fake errors. Think of BART as a robot chef with a very strict, dumb rulebook.

• The Rule: "If you see a word, change one letter."
• The Result: If the recipe says "Tomato," BART changes it to "Tomato" (maybe "TomatX").
• The Problem: In the real world, errors aren't usually just typos. Sometimes the chef forgets to write the ingredient entirely (Missing Value). Sometimes they write "5000 pounds of salt" instead of "5 pounds" (Outlier). Sometimes they write "Chicken" in the "Vegetable" column (Rule Violation).
• The Analogy: BART is like a prankster who only changes the color of the food. It doesn't understand that the food might be rotten, missing, or the wrong type entirely. It creates "fake" errors that look nothing like real human mistakes.

2. The New Way: The "Smart Intern" (TableEG)

The authors created TableEG, which uses a Large Language Model (LLM)—basically a super-smart AI that has read almost everything on the internet.

Instead of giving the AI a dumb rulebook, they treated it like a smart culinary intern.

• The Training: They didn't just ask the AI to "make mistakes." They showed it thousands of real examples of real mistakes (like a chef forgetting to list an ingredient or writing the wrong price).
• The "Triplet" Method: They taught the AI using a three-step lesson plan (called a Triplet):
  1. The Instruction: "Here is a clean recipe. Now, make a realistic mistake."
  2. The Context: The clean table (the recipe).
  3. The Output: The specific error (e.g., "Change 'Chicken' to 'Tofu' because the customer is vegetarian").

By training the AI on real messy data, it learned the patterns of human error. It learned that humans often forget to fill in a cell, or they mix up numbers, or they use the wrong format.

3. Why This Matters: The "Thief Test"

The paper proves that TableEG is amazing by doing a "Thief Test."

• The Setup: They took a group of "Security Guards" (Data Cleaning Algorithms) and showed them two types of bad food:

  1. Real Spoiled Food: Actual errors found in real databases.
  2. TableEG Fake Food: Errors generated by the new AI.
  3. BART Fake Food: Errors generated by the old rule-based robot.

• The Result:

  • When the guards looked at BART's fake food, they were confused. "Is this a typo? Is it a missing item? I don't know!" The guards performed poorly because the fake errors were too weird.
  • When the guards looked at TableEG's fake food, they acted exactly the same as they did with Real Spoiled Food. They spotted the errors with the same speed and accuracy.

The Metaphor: It's like training a police dog.

• BART trains the dog with a rubber chicken. The dog learns to chase rubber chickens. When a real criminal runs by, the dog ignores them.
• TableEG trains the dog with a realistic-looking (but fake) criminal. The dog learns to chase the behavior of a criminal. When a real criminal runs by, the dog catches them immediately.

4. The Big Picture

The paper concludes that TableEG is a game-changer because:

1. It's Realistic: It creates errors that look, feel, and act like real human mistakes (typos, missing data, weird numbers).
2. It's Flexible: It can handle different types of data, from movie lists to flight schedules to medical records.
3. It's a Better Benchmark: Now, instead of testing data cleaning tools on fake, boring data, scientists can test them on TableEG's "perfectly imperfect" data. If a tool works on TableEG's data, it will likely work in the real world.

In short: The authors built a "simulator" for data errors. Just as flight simulators train pilots to handle real emergencies without crashing a real plane, TableEG trains data cleaners to handle real-world messiness without needing to find actual messy data first.

Technical Summary

Here is a detailed technical summary of the paper "Towards Practical Benchmarking of Data Cleaning Techniques: On Generating Authentic Errors via Large Language Models."

1. Problem Statement

Data quality is critical for downstream analytics and machine learning, yet evaluating data cleaning techniques is hindered by a lack of diverse, real-world error datasets.

• Limitations of Manual Annotation: Creating labeled error datasets is time-consuming, expensive, and often inconsistent.
• Limitations of Existing Synthetic Methods: Current state-of-the-art methods, such as BART, rely on rule-based approaches (e.g., functional dependencies, denial constraints). These methods suffer from two major flaws:
  1. Lack of Diversity: They generate errors based on predefined patterns (e.g., random character insertion), failing to capture the nuanced, complex semantics of real-world errors.
  2. Constraint Bias: They primarily generate constraint violations, struggling to produce realistic missing values, domain-specific semantic inconsistencies, or outliers that do not violate explicit rules.
• The Challenge: Directly using Large Language Models (LLMs) for error generation is ineffective because standard LLMs lack awareness of two-dimensional table structures (inter-row/column dependencies) and tend to produce superficial or nonsensical corruptions rather than realistic error distributions.

2. Methodology: The TableEG Framework

The authors propose TableEG, a framework that leverages instruction-tuned Large Language Models to generate authentic synthetic errors. The core innovation lies in a structured training paradigm that bridges the gap between natural language processing and tabular data integrity.

A. Triplet Representation $(I, T, O)$

To model table tasks effectively, the framework uses a triplet representation:

• $I$ (Instruction): A natural language directive specifying the task (e.g., "introduce an outlier") and error type.
• $T$ (Input Table): A sampled sub-table representing the structural context.
• $O$ (Output): Structured annotations specifying the error location, type, erroneous value, and the correct value.

B. Multi-Task Learning Strategy

TableEG is trained on a diverse set of tasks to enhance structural awareness and error understanding:

1. Error-Related Tasks:
   • Error Generation (EGT): Given a clean table, generate realistic errors.
   • Error Detection (EDT): Identify erroneous cells in a dirty table.
   • Error Correction (ECT): Restore correct values for detected errors.
   • Rationale: These tasks are coupled; detection and correction provide supervisory signals that help the model understand the nature of errors, improving its ability to generate them.
2. Error-Unrelated Tasks:
   • Includes row/column augmentation, swapping, filtering, header matching, and table summarization.
   • Rationale: These tasks reinforce the model's understanding of 2D table structures and positional dependencies.

C. Training Pipeline

• Base Model: LLaMA3.1-8B.
• Fine-Tuning: Uses LoRA (Low-Rank Adaptation) for efficient parameter tuning.
• Data Source: Trained on 12 real-world datasets spanning 10 domains (e.g., Finance, Healthcare, Entertainment) containing manually annotated errors.
• Generation Process: When generating errors for a new dataset, the system samples sub-tables, constructs specific instructions based on desired error ratios/types, and applies the model's output to inject errors while avoiding overlapping modifications.

3. Evaluation Strategy

The authors propose a comprehensive evaluation framework to measure "authenticity" in two dimensions:

1. Pattern Alignment ($S_{EPA}$): Measures the semantic similarity between the transformation vector of a generated error and real-world errors using cosine similarity in the embedding space (specifically using hidden layer features from the model).
2. Distribution Alignment:
   • Weighted Jaccard Similarity ($J^w_{col}$): Measures how similarly errors are distributed across columns in real vs. generated data.
   • Jensen-Shannon Divergence ($D_{JS}$): Quantifies the divergence in error type distributions (e.g., ratio of outliers vs. missing values).
3. Downstream Utility: Tests whether existing error detection algorithms (e.g., Raha, Holistic) perform similarly on TableEG-generated data compared to real-world dirty data.

4. Key Results

Experiments were conducted on both "seen" (training) and "unseen" datasets, comparing TableEG against BART (rule-based) and GPT-3.5 (Turbo) (zero-shot LLM).

• Superior Pattern Similarity: TableEG achieved an average $S_{EPA}$ score of 77.76%, significantly outperforming BART (49.36%) and GPT-3.5 (50.45%). This indicates TableEG generates errors that are semantically closer to real-world corruption.
• Better Distribution Alignment:
  • On the Flight dataset, TableEG achieved a $J^w_{col}$ of 82.3, compared to 34.69 for BART.
  • TableEG showed significantly lower $D_{JS}$ (divergence), indicating its error type distributions closely match real-world statistics.
• Generalization: TableEG maintained high performance on unseen datasets (e.g., Soccer, Restaurant), demonstrating strong transferability of learned error patterns.
• Benchmark Validity: Error detection algorithms (like Raha) showed nearly identical Precision, Recall, and F1-scores on TableEG-generated data compared to real-world data. This confirms that TableEG creates a realistic "challenge" for detection models, unlike BART which creates artificial, easily detectable errors.

5. Key Contributions

1. Framework Design: Introduced TableEG, the first framework to leverage instruction-tuned LLMs specifically for generating authentic, diverse tabular errors.
2. Structured Methodology: Developed a triplet-based representation and a multi-task learning approach (Generation, Detection, Correction) that enables LLMs to understand complex 2D table dependencies.
3. Comprehensive Evaluation: Established a rigorous evaluation suite ($S_{EPA}$, $J^w_{col}$, $D_{JS}$) to quantify the realism of synthetic errors beyond simple accuracy metrics.
4. Robust Benchmark: Demonstrated that TableEG-generated data serves as a reliable proxy for real-world data, effectively bridging the gap for training and benchmarking data cleaning techniques.

6. Significance

This work addresses a critical bottleneck in the data cleaning field: the scarcity of high-quality, diverse error datasets. By proving that LLMs can be fine-tuned to generate authentic errors that mimic real-world statistical and semantic properties, the authors provide a scalable solution for:

• Training: Generating infinite, diverse training data for error detection and correction models.
• Benchmarking: Creating standardized, realistic testbeds to fairly evaluate new data cleaning algorithms.
• Research: Moving the field away from artificial, rule-based error injection toward data-driven, context-aware error simulation.

The source code and datasets are publicly available, facilitating further research in practical data cleaning benchmarking.