Automatic End-to-End Data Integration using Large Language Models

This paper introduces an automatic end-to-end data integration pipeline powered by GPT-5.2 that generates all necessary configuration artifacts, demonstrating comparable or superior performance to human-designed pipelines across multiple case studies at a significantly lower cost.

The Gist

Imagine you are a chef trying to create a massive, perfect recipe book. But instead of having one kitchen, you have three different kitchens (Kitchens A, B, and C) sending you ingredients.

• Kitchen A writes "Tomato" on the label.
• Kitchen B writes "Red Fruit" on the label.
• Kitchen C writes "Solanum lycopersicum" on the label.

Furthermore, Kitchen A lists prices in "Dollars," Kitchen B in "Euros," and Kitchen C in "Yen." Kitchen A lists the chef's name as "John," while Kitchen B lists him as "J. Smith."

The Problem:
Traditionally, to make one unified recipe book, you need a team of human data engineers (let's call them "Super Chefs"). These Super Chefs have to:

1. Read every label and figure out that "Tomato," "Red Fruit," and "Solanum" are the same thing.
2. Convert all the prices to Dollars.
3. Realize "John" and "J. Smith" are the same person.
4. Decide which price is the "correct" one if they conflict.

This takes weeks of manual work, costs a fortune, and is prone to human error.

The Solution (The Paper's Idea):
The researchers asked: What if we hire a super-smart AI (a Large Language Model, or LLM) to do all this thinking for us?

They built a system where an AI (specifically a futuristic version called GPT-5.2) acts as the "Head Chef" who designs the entire process from scratch. The AI doesn't just cook; it writes the instructions, creates the training manuals, and sets up the rules for the kitchen.

How the AI Chef Works (The 4 Steps)

1. The Translator (Schema Matching)

• Human Way: A human reads the labels and manually draws lines connecting "Tomato" to "Red Fruit."
• AI Way: The AI looks at the list of ingredients and the target recipe. It instantly realizes, "Oh, 'Red Fruit' is just a fancy way of saying 'Tomato'." It creates a perfect translation guide in seconds.
• Result: The AI was 100% accurate, even when the labels were nonsense (like "Attribute 1"), because it looked at the content to guess the meaning.

2. The Standardizer (Value Normalization)

• Human Way: A human creates a list: "If you see 'PS4', write 'PlayStation 4'. If you see 'SNES', write 'Super Nintendo'."
• AI Way: The AI knows the whole world of video games. It automatically converts every weird abbreviation or slang term into the standard name. It doesn't just do the easy ones; it handles thousands of variations automatically.
• Result: The AI did a much more thorough job than the humans, who only fixed the obvious ones.

3. The Matchmaker (Entity Matching)

• Human Way: A human has to read thousands of pairs of records and say, "Yes, these are the same company," or "No, these are different." They then teach a computer program based on these examples.
• AI Way: The AI acts as a tireless intern. It looks at the data, picks the most interesting examples, and labels them for the computer to learn from. It's like a master teacher showing a student the best examples to learn from, rather than just random ones.
• Result: The computer trained by the AI performed just as well (or better) than the one trained by humans, but it cost pennies to generate the examples.

4. The Judge (Data Fusion)

• Human Way: If two sources say different things (e.g., one says the movie came out in 2020, another says 2021), a human has to research the internet to find the truth and decide which rule to use.
• AI Way: The AI simulates a research team. It picks famous examples (like "The Beatles" or "Apple Inc.") and asks itself, "What is the truth?" It then uses that answer to set the rules for the whole dataset.
• Result: For static things (like music genres), the AI was perfect. For time-sensitive things (like a company's current revenue), the AI sometimes got confused because its internal knowledge was slightly outdated, but it was still very close.

The Big Reveal: Cost and Speed

The researchers ran this experiment on three real-world scenarios: Video Games, Companies, and Music.

• The Human Team: Took about 19 hours of work per project. It cost a lot of money in salaries.
• The AI Team: Took about 2 hours of computer time. It cost roughly $9 in API fees.

The Verdict:
The final "recipe books" (the integrated datasets) created by the AI were almost identical in quality to the ones made by humans.

• They had the same number of recipes.
• They had the same level of detail.
• They were just as accurate.

The Catch (Limitations)

The AI isn't magic yet.

1. It needs flat data: It struggles if the data is split across many complex, connected tables (like a messy filing cabinet vs. a neat spreadsheet).
2. It might be "memorizing": Since the test data (like famous video games and companies) is likely in the AI's training data, it might have just "remembered" the answers rather than truly "reasoning" them out. We don't know how well it would work on secret, brand-new company data.
3. Time Travel issues: The AI sometimes gets confused about when something happened because its knowledge cutoff is in the past, while the data might be from the future (or vice versa).

The Bottom Line

This paper proves that we can replace the expensive, slow, manual work of data engineers with an AI that acts as an "Auto-Pilot" for data integration. It's like upgrading from a team of scribes manually copying books to a high-speed photocopier that also edits the text.

For standard data tasks, the AI is now ready to take the wheel, saving companies time and money while producing results that are just as good as the human experts.

Technical Summary

Here is a detailed technical summary of the paper "Automatic End-to-End Data Integration using Large Language Models" by Aaron Steiner and Christian Bizer.

1. Problem Statement

Data integration is a complex process involving schema matching, value normalization, entity matching, and data fusion. Traditionally, this requires substantial manual effort from data engineers to:

• Configure pipeline components.
• Label training and validation data.
• Select and tune algorithms.

While Large Language Models (LLMs) have shown promise in automating individual steps (e.g., schema matching or entity matching), it remains uninvestigated whether LLMs can replace all human input to create a fully automated, end-to-end integration pipeline. The core research question is: To what extent can an LLM autonomously generate all necessary artifacts (mappings, training data, validation sets) to configure a data integration pipeline, matching or exceeding the quality of human-engineered pipelines?

2. Methodology

The authors propose a fully automated pipeline powered by GPT-5.2 (a hypothetical future model in the context of this paper, dated March 2026) that integrates with the PyDI open-source data integration framework. The pipeline operates through four distinct stages:

A. Schema Matching

• Approach: The LLM is prompted with source table representations (column names, value summaries, sample rows) and the target schema (JSON format).
• Mechanism: It generates a JSON array of correspondences. Crucially, for datasets with non-descriptive headers (e.g., "Attribute 1"), the LLM infers semantics based on value patterns and distributions.
• Comparison: Compared against human manual mapping and traditional label/instance-based matchers (Monge-Elkan, TF-IDF).

B. Value Normalization

• Approach: Uses a hybrid strategy.
  • Standard Types: Uses deterministic code-based normalizers (Python libraries like Babel, Pint, pycountry) for dates, numbers, and units.
  • Categorical/Taxonomic: The LLM generates mappings to unify synonyms, abbreviations, and hierarchical concepts (e.g., mapping "PS4" to "PlayStation 4" or industry codes to GICS taxonomy).
• Comparison: Compared against human-configured normalization, which was limited to specific use cases (e.g., only platform names in the Games dataset).

C. Entity Matching

• Approach: Replaces manual labeling and rule configuration with an Active Learning workflow.
  1. Candidate Generation: Embedding-based blocking (using text-embedding-3-small) retrieves $k=20$ nearest neighbors.
  2. Seed Labeling: GPT-5.2 labels pairs from the candidate pool, prioritizing high-similarity pairs until a minimum of positive/negative examples is found.
  3. Iterative Training: Machine Learning classifiers (RandomForest, XGBoost, etc.) are trained on string similarity features. The active learning loop identifies pairs where classifiers disagree most, sends them to the LLM for labeling, and retrains the model.
  4. Selection: The best classifier and threshold are selected based on an LLM-labeled validation set.
• Comparison: Compared against human-configured rules and ML models trained on human-labeled data.

D. Data Fusion (Conflict Resolution)

• Approach: The pipeline generates validation sets to select conflict resolution heuristics (e.g., voting, median, most recent).
  • Validation Generation: GPT-5.2 selects well-known entities and generates ground truth values.
  • Variants: Tested "LLM-only" (parametric knowledge) vs. "RAG" (using OpenAI web search to verify current values).
  • Selection: The pipeline iterates through candidate resolver configurations and selects the one with the highest accuracy on the generated validation set.
• Comparison: Compared against human-configured strategies selected via manual web searches.

3. Key Contributions

1. Fully Automated Pipeline: A system that uses an LLM to perform all configuration tasks (schema matching, normalization mapping, training data generation, and validation set creation) without human intervention.
2. Comprehensive Evaluation: A rigorous comparison across three diverse domains (Video Games, Companies, Music) involving heterogeneous data sources, comparing the LLM pipeline against pipelines designed by graduate-level data engineers.
3. Cost and Efficiency Analysis: A detailed breakdown of the time and financial costs, demonstrating that the LLM pipeline reduces configuration effort by approximately 10x (from ~19 person-hours to ~2 hours of compute) and costs only ~$9 per case study.
4. Open Benchmark: The release of all datasets, training/validation sets, and code to facilitate future research in end-to-end data integration.

4. Results

The study evaluated the pipelines using three case studies (Games, Companies, Music) and found the following:

• Schema Matching: The LLM achieved 100% F1 score (67/67 correct correspondences), matching human performance perfectly, even on datasets with non-descriptive headers where traditional algorithms failed (F1 < 0.4).
• Entity Matching:
  • ML matchers trained on LLM-labeled data achieved the highest average F1 (0.937), outperforming human-labeled data (0.916) and manually configured pipelines (0.894).
  • The active learning approach was highly cost-effective, labeling only 600–2,600 pairs per case study (costing ~$5 total) compared to the cost of labeling all candidates directly ($466).
• Data Fusion:
  • Static Domains (Games, Music): LLM-generated validation sets (both RAG and non-RAG) selected configurations with accuracy comparable to human validation (e.g., Games: 0.866 vs 0.832 human config).
  • Dynamic Domains (Companies): LLM-only validation struggled with time-sensitive attributes (e.g., revenue, headquarters) due to temporal mismatches between the dataset's historical data and the LLM's current knowledge. RAG improved this slightly but did not fully close the gap.
• End-to-End Metrics:
  • Output Size: LLM and Human pipelines produced comparable output dataset sizes and row gains.
  • Density: Results were mixed; the LLM pipeline improved density in the Companies case (+5.6pp) but decreased it in Music (-1.8pp), suggesting the optimal strategy depends on the specific data structure.
  • Cost: The LLM pipeline cost approximately $9–$10 per case study, whereas the human baseline required an estimated 58 person-hours (valued significantly higher).

5. Significance and Limitations

Significance:
The paper demonstrates that LLMs can effectively replace human data engineers in configuring end-to-end data integration pipelines. The results show that for standard integration scenarios, an automated LLM pipeline can achieve comparable or superior quality to human-designed pipelines at a fraction of the cost and time. This suggests a paradigm shift where data integration becomes a "low-code" or "no-code" process driven by AI agents.

Limitations:

• Data Distribution: The strong performance may be partly due to the LLM's training data containing the specific public datasets used (Games, Companies, Music). Performance on proprietary or niche data outside the training distribution is unproven.
• Schema Complexity: The pipeline assumes flat tables with 1:1 attribute correspondences. It does not yet handle normalized tables, 1:n relationships, or complex data aggregation.
• Temporal Knowledge: The pipeline struggles with time-sensitive attributes where the source data is historical but the LLM's knowledge is current (e.g., company revenue).
• Bias in Validation: The use of "well-known" entities for validation introduces bias, potentially overestimating performance for obscure entities.

Future Directions:
The authors suggest future work in agentic architectures that allow feedback loops (e.g., using downstream fusion errors to refine upstream entity matching), human-in-the-loop collaboration for critical review, and the use of coding agents to dynamically generate normalization functions.