🔬 materials science

No Value Is Better Than a Wrong Value: Finding and Handling Erroneous Data in Literature-Mined Datasets for AI

This paper addresses the critical challenge of erroneous data in literature-mined AI datasets by proposing a taxonomy of extraction errors and a layered verification framework that combines physical laws, statistical consensus, and provenance checks to significantly improve model accuracy and prevent false discoveries in materials science and related fields.

Original authors: Achyut Manoj, Advai Aiyer, Andrew Lee, Nayan Saini, Pragya Birla, Dibakar Datta

Published 2026-10-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Achyut Manoj, Advai Aiyer, Andrew Lee, Nayan Saini, Pragya Birla, Dibakar Datta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To discover new materials for faster computers, longer-lasting batteries, or more effective medicines, scientists increasingly turn to artificial intelligence. These digital assistants learn by studying vast collections of data, hoping to find patterns that reveal which chemical combinations will work best. For many properties, such as the energy a battery can hold or how a crystal bends light, the only place these numbers exist is scattered across thousands of published scientific papers. Researchers have built tools to read these papers automatically, pulling out numbers and turning them into databases that machines can understand. However, the process of reading a printed page and turning it into a clean list of numbers is far from perfect. A decimal point can vanish, a unit of measurement can be confused with another, or a value can be attached to the wrong material entirely. If a computer learns from these mistakes, it will not just make small errors; it will confidently point researchers toward the wrong discoveries, often selecting the most extreme and incorrect numbers as the best candidates.

A team of researchers at the Dibakar Datta Laboratory set out to solve this problem by treating data errors not as a rare glitch, but as an expected part of the process. They focused on electro-optic materials, which are crystals that change how they bend light when an electric voltage is applied. These materials are crucial for the fiber-optic networks that carry data across the globe and for future quantum computers. The team built a system to extract values for these materials from the scientific literature, but they designed it with a specific philosophy: no value is better than a wrong value. Instead of trying to extract every number perfectly, which they found impossible, they built a multi-layered safety net to catch mistakes before they could poison the artificial intelligence models.

The researchers identified that errors enter the database at several distinct stages. Sometimes the mistake is in the original paper, where a value is misprinted or copied incorrectly from an older study. Other times, the error happens when a computer program tries to read a scanned document, mistaking a zero for the letter O or losing a negative sign. Even when the text is read correctly, the computer might attach a number to the wrong label, confusing a measurement of one material with another, or it might miss the specific conditions under which the measurement was taken, such as the temperature or the pressure. To handle this, the team created a five-step filtering process. The first step checks if the data looks right on the surface, ensuring units are correct and that the number actually appears in the source text. The second step applies the laws of physics, checking if a value is physically possible, such as ensuring a crystal's symmetry rules are not violated or that a battery's capacity does not exceed its theoretical limit.

The third step looks for agreement between different studies. If ten papers report a value for the same material, the correct numbers should cluster together, while random errors will stand out as outliers. The fourth step uses an artificial intelligence model trained on the rest of the data to spot values that look strange compared to similar materials. The final step is a targeted review by a human expert, who looks only at the values that the previous steps flagged as suspicious. This approach allows the team to handle errors efficiently without needing a human to check every single number. In their latest run, this system stopped 402 out of 816 candidate values, recording a specific reason for why each one was rejected or held for further review.

Despite these strict filters, the researchers found that errors still slipped through. When they compared the values their system accepted against a carefully curated reference set, they discovered that about 27 percent of the accepted values were still defective. Simple checks that did not require a reference database, such as verifying that a crystal's symmetry rules were followed, caught 70 percent of these remaining mistakes. In a controlled test where they knew exactly which numbers were wrong, their layered system found 81 percent of the errors with high precision. Crucially, they showed that without this cleaning process, the artificial intelligence models would have been completely misled. In their uncleaned data, 89 percent of the twenty largest values were errors. Since AI models designed to find the best materials naturally look for the largest values, an uncleaned database would lead researchers to the worst possible candidates, mistaking a massive error for a breakthrough discovery.

The study also revealed a second, more subtle problem. Even after the wrong numbers were removed, the researchers found that many records were still missing the context needed to make them useful. For example, two records might list the same material and the same value, but one was measured under high pressure and the other at room temperature, and the system could not tell them apart because the pressure information was missing from the text. This lack of context, which the researchers measured as a fraction of indistinguishable records, became the main limit on how well the AI could learn. The team concluded that the future of data-driven science depends not just on finding more numbers, but on building databases that track their own quality, report their own errors, and store the specific conditions under which every measurement was taken. By admitting that errors are inevitable and building systems to catch them, scientists can ensure that the artificial intelligence they rely on is learning from reality, not from a collection of mistakes.

Technical Summary: No Value Is Better Than a Wrong Value

1. Problem Statement

Machine learning in materials science, chemistry, and medicine increasingly relies on datasets constructed by mining published literature, as curated databases often do not exist for many functional properties. While automated extraction tools (rule-based and Large Language Models) can process thousands of papers, they are not error-free. Errors arise from optical character recognition (OCR) failures, unit confusion, misattribution of values to the wrong material or condition, and the propagation of cited values as new measurements.

The core problem is that these errors are not merely noise; they are often systematic and can dominate the "extreme" values that AI models seek during inverse design (discovery of materials with maximum properties). A single erroneous value can distort a model's training gradient more than thousands of correct records. The paper argues that the goal should not be an error-free database (which is unattainable at scale) but a database where errors are detected, labeled, and prevented from corrupting AI models.

2. Methodology: A Layered Detection Framework

The authors propose a five-layer detection framework, organized by increasing cost and data requirements, to identify and handle erroneous data. This framework is illustrated using a case study of an electro-optic (Pockels) coefficient database built from literature.

The Five Layers:

Layer 0 (Schema, Units, and Provenance): Checks if a record is well-formed. It verifies labels against controlled vocabularies, checks unit dimensions, ensures required context fields exist, and validates that values are supported by verbatim snippets in the source text (preventing hallucinations). It also identifies cited values versus original measurements.
Layer 1 (Physics): The only layer capable of rejecting values without comparing them to other data. It applies:
- Hard Bounds: Physical limits (e.g., refractive index $\ge 1$ , efficiency $\le 100\%$ ).
- Symmetry Rules: Neumann's principle to reject tensor components forbidden by crystal point groups.
- Constitutive Identities: Governing equations (e.g., relations between clamped/unclamped electro-optic tensors, piezoelectric coefficients, and elastic constants) to detect inconsistencies within a single report.
Layer 2 (Cross-Source Statistical Consensus): Uses robust statistics (median and Median Absolute Deviation) to identify outliers among multiple reports of the same quantity. It includes a "decade-shift repair" mechanism to correct unit slips (e.g., $10^{-11}$ vs. $10^{-12}$ ) if the error is an integer power of ten. It also calculates an "indistinguishable fraction" ( $\Phi$ ) to measure how many records share identical inputs but different outputs due to missing context.
Layer 3 (Model-Based Auditing): Trains a surrogate model on the dataset and flags values that deviate significantly from the model's out-of-fold predictions. This layer borrows strength from chemically similar materials, allowing it to flag errors in materials reported only once (the "long tail").
Layer 4 (Targeted Human Review): Human experts review a shortlist of values ranked by a combined suspicion score derived from the previous layers. This minimizes manual effort by focusing only on the most suspicious records.

Handling Flagged Values:
The framework does not silently delete data. Flagged values are assigned one of five outcomes: accepted (with a weight), repaired (if the error is verified, e.g., a unit shift), held (for review), down-weighted (in training), or rejected (if they violate hard physical constraints). Every decision is logged with a typed reason.

3. Key Contributions

Taxonomy of Errors: A classification of errors based on where they arise in the pipeline (source, conversion, extraction, context, harmonization) and the specific "signature" they leave in the data (e.g., decade shifts, symmetry violations, within-source conflicts).
Layered Detection Framework: A systematic approach combining schema checks, physics constraints, statistical consensus, and model auditing, explicitly stating what each layer can and cannot detect.
Quantitative Demonstrations:
- Analysis of a real production run of an electro-optic pipeline.
- An audit of accepted values against a curated reference.
- A demonstration of how constitutive identities expose single-value errors in real tensor data.
- A controlled synthetic benchmark with known ground truth to measure detection rates.
Impact Analysis on AI: A quantitative assessment of how errors affect model accuracy, evaluation metrics, and, crucially, inverse design (ranking).

4. Results

Real Production Data (Electro-Optic Pipeline)

In the latest production run, the verification gate stopped 402 of 816 candidate values (49%), recording a reason for each.
Despite the gate, an audit against a curated reference of three benchmark papers revealed that 30 of 112 (27%) accepted values were still defective.
Simple reference-free checks (symmetry rules for all crystal classes, label checks, and constitutive identities) caught 21 of the 30 defective records (70%).
On measured BaTiO $_3$ and KNbO $_3$ tensors, constitutive identities caught 8 of 10 injected realistic single-value errors and correctly pointed to the corrupted value.

Controlled Synthetic Benchmark

In a benchmark with 15% injected error rates, the combined layered framework achieved 81% recall at 85% precision.
The framework successfully caught 96–98% of label swaps and context errors, and all unit/scale errors.
Redundancy Dependence: Consensus (Layer 2) is blind to materials reported once or twice. However, the model-based audit (Layer 3) maintains high recall for these "singletons" by leveraging chemical similarity.
Inverse Design Impact: Without cleaning, 89% of the twenty largest values in the dataset were errors. After cleaning, this dropped to 1%. This demonstrates that uncleaned data leads AI discovery pipelines to select errors as the "best" materials.

The Binding Constraint: Missing Context

Once wrong numbers are controlled, the primary limitation becomes missing measurement conditions. The authors define an "indistinguishable fraction" ( $\Phi$ ), which measures the proportion of records that share all recorded features but have different values. In their electro-optic corpus, $\Phi$ ranged from 39% to 72%, indicating that a model cannot learn from these records because the distinguishing conditions were never extracted.

5. Significance and Claims

The paper claims that "no value is better than a wrong value." Its significance lies in shifting the paradigm from seeking perfect extraction to building error-resilient databases.

For Discovery: The paper argues that error handling must precede inverse design. Because optimization algorithms seek extreme values, and errors often manifest as extreme outliers, uncleaned data systematically biases discovery toward incorrect materials.
For Modeling: Cleaning data restores model accuracy to within ~3% of a model trained on error-free data. Conversely, errors in test sets inflate evaluation metrics and hide real improvements.
For the Field: The authors advocate for:
- Measured error rates for real extractors (currently lacking).
- Shared, machine-readable physics rule libraries.
- Databases that report their own quality (including $\Phi$ and provenance) rather than just values.
- Upstream improvements in scientific reporting (structured data tables).

The paper concludes that a database which knows which values to doubt is more valuable to AI-driven science than a larger database that treats every value as correct.