🤖 AI

Symbolic Augmentation Closes a Canonical-Equivalence Blind Spot in Neural Fact-Checkers

This paper addresses the failure of neural fact-checkers to recognize physically equivalent quantities (e.g., 95°C vs. 368.15 K) by introducing a typed-quantity error taxonomy and a "Symbolic Augmentation" training framework that generates label-preserving data, thereby achieving near-perfect robustness to canonical-equivalent rewrites without increasing inference costs.

Original authors: Genpei Zhang

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Genpei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Silent Saboteur in Scientific Summaries

Imagine you are a detective trying to solve a mystery, but the only clues you have are summaries written by a very confident, very fast, but occasionally confused robot. This robot is great at writing smooth sentences and using big words, but it has a dangerous habit: it sometimes swaps out numbers or units without you noticing. In the world of science, this isn't just a typo; it's a disaster. If a robot says a medicine works at "12 milligrams" when the real study said "12 nanograms," the difference is a million times. One is a cure; the other is poison. This is the world of Large Language Models (LLMs), the super-smart AI tools that write summaries of scientific papers for us. While they are amazing at understanding language, they often trip over the hard logic of math and physics, silently inventing facts that sound right but are completely wrong.

To catch these mistakes, scientists usually use "fact-checkers," which are like digital referees. Traditionally, these referees just shout "Yes, that's true" or "No, that's false." But that's not enough when the robot is lying about a specific number. We need a referee that can say, "You got the unit wrong," or "You changed the scale," or "You added a claim that wasn't there." This paper dives into that specific problem: how do we teach AI to spot these sneaky, number-based lies in scientific text, especially when the robot tries to trick us by using a different way of writing the same number (like saying "95 degrees Celsius" instead of "368.15 Kelvin")?

The Paper's Story: Catching the "Shape-Shifting" Numbers

The authors of this paper, Genpei Zhang from the University of Wisconsin-Madison, decided to build a better way to catch these errors. They realized that current AI fact-checkers are like a security guard who can spot a thief in a red shirt but fails completely if the thief puts on a blue shirt, even if it's the exact same person. In the world of science, "red shirt" and "blue shirt" are like canonical equivalents: different ways of writing the exact same physical quantity. For example, $95^\circ\text{C}$ and $368.15\text{ K}$ are the same temperature, just written differently.

The team started by creating a massive training ground called a benchmark. They took real scientific sentences from medical and computer science papers and used AI to rewrite them, intentionally introducing five specific types of errors:

Correct: The summary is perfect.
Unit Error: The unit is wrong (e.g., mixing up mass and volume).
Scale Error: The number is off by a huge amount (e.g., saying 100 instead of 1).
Relation Error: The comparison is flipped (e.g., "higher" becomes "lower").
Unsupported: The summary adds a claim that wasn't in the original text (e.g., "this is the best drug ever").

They built a dataset of 1,500 of these sentences, carefully labeled by two different AI systems and checked by humans to ensure the labels were trustworthy. When they tested a standard, high-tech AI encoder (a type of model called ModernBERT) on this dataset, it did surprisingly well, getting a score of 0.899 (where 1.0 is perfect). This was much better than any off-the-shelf fact-checker they tried.

However, the paper uncovered a shocking blind spot.

When the researchers tested the AI with "shape-shifting" numbers—rewriting correct summaries using canonical equivalents (like changing $95^\circ\text{C}$ to $368.15\text{ K}$ )—the AI's performance crashed. Its accuracy on these specific, physically identical rewrites plummeted from 96.7% down to just 36.5%. The AI was so confused by the different numbers that it thought the correct summary was actually an error nearly two-thirds of the time. It was like a security guard who knows a thief but fails to recognize them if they wear a hat.

The Solution: Symbolic Augmentation

To fix this, the authors introduced a clever trick called Symbolic Augmentation. Instead of trying to teach the AI to do math on the fly (which it is bad at), they decided to teach it during training by showing it examples of these "shape-shifters" beforehand.

They used a simple, rule-based tool (a symbolic verifier) that knows the rules of physics and math perfectly. This tool can instantly tell that $95^\circ\text{C}$ and $368.15\text{ K}$ are the same. The authors ran this tool in "reverse" to generate new training data. They took a correct sentence and used the tool to rewrite the numbers into their equivalent forms, creating thousands of new examples where the answer was still "Correct," but the numbers looked different.

When they trained their AI model on this new, augmented data, the results were dramatic:

The AI's ability to handle these "shape-shifting" numbers jumped from 36.5% to 98.2%.
Its overall accuracy actually improved slightly, from 0.899 to 0.902.
It became so good at this that it matched the performance of massive, expensive "closed-frontier" AI models (like GPT-5.5 or Claude Opus 4.7) but ran much faster and cheaper because it didn't need to do complex math during the actual check.

What Didn't Work (And Why It Matters)

The paper is also very clear about what doesn't work, which is just as important as the success. The authors tried several other ways to combine the rule-based math tool with the AI:

Feeding the math rules as extra input: They tried giving the AI the math answers as a hint while it was guessing. This didn't help at all; the AI ignored the hints.
Using the math tool to label training data: They tried using the math tool to create a huge pile of "silver labels" (guesses) to train the AI. This actually made things worse, because the math tool isn't perfect at understanding context, and its mistakes confused the AI.

The paper concludes that the only way to successfully combine the rigid logic of math with the flexible learning of AI is to bake the math rules directly into the training data itself. By doing this, the AI learns to recognize that different numbers can mean the same thing, closing the blind spot and making scientific fact-checking much more reliable.

In short, the paper shows that while AI is great at language, it needs a little help with the math. By teaching it to recognize the "shape-shifters" of science during its training, we can stop it from silently inverting scientific claims and make it a much safer tool for researchers and students alike.

Technical Summary: Symbolic Augmentation for Neural Fact-Checkers

Problem Statement

Large Language Models (LLMs) frequently hallucinate numbers and units when summarizing scientific text. Unlike paraphrase errors, which are often detectable via context, a single misstated unit or magnitude can silently invert a scientific claim (e.g., changing a concentration from nanomolar to milligram-per-milliliter). Current fact-checking paradigms largely rely on binary or ternary entailment judgments, which fail to distinguish between specific types of quantitative errors (e.g., unit dimension vs. magnitude scale) and provide no signal for targeted repair. Furthermore, while fine-tuned encoders perform well on standard benchmarks, they exhibit a critical structural blind spot: they fail to recognize canonical-equivalent rewrites (e.g., treating $95^\circ\text{C}$ and $368.15\text{ K}$ as errors rather than equivalent statements), causing accuracy to collapse on physically equivalent but textually distinct inputs.

Methodology

1. Task Formulation and Benchmark

The authors recast fact-checking as a five-class typed-quantity verification task. Given an evidence sentence and a summary, the verifier assigns one of five labels:

CORRECT: Faithful restatement, including canonical-equivalent rewrites.
UNIT ERROR: Mismatch in physical dimension (e.g., concentration vs. mass).
SCALE ERROR: Mismatch in magnitude by at least one order of magnitude.
RELATION ERROR: Flipped comparators, swapped entities, or altered scope.
UNSUPPORTED: Addition of claims absent from evidence (superlatives, causal, significance).

To enable systematic study, the authors constructed LLM-rewritten, dual-annotated benchmark of 1,500 items derived from PMC (biomedical/materials science) and arXiv (computer science) sources. The dataset features high inter-annotator agreement ( $\alpha = 0.882$ ) and is balanced across the five classes.

2. The UNITGRAPH VERIFIER (UGV)

A deterministic, rule-based pipeline serves as a symbolic baseline. It operates in four stages without language models:

Extraction & Normalization: Uses quantulum and a custom normalizer to extract and canonicalize quantities.
Dimensional & Magnitude Checks: Compares physical dimensions and logarithmic magnitude differences.
Relational Checks: Detects comparator flips and entity swaps.
Unsupported Triggers: Identifies superlative, causal, or significance phrases absent in the evidence.
UGV achieves a macro- $F_1$ of 0.537, outperforming off-the-shelf neural fact-checkers but falling short of fine-tuned encoders.

3. Symbolic Augmentation Framework

The core contribution is Symbolic Augmentation, a training-time framework that integrates symbolic reasoning into a neural encoder.

Mechanism: The modules of the symbolic verifier (UGV) are run in reverse. For every gold CORRECT item, the system applies deterministic, label-preserving rewrite rules to generate augmented training data.
Four Augmentation Families:
- F1 (Canonical Equivalence): Converts quantities to physically equivalent units (e.g., $^\circ\text{C} \leftrightarrow \text{K}$ ).
- F2 (Comparator Synonyms): Swaps comparative words symmetrically (e.g., "higher" $\leftrightarrow$ "elevated").
- F3 (Unit-name Paraphrase): Substitutes unit symbols with words (e.g., "M" $\leftrightarrow$ "molar").
- F4 (Scientific Notation): Converts between exponential and decimal forms.
Integration: The augmented data is used to fine-tune a ModernBERT encoder. Crucially, at inference time, the symbolic modules are not used; the encoder alone performs the classification.

4. Integration Strategy Comparison

The paper rigorously evaluates five points of integration for symbolic information:

Inference-time Ensembling: Combining UGV and encoder logits (No improvement).
Agreement Filtering: Using UGV only when it agrees with the encoder (No improvement).
Feature-level Concatenation: Feeding UGV outputs as auxiliary input vectors (Performance drop).
Silver-label Teaching: Using UGV to label training data (Severe performance drop due to noise).
Training-time Augmentation: Using UGV rules to generate augmented training pairs (Significant improvement).

Key Results

Benchmark Performance

Baseline: Off-the-shelf neural fact-checkers (e.g., MiniCheck, FactCG) achieve macro- $F_1 \le 0.295$ and score 0.000 on SCALE ERROR, failing to detect magnitude errors.
Fine-tuned Encoder: A ModernBERT encoder fine-tuned on the benchmark reaches macro- $F_1 = 0.899$ , approaching closed-frontier LLMs (e.g., Claude Opus 4.7 at 0.973) at a fraction of the cost.
The Blind Spot: Despite high overall accuracy, the canonical encoder scores only 36.5% on a probe set of canonical-equivalent rewrites (F1), frequently misclassifying them as SCALE or UNIT errors.

Impact of Symbolic Augmentation

Robustness: Training with the F1 (canonical-equivalence) family raises probe accuracy from 36.5% to 98.2%, effectively closing the blind spot.
In-Distribution Accuracy: The augmentation slightly improves overall macro- $F_1$ from 0.899 to 0.902.
Transferability: The augmented model transfers to the external SciFact-Open benchmark, improving binary macro- $F_1$ from 0.791 to 0.828 without any SciFact training data.
Efficiency: The augmented encoder matches the performance of a closed-frontier LLM on canonical-equivalence robustness while running locally with no inference cost for symbolic modules.

Negative Results

The paper demonstrates that simply adding symbolic features as inputs or using the symbolic verifier as a noisy teacher degrades performance. This confirms that the training-time augmentation of data is the necessary integration point to transfer the deterministic guarantees of symbolic logic into the learned representations of the encoder.

Significance and Claims

The paper claims to identify training-time augmentation as the correct mechanism for integrating symbolic and learned components in fact-checking. By deriving label-preserving rewrite rules from a symbolic verifier, the authors demonstrate that:

Blind Spots are Structural: The failure to recognize canonical equivalents is a specific structural gap in neural encoders, not a general lack of knowledge.
Augmentation is Superior to Ensembling: Injecting symbolic knowledge during training (via data generation) is more effective than combining symbolic and neural outputs at inference or feature levels.
Robustness is Transferable: The ability to canonicalize numerical surfaces is a generalizable capability that improves performance on external benchmarks, suggesting that symbolic augmentation instills a fundamental reasoning skill rather than overfitting to a specific dataset.

The work provides a five-class taxonomy, a high-quality benchmark, and a framework that allows small, efficient encoders to match the robustness of much larger models on scientific fact-checking tasks.