💻 computer science

Auditing and mitigating high-confidence false relation edges in LLM-assisted scientometric extraction

This paper proposes and evaluates a perturbation-based audit and mitigation framework that demonstrates how entity noise increases high-confidence false relation risks in LLM-generated scientometric graphs, revealing that entity- and evidence-aware abstention strategies effectively outperform simple confidence filtering in preserving graph integrity.

Original authors: Ce Yin, yunqiu zhang

Published 2026-09-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ce Yin, yunqiu zhang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast library of human knowledge, scientists and librarians are increasingly turning to artificial intelligence to organize the flood of new research. They use these smart computer programs to read millions of scientific abstracts and pull out specific facts: which researcher invented a new method, which software tool was used for a specific task, or how one discovery connects to another. These extracted facts are then stitched together to build massive digital maps of science, helping researchers find patterns, track the history of ideas, and evaluate the impact of different fields. For these maps to be useful, the connections between facts must be accurate. If the computer links a method to the wrong task, or claims a connection that doesn't exist, the entire map becomes distorted, leading to confusion rather than clarity.

The core challenge lies in how these systems handle the building blocks of a sentence: the entities. In a scientific text, an entity is a specific name or term, like a drug name or a research technique. The computer must first identify these names correctly and then decide if a relationship exists between them. A new study by researchers at Jilin University investigates what happens when the computer gets the names slightly wrong. They asked a critical question: if the input data is corrupted or slightly altered, will the artificial intelligence still confidently invent a relationship that isn't there? Their findings reveal a troubling tendency where the system, despite receiving broken or mismatched information, often insists on creating a strong connection with high certainty, effectively hallucinating facts that could mislead the entire scientific community.

To test this, the researchers set up a controlled experiment using two different collections of scientific text. One dataset contained general scientific abstracts covering a wide range of topics, while the other focused on a narrower field of diabetes research. They took pairs of entities that were known to have a real relationship and deliberately introduced small errors. In some cases, they shifted the boundaries of a word, cutting off a letter or adding one. In other cases, they kept the word exactly the same but told the computer it belonged to a different category, such as labeling a software tool as a chemical compound. The goal was to see how the artificial intelligence reacted when the evidence for a relationship was fundamentally flawed.

The results showed that when the input data was corrupted, the computer's performance dropped significantly, but not in the way one might hope. Instead of simply saying "I don't know" or refusing to make a connection, the system frequently generated a specific relationship anyway. More concerning was that it did so with high confidence. In the general science dataset, when the researchers introduced these errors, the rate of these confident but false connections jumped from a baseline of about 24 percent to nearly 49 percent. In the diabetes dataset, the risk also increased, though the overall numbers were lower, suggesting that the structure of the scientific field itself can sometimes buffer against these errors. The study found that the computer was not just guessing; it was confidently asserting facts based on invalid evidence, a phenomenon the researchers call "entity-evidence substitution."

The researchers discovered that the computer was relying on broad patterns it had learned from its training data rather than the specific evidence in front of it. Even when the specific names didn't match up correctly, the system saw a plausible pattern and filled in the gap with a confident assertion. For example, if the text mentioned a method and a task, the computer might link them together even if the specific names of the method or task had been garbled. The system's internal logic seemed to prioritize the general shape of the sentence over the actual validity of the words, leading it to produce a relationship that looked correct on the surface but was built on a broken foundation.

This behavior has serious implications for how we build knowledge maps. The researchers simulated what would happen if these false connections were added to a network of scientific facts. They found that these confident errors did more than just add a few wrong lines; they changed the structure of the entire network. The false connections created new pathways between unrelated ideas and shifted the importance of certain nodes, making some topics appear more central or connected than they truly are. In a real-world scenario, this could mean that a researcher looking for a specific tool might be led down a false path, or that the history of a scientific field could be rewritten with incorrect links.

To address this, the study tested a new safety mechanism called "entity-aware abstention." Instead of relying solely on the computer's self-reported confidence score, this approach asks the system to double-check its own work. Before finalizing a connection, the system is instructed to verify that the names it is linking are valid and that there is clear text evidence supporting the specific relationship. When this extra check was applied, the number of retained false connections dropped significantly. In the general science dataset, this method reduced the rate of high-confidence false edges more effectively than simply filtering out low-confidence answers. The study suggests that for artificial intelligence to be a reliable partner in science, it must be taught to recognize when its own evidence is weak and to step back rather than force a connection.

The research concludes that while large language models are powerful tools for organizing scientific knowledge, they are not infallible. They can be tricked by subtle errors in the input, leading to confident but false conclusions that can ripple through scientific databases. The study provides a practical framework for auditing these systems, showing that checking the validity of the building blocks is just as important as checking the final answer. By combining confidence scores with direct checks on the evidence, scientists can build more robust systems that are less likely to invent connections where none exist, ensuring that the maps of human knowledge remain accurate and trustworthy.

Technical Summary: Auditing and Mitigating High-Confidence False Relation Edges in LLM-Assisted Scientometric Extraction

1. Problem Statement

The paper addresses a critical reliability gap in using Large Language Models (LLMs) for scientometric data production, specifically in relation extraction for knowledge graphs and science maps. While LLMs effectively convert scientific text into semantically enriched representations, their outputs depend heavily on the integrity of upstream entity recognition.

The core problem identified is entity-noise-induced overconfidence: when upstream entity evidence is corrupted (e.g., via boundary shifts or type substitutions), LLMs often still predict substantive relations with high reported confidence. These "high-confidence false relation edges" are particularly dangerous because they survive downstream filtering and enter scientometric infrastructure (e.g., knowledge graphs, retrieval systems, evaluative indicators) as valid connections, potentially distorting field mapping and research assessment.

The study posits that standard accuracy metrics and confidence-only filtering are insufficient to detect these failures, as models may rely on broad relation-schema priors and surface co-occurrence cues to "complete" a relation even when specific entity-level evidence is invalid.

2. Methodology

The authors developed a perturbation-based audit and mitigation framework to quantify and address this risk.

Experimental Design

Datasets:
- SciERC: The primary setting, comprising annotated entities and relations from scientific abstracts (open scientific schema).
- Diabetes: A boundary-condition setting from the Heywhale Community, featuring a constrained biomedical relation schema to test domain regularity effects.
Perturbation Conditions: Clean positive relations were transformed into "expected-NA" (Not Applicable) cases through deterministic perturbations:
- Boundary Noise: One-character expansion or truncation of head/tail entity mentions.
- Type Noise: Replacement of the displayed entity type with another valid type from the inventory, while preserving the mention string.
Models: Experiments were conducted using DeepSeek v4 flash and Kimi K2.5 (in thinking mode).
Protocol: The pipeline follows a pipelined extraction approach where entities are identified, and then relations are classified. The study evaluates how the relation classifier behaves when presented with corrupted entity pairs.

Metrics

The audit evaluates performance beyond standard accuracy, focusing on:

Expected-NA False-Positive Rate (NA-FPR): The rate at which corrupted pairs receive non-NA labels.
High-Confidence Expected-NA False-Positive Rate (HC-NA-FPR): The rate at which these errors occur with reported confidence $\ge$ 0.8.
Reported-Confidence ECE: Expected Calibration Error to measure the alignment between self-reported confidence and empirical correctness.
Relation-Graph Stress Tests: Quantifying downstream impacts on graph connectivity, density, and degree centrality when false edges are retained.

Mitigation Strategy

The study tests Entity-Aware Abstention, a screening mechanism where the model's own reported fields (entity validity, evidence availability, and rationale) are used to gate predictions. This is compared against Confidence-Only Filtering (rejecting based solely on low confidence scores).

3. Key Results

Amplification of Risk under Entity Noise

Accuracy Degradation: In SciERC, entity noise reduced accuracy from ~0.57 (clean) to a range of 0.27–0.42. In the Diabetes dataset, accuracy degradation was less pronounced, suggesting domain regularity buffers some errors.
High-Confidence Errors: Entity noise significantly increased HC-NA-FPR, though the magnitude varied by noise type and domain.
- SciERC: HC-NA-FPR rose from 0.243 (clean NA) to 0.452–0.520 across all noise conditions.
- Diabetes: The increase was driven primarily by boundary noise, which raised HC-NA-FPR from 0.035 to 0.244–0.266, while type noise produced smaller increases (0.067–0.136).
Calibration Failure: Reported confidence remained high (often >0.8) even for these invalid predictions, indicating that confidence scores are poor proxies for correctness in the presence of entity corruption.

Mechanism Evidence: Entity-Evidence Substitution

Analysis of model outputs revealed that in high-confidence false edges:

Models often generated invalid-entity cues in their internal validity or rationale fields (e.g., noting weak evidence).
However, the final relation label was still asserted as substantive.
This supports the entity-evidence substitution hypothesis: the model's decision is driven by strong schema priors and co-occurrence patterns, overriding the weak or invalid entity-specific evidence.

Graph-Level Impact

Stress tests on the SciERC relation graph demonstrated that retaining these false edges:

Increased the number of asserted edges significantly (e.g., from 351 to 807 in DeepSeek).
Altered connectivity (largest weak component size increased) and degree centrality (shifting which mentions appeared central).
Introduced broad, potentially misleading labels (e.g., "Used-for," "Conjunction") that blur method-task links.

Mitigation Effectiveness

Entity-aware abstention proved superior to confidence-only filtering:

At matched coverage levels (~90%), entity-aware strategies reduced retained HC-NA-FPR more effectively.
In SciERC, hybrid strategies (combining entity cues and evidence checks) reduced retained HC-NA-FPR from 0.488 (confidence-only) to 0.415.
In Diabetes, the reduction was from 0.151 to 0.049.
This confirms that checking entity validity and textual evidence directly is more effective than relying on the model's self-reported confidence.

4. Key Contributions

Definition of a Pipeline-Level Reliability Problem: The paper formally defines the risk of corrupted entity evidence generating high-confidence false relation edges in LLM-assisted scientometrics.
Transparent Perturbation Protocol: It introduces a method to separate clean positives, clean expected-NA pairs, and specific noise types (boundary vs. type), reporting distinct risk profiles for each.
Downstream Risk Quantification: It links extraction errors to downstream representation risks via relation-graph stress tests, quantifying how false edges alter graph topology and centrality.
Practical Mitigation Strategy: It validates entity-aware and evidence-aware abstention as a more effective screening control than confidence-only filtering, providing a workflow for safer integration of LLM outputs into knowledge graphs.

5. Significance and Claims

The paper claims that for LLM-extracted relations to be safely used in scientometric infrastructure (knowledge graphs, science maps, evaluative indicators), they require entity-level validation before downstream reuse.

Reliability Audit: The study provides a reproducible audit framework that goes beyond standard benchmark accuracy to test how pipelines behave when upstream entity evidence is defective.
Limitation of Confidence: It demonstrates that reported confidence is not a safe proxy for risk in this context; models can be highly confident about incorrect relations derived from corrupted entities.
Mitigation: The findings support a workflow where entity validity and evidence availability are treated as first-order risk signals. The authors argue that combining these checks with reported confidence and selective human review offers a practical path to mitigating the propagation of false edges into scientific knowledge structures.

The study remains modest regarding generalizability, noting that effect sizes are specific to the audited settings (SciERC, Diabetes, specific models) and may vary with different model families, decoding configurations, and relation schemas. However, the mechanism of entity-evidence substitution is presented as a generalizable explanation for this failure mode.