💬 NLP

Ontology-Driven Structural Regularization for Document-Level Relation Extraction

This paper introduces an ontology-driven structural regularization framework that identifies and mitigates logical inconsistencies in distant supervision datasets, thereby significantly improving the generalization performance of document-level relation extraction models.

Original authors: Laura Menotti, Stefano Marchesin, Gianmaria Silvello

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Laura Menotti, Stefano Marchesin, Gianmaria Silvello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, unstructured ocean of human language, computers struggle to find the hidden connections that turn a collection of words into a map of knowledge. To build a machine-readable encyclopedia, or a knowledge graph, researchers must teach computers to read a full document and identify every relationship between the people, places, and things mentioned within it. This task, known as document-level relation extraction, is the foundation for systems that answer complex questions or recommend products. However, teaching these systems is expensive because it requires humans to meticulously label every connection by hand. To solve this, scientists have turned to "distant supervision," a method that automatically creates massive training datasets by matching text to existing databases. While this approach generates millions of examples, it is notoriously noisy, often teaching the computer incorrect facts because the automatic matching process is imperfect.

A team of researchers at the University of Padua has discovered that the problem with these noisy datasets goes deeper than just wrong facts; the data itself is often structurally broken. They found that the automatically generated triples—statements linking a subject, a relationship, and an object—frequently violate basic logical rules. For instance, a computer might be taught that a person was born in a year, or that a city is the capital of another city, creating contradictions that defy the very logic the system is trying to learn. The researchers argue that these structural inconsistencies are a critical, overlooked source of error that poisons the training process, causing models to learn bad habits that persist even when they are tested on clean data.

To address this, the team developed a framework that acts like a logical editor for these datasets. They used a formal system of rules, similar to the grammar of a language but applied to the structure of facts, to scan millions of automatically generated examples. They looked for specific types of errors: triples where the entities did not fit the relationship, missing reverse connections that should logically exist, and impossible contradictions where a relationship appeared in both directions when it should only go one way. When they applied this scan to the massive distant supervision dataset known as DocRED, they found a startling amount of structural noise. The automated dataset contained three times as many invalid facts as the carefully human-annotated versions, nearly five times as many missing reverse connections, and twice as many logical contradictions.

The researchers then took a decisive step to see if fixing these structural flaws would actually help the computers learn better. Instead of trying to manually correct every single error, which would be impossible at this scale, they built a cleaning pipeline that simply removed any data points that violated the logical rules. They stripped out the problematic examples and added the missing reverse connections where they were clearly required by logic. They then trained two of the most advanced computer models on this cleaned data and compared the results to models trained on the original, messy data. The results were clear: the models trained on the structurally consistent data made far fewer logical errors. The number of impossible predictions dropped significantly, and the models became much better at understanding the direction of relationships.

Perhaps most importantly, this structural cleaning did not just make the models more logical; it made them more accurate. When tested on standard benchmarks, the models trained on the cleaned data showed consistent improvements in their ability to find the correct relationships. The study suggests that by enforcing a basic standard of structural well-formedness before training even begins, researchers can unlock the full potential of massive, automated datasets. This approach offers a new way to leverage the scale of distant supervision without being held back by its inherent noise, proving that sometimes the best way to teach a machine is to ensure the lessons it learns make logical sense.

Technical Summary: Ontology-Driven Structural Regularization for Document-Level Relation Extraction

Problem Statement

Document-Level Relation Extraction (DocRE) is critical for constructing Knowledge Graphs (KGs) but relies heavily on costly, manually annotated datasets. While large-scale distant supervision (DS) resources like the DocRED distant dataset offer a solution, they remain underutilized due to significant noise. Previous research has primarily focused on semantic noise (e.g., incorrect relation labels) and has employed model-driven denoising strategies (e.g., uncertainty estimation, active learning).

This paper identifies a critical, overlooked source of noise: structural inconsistencies within relational triples. These inconsistencies include violations of ontology constraints (e.g., domain/range mismatches), missing inverse relations, logical contradictions in asymmetric relations, and cardinality conflicts. The authors argue that these structural flaws constitute a systematic source of supervision noise that propagates during model training, impairing the model's ability to learn and generate well-formed triples.

Methodology

The authors propose an ontology-driven framework to quantify and enforce structural consistency in DocRE datasets. The approach treats DocRE not merely as a classification task but as the construction of a KG subject to Web Ontology Language (OWL) axioms.

1. Structural Rule Definition

The methodology defines four specific types of structural violations based on OWL semantics and Wikidata metadata:

Invalid Triples: A triple $\langle s, p, o \rangle$ is invalid if the entity types of the subject ( $s$ ) or object ( $o$ ) do not satisfy the domain ($dom(p)$) or range ($ran(p)$) constraints defined for the relation $p$ .
Missing Inverse Relations: Under a local closed-world assumption, if a relation $r$ exists between entities $e_s$ and $e_o$ , its declared inverse $r'$ must also exist in the extracted graph.
Asymmetric Relation Violations: For relations defined as asymmetric (non-symmetric), the simultaneous presence of $\langle e_s, r, e_o \rangle$ and $\langle e_o, r, e_s \rangle$ constitutes a logical contradiction.
Cardinality Violations: These occur when the number of distinct object entities for a subject-relation pair exceeds the defined maximum cardinality (e.g., functional properties where $k_r=1$ ).

2. Data Analysis and Cleaning Pipeline

The authors operationalize this framework to analyze the DocRED and ReDocRED datasets, with a specific focus on the distant supervision split.

Quantification: They measure the prevalence of each violation type across manual and distant splits.
Error Attribution: Using binomial proportion estimation on a sampled subset of invalid triples, they determine the root causes (entity annotation errors vs. relation annotation errors).
Pre-processing Pipeline: To mitigate these issues without altering model architectures, they introduce a lightweight, model-agnostic cleaning pipeline:
- Invalid Triples: Instead of manual correction, they remove the entity involved in the violation from the training set, thereby eliminating all associated invalid triples.
- Missing Inverses: Automatically added to the dataset.
- Asymmetric Violations: Entities involved in contradictions are removed.
- Cardinality: Addressed by the removal strategy where necessary (though violations were found to be rare).

Key Contributions

Ontology-Driven Framework: A rigorous method to diagnose structural consistency in DocRE datasets using OWL constraints, shifting the focus from semantic noise to structural well-formedness.
Quantitative Analysis: The first in-depth study of structural inconsistencies in the DocRED distant dataset and its denoised variants (UGDRE, DOREMI), revealing extensive, previously unreported violations.
Structural Regularization Strategy: Demonstration that enforcing structural well-formedness as a pre-processing step acts as a form of regularization, reducing logical contradictions and improving generalization.
Open-Source Toolkit: Release of a cleaning framework compatible with any dataset formatted in the DocRED schema, allowing for custom constraint definitions.

Results

The study was evaluated using two state-of-the-art DocRE models: ATLOP and DREEAM, trained on the DocRED distant dataset and its structurally cleaned version.

Structural Consistency in Predictions

Training on structurally inconsistent data leads to noisy predictions. The results show a direct propagation of errors:

Invalid Triples: Reduced by an average of 62.92% in model predictions when trained on cleaned data.
Missing Inverses: The most significant improvement was observed here. For ATLOP-RoBERTa, missing inverse relations dropped by 87.44%.
Asymmetric Violations: Nearly eliminated in models trained on cleaned data (e.g., dropping from 0.60% to 0.00% for ATLOP-RoBERTa).

Generalization Performance

Enforcing structural consistency consistently improved generalization performance on the ReDocRED test set:

ATLOP: Showed improvements across all metrics. For instance, using RoBERTa, the F1 score increased by 1.74%, and IgnF1 (ignoring false positives) increased by 4.71%.
DREEAM: Demonstrated substantial gains, particularly in Recall and F1. The BERT-based DREEAM model saw a 13.76% increase in F1 and a 19.32% increase in Recall when trained on the cleaned data.
Comparison: The improvements were consistent across different PLM backbones (BERT, RoBERTa) and model architectures, suggesting the benefits are robust and not specific to a single model design.

Significance and Claims

The paper posits that structural consistency is a missing axis of supervision in DocRE. The authors claim that:

Structural noise is substantially more prevalent in distantly supervised data than in manually curated datasets (e.g., the DocRED distant dataset contains three times more invalid triples and nearly five times more missing inverse relations than ReDocRED manual training data).
Entity annotation errors are the dominant source of invalid triples (69%), indicating that entity extraction is a bottleneck for relation extraction.
Enforcing structural well-formedness acts as an effective structural regularization strategy. It allows models to leverage distant data at scale more effectively by preventing the propagation of logical contradictions during training.
The proposed pre-processing pipeline is a practical, low-cost intervention that improves model performance without requiring changes to model architectures, loss functions, or inference procedures.

The authors conclude that while their intervention is limited to the triple level (not refining entity extraction itself), the observed performance gains serve as conservative estimates of the potential benefits of addressing structural noise in DocRE.