DOREMI: Optimizing Long Tail Predictions in Document-Level Relation Extraction

Technical Summary: DOREMI – Optimizing Long Tail Predictions in Document-Level Relation Extraction

Problem Statement
Document-Level Relation Extraction (DocRE) faces two primary challenges: the necessity of cross-sentence context to identify relations and the severe long-tail distribution of relation types. In standard datasets like DocRED and Re-DocRED, a small number of frequent relations dominate the training data, while the majority of relations (long-tail) have scarce training examples (often fewer than 100 instances). This class imbalance biases models toward frequent relations, degrading their ability to accurately identify rare relations. Furthermore, existing approaches to mitigate noise in Distantly Supervised (DS) datasets, such as UGDRE, often fail to specifically address long-tail relations or rely on large-scale noisy data that exacerbates bias. While Large Language Models (LLMs) show promise, they currently underperform compared to state-of-the-art sequence-based models for this specific task.

Methodology: The DOREMI Framework
The authors propose DOREMI (DOcument-level Relation Extraction optiMizing the long taIl), an iterative, human-in-the-loop framework designed to enhance underrepresented relations through minimal, targeted manual annotations. Unlike previous denoising strategies that rely on heuristic filtering or large-scale DS data, DOREMI actively selects the most informative examples to improve training efficiency and robustness.

The framework operates through the following computational blocks:

1. Core Model Ensemble: DOREMI maintains a pool ($\Gamma$) of $n$ diverse DocRE core models (specifically CNN, LSTM, BiLSTM, ContextAware, and BERT-based models). These models are initially pre-trained on available Human-Annotated (HA) data.
2. Disagreement Computation: The models predict relations over the noisy DS dataset. For each entity pair $(s, o)$, the system computes the disagreement ($\phi_\Gamma$) among the models. The disagreement is calculated based on the probability that models either all predict a relation or all predict "no relation." To handle the multi-label nature of DocRE, the overall disagreement is derived from the product of per-relation disagreements. A logarithmic transformation is applied to amplify small differences and enhance interpretability.
3. Iterative Sampling and Annotation: The system identifies "Hard-To-Classify" examples by selecting the top-$k$ entity pairs with the highest disagreement scores. Crucially, this sampling is restricted to candidate long-tail triples (where at least one model predicts a long-tail relation) to ensure annotation efforts target the specific performance bottleneck.
4. Label Aggregation: Once a stopping condition is met (either the mean disagreement drops below a threshold $\epsilon$ or the annotation budget $b$ is exhausted), the predictions are aggregated to construct a Denoised Distantly Supervised (DDS) dataset. A precision-oriented filter is applied: a relation is retained in the final DDS only if at least one model predicts it with high confidence (above a threshold $\tau$).
5. Iterative Training: The selected samples are manually annotated, added to the training pool, and the core models are fine-tuned on the expanded dataset. This cycle repeats until the stopping condition is reached.

Key Contributions

1. DOREMI Framework: The introduction of a novel iterative system tailored for long-tail relations that enhances distantly supervised datasets through disagreement-driven annotations.
2. Disagreement as a Proxy: The demonstration that measuring disagreement between multiple models is an effective proxy for identifying Hard-To-Classify examples specifically for DocRE, yielding substantial performance improvements with negligible human effort.
3. New Datasets: The release of two new Denoised Distantly Supervised Datasets (DDSs) based on DocRED and Re-DocRED. These datasets are explicitly optimized for long-tail relations and are designed to be model-agnostic, allowing any downstream DocRE model to benefit from improved long-tail coverage.

Experimental Results
The authors evaluated DOREMI using state-of-the-art baselines (ATLOP and DREEAM) on both the DocRED development set and the Re-DocRED test set.

• DocRED: Annotating merely 0.001% of the DS dataset (400 triples) significantly enhanced model performance. Compared to the leading denoising technique (UGDRE), DOREMI increased overall precision by up to +8.2% and F1 by +0.7%. For long-tail triples (<100 examples), precision increased by +76.0% and F1 by +5.0%. Notably, for long-tail triples involving entity pairs not seen in training, DOREMI boosted ignored F1 (ignF1) by up to 28.7% and ignored Precision (ignPrecision) by 137.6% relative to UGDRE.
• Re-DocRED: On the larger Re-DocRED dataset, annotating 0.003% (1,200 triples) resulted in a +16.2% gain in long-tail precision and +19.2% on ignPrecision. For "extreme long-tail" triples (<100 examples), DOREMI achieved a +83.2% increase in precision and a +207.9% increase in ignPrecision compared to UGDRE, with an ignF1 gain of +33.5%.
• Hybrid Approach: A hybrid setting, combining DOREMI for long-tail relations and UGDRE for frequent relations, demonstrated that DOREMI effectively complements existing denoising approaches, yielding further improvements in overall metrics.

Significance and Claims
The paper claims that DOREMI offers a scalable and efficient solution to the long-tail bias problem in DocRE. By prioritizing precision over recall in its design, DOREMI ensures the reliability of extracted relations, which is crucial for downstream Knowledge Base Construction (KBC) tasks. The authors emphasize that their approach achieves significant performance gains with minimal human annotation cost (approximately 20 annotator-hours for the Re-DocRED experiment). The work establishes that disagreement-based active learning is a viable and effective strategy for tailoring datasets to improve generalization on rare relations, marking the first effort to apply such an approach specifically to DocRE. The resulting datasets and methodology provide a foundation for training arbitrary DocRE models with enhanced capabilities in handling long-tail distributions.