💻 computer science

Confidence Calibration and Semantic Error Analysis for Ambiguous Coreference Resolution in User-Generated Social Media Text

This paper introduces AmbiGraph-Coref, a graph-based model enhanced with confidence calibration and an explicit unresolvable mechanism that achieves high performance in resolving ambiguous coreference in synthetic social media text, while emphasizing the need for future validation on authentic real-world data.

Original authors: Yuewei Yuan, Jialei Huang, Jiayang Yin, Tianyi Xu

Published 2026-09-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuewei Yuan, Jialei Huang, Jiayang Yin, Tianyi Xu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the bustling, chaotic world of online social media, language often behaves differently than it does in a book or a news article. People write in fragments, drop subjects, and rely on shared context that exists only in the immediate thread of conversation. This creates a specific puzzle for computers trying to understand human speech: figuring out exactly who or what a pronoun refers to when the sentence is incomplete or the situation is unclear. This task is known as coreference resolution. While modern computers have become quite good at this in formal writing, they often stumble when faced with the elliptical, messy, and ambiguous nature of user-generated posts. The real danger lies not just in making a mistake, but in making a mistake with absolute certainty. When a system confidently links a pronoun to the wrong person, that error can ripple through other automated tasks, such as judging the mood of a comment or flagging a post as risky, leading to flawed decisions based on a false foundation.

A team of researchers set out to solve this problem of "confident confusion" by building a new system designed specifically for the ambiguity of social media. They began by creating a massive, controlled library of text fragments that mimicked the five most common types of social media interactions: short video titles, comment replies, community messages, creator profiles, and anonymous feedback. In total, they constructed 86,400 distinct text samples containing over 214,000 mentions of people or things and nearly 67,000 instances where one phrase referred back to another. Crucially, they did not just ask the computer to pick the best answer; they taught it to recognize when there was no good answer at all. The system was trained to distinguish between three states: a clear connection, a situation where multiple options are plausible, and a scenario where the evidence is simply missing.

The core of their solution, which they named AmbiGraph-Coref, treats the text not as a simple list of words but as a complex web of relationships. Imagine the text as a map where every person, object, and comment is a point, and the connections between them are roads. The system builds a "heterogeneous relational graph," a structure that links the specific phrase being analyzed to the possible people it could refer to, while also accounting for the depth of the conversation and the history of the thread. By using a powerful language model to understand the meaning of each point and a graph-based method to travel along the connections, the system can weigh how likely each candidate is to be the correct one. Instead of forcing a choice, it ranks the possibilities and calculates a score for how much evidence supports the top choice versus the runner-up.

The most significant breakthrough in this work is the system's ability to calibrate its own confidence. In many previous attempts, a computer might assign a 90% probability to an answer even when the evidence was weak, simply because the math of the model pushed it that way. The new system uses a technique called temperature scaling to smooth out these probabilities, ensuring that a high score truly reflects a high degree of certainty. When the system detects that the gap between the best candidate and the second-best candidate is too small to be sure, or that the necessary context is missing, it does not guess. Instead, it activates a specific "unresolvable" branch, effectively saying, "I cannot determine the answer with the information provided." This mechanism prevents the system from making high-confidence errors, which are the most dangerous kind.

When tested on their synthetic dataset, the new system outperformed existing methods, achieving a success rate of 85.6% in correctly identifying references. More importantly, the calibration process drastically reduced the frequency of high-confidence errors. Before these adjustments, the system made confident mistakes nearly 19% of the time; after calibration, that number dropped to just under 7%. The researchers also tested how this improved understanding affected other tasks. When the system's output was used to help a computer determine the sentiment of a post or identify potential risks, the accuracy improved significantly. The rate of incorrect links between entities fell from over 21% to under 10%, proving that knowing when not to decide is just as valuable as knowing how to decide.

However, the researchers are careful to note the limits of their findings. The data used to train and test the system was generated using controlled templates and manually validated, meaning it represents a structured simulation of social media rather than the raw, unfiltered flow of a real platform. While the results are strong within this constructed environment, the authors acknowledge that the true test will come from applying these methods to authentic, messy interactions across different platforms and topics. They suggest that future work will need to verify if the system can handle sarcasm, longer conversations, and the unpredictable nature of real-world user behavior without the safety net of a controlled dataset. For now, the study offers a compelling blueprint for building machines that are not only smarter at reading text but also more humble about what they do not know.

Technical Summary: Confidence Calibration and Semantic Error Analysis for Ambiguous Coreference Resolution in User-Generated Social Media Text

1. Problem Statement

User-generated social media text is characterized by brevity, ellipsis, and structural irregularity, which creates significant challenges for ambiguous coreference resolution. In these contexts, referential expressions often lack clear antecedents due to omitted subjects, cross-level shifts in comment threads, or competing entities.

Existing neural and generative models have advanced antecedent prediction but generally focus on determining whether a selected antecedent is correct, rather than assessing the reliability of the prediction's confidence. Conventional softmax-based systems often return high-confidence Top-1 choices even when inputs are incomplete or corrupted, failing to distinguish between:

Competition among plausible candidates (where multiple antecedents are possible).
Absence of adequate evidence (where no antecedent is supported).

This lack of distinction leads to forced entity bindings that propagate semantic errors to downstream tasks such as sentiment analysis and risk identification. The paper posits that explicit modeling of candidate-level confidence and the ability to abstain (selective prediction) are critical for reliable resolution in social media environments.

2. Methodology

2.1 Data Construction

To address the scarcity of labeled data for ambiguous social media coreference, the authors constructed a primarily synthetic dataset of 86,400 text fragments containing 214,700 entity mentions and 67,300 referential expressions.

Source: Controlled templates covering five scenarios: short-video titles, comment replies, community messages, creator profiles, and anonymous feedback.
Labeling: Samples are assigned one of three mutually exclusive states:
- CLEAR: Decisive evidence exists.
- AMBIGUOUS: Plausible candidates exist but lack decisive evidence.
- UNRESOLVABLE: No supported antecedents exist.
Validation: 15% of the data (12,960 samples) underwent manual validation by three annotators, achieving a Fleiss' $\kappa$ of 0.86.

2.2 Model Architecture: AmbiGraph-Coref

The proposed model, AmbiGraph-Coref, integrates heterogeneous relational graphs with confidence calibration and an explicit abstention mechanism.

Heterogeneous Relational Graph:
- Constructs a graph $G = (V, E, R)$ where nodes include referential expressions ( $V_m$ ), candidate entities ( $V_e$ ), comments ( $V_c$ ), replies ( $V_r$ ), and structural fragments ( $V_s$ ).
- Edges encode candidate, context, reply, and co-occurrence relations.
- Uses DeBERTa-v3-base for 256-token node representations.
- Employs a graph propagation mechanism (Eq. 4) that updates node representations using relation-specific attention weights, which combine semantic relevance with distance decay (Eq. 5).
Candidate Ranking & Ambiguity Detection:
- Instead of single-label classification, the model ranks candidate antecedents using a scoring function $s(e_i, m)$ that combines semantic matching, graph-relational relevance, and contextual evidence strength.
- Ambiguity Trigger: Computed as the calibrated probability gap between the Top-1 and Top-2 candidates.
Multi-Task Learning Framework:
- A shared encoder feeds into four heads: ambiguity classification, candidate ranking, UNRESOLVABLE detection, and confidence calibration.
- The joint loss function (Eq. 7) optimizes these tasks simultaneously.
Confidence Calibration & Abstention:
- Temperature Scaling: Applied to candidate logits to prevent overconfidence when ranking gaps are small.
- UNRESOLVABLE Mechanism: A hybrid decision rule (Eq. 10) fuses the learned rejection head with evidence-based thresholds. If the maximum calibrated probability is below a threshold, or if the probability gap and contextual evidence are insufficient, the sample is flagged as UNRESOLVABLE.

3. Key Results

3.1 Coreference Resolution Performance

On the test set, AmbiGraph-Coref outperformed baselines including End-to-End Coref, BERT-Classifier, Span-Ranking, Graph-Coref (without calibration), and LLMs (Zero/Few-shot).

F1 Score: 0.856
Accuracy: 0.842
Mean Reciprocal Rank (MRR): 0.887

3.2 Confidence Calibration Efficacy

The study demonstrates that joint calibration significantly improves the reliability of predictions:

Expected Calibration Error (ECE): Reduced from 0.184 (uncalibrated) to 0.057.
High-Confidence Error Rate: Reduced from 18.7% to 6.9%.
Behavior: Calibration shifts incorrect predictions toward lower confidence while retaining high confidence for correct samples, particularly in cases of small candidate margins or weak contextual evidence.

3.3 Downstream Task Validation

When calibrated probabilities and ambiguity labels were fed into downstream sentiment and risk identification tasks:

Sentiment Classification (Macro-F1): Improved from 0.713 (raw text) to 0.781.
Risk Identification (AUC): Improved from 0.764 to 0.823.
Error Reduction: Incorrect entity binding rates dropped from 21.6% to 9.7%, and high-risk false positives decreased from 16.8% to 9.8%.

4. Significance and Claims

The paper claims to support reliability-aware coreference resolution for ambiguous social media text by explicitly modeling uncertainty and the "UNRESOLVABLE" state. The primary contributions are:

Methodological Integration: Combining heterogeneous graph propagation with candidate ranking and explicit confidence calibration to handle structural irregularities in social media.
Error Mitigation: Demonstrating that calibrated uncertainty reduces high-confidence semantic errors, thereby protecting downstream tasks from propagating incorrect entity bindings.
Dataset Construction: Providing a large-scale, multi-scenario synthetic dataset with validated labels for ambiguous coreference.

5. Limitations and Future Directions

The authors maintain a modest stance regarding the generalizability of their findings:

Synthetic Data: The results are based on a primarily synthetic dataset. The authors explicitly state that this cannot establish effectiveness on authentic real-world social media platform data.
Scope: The current evaluation covers specific interaction conditions and evidence levels.
Future Work: The paper calls for validation on anonymized, manually annotated real-world interactions, evaluation of cross-topic and cross-platform transfer, and addressing challenges like sarcasm and longer conversation histories.