Technical Summary: Interpretable Coreference Resolution Evaluation Using Explicit Semantics

Problem Statement

Coreference resolution (CR) is currently evaluated primarily through aggregate statistical metrics such as CoNLL-F1, MUC, B3, and CEAF$\phi_4$. While these metrics provide stable performance estimates, they suffer from significant limitations:

1. Lack of Diagnostic Insight: They measure structural overlap without revealing whether a system struggles with specific semantic categories (e.g., people vs. events vs. objects).
2. Semantic Blindness: Standard metrics penalize minor span discrepancies or annotation mismatches as full errors, obscuring qualitative differences in model behavior.
3. Limited Interpretability: Single aggregate scores often hide systematic failure modes. For instance, a model might perform well on person-centric narratives but fail on event- or object-related coreference, a degradation invisible to standard scores.
4. Inadequate Semantic Granularity: Prior attempts to incorporate semantics into evaluation have relied on standard Named Entity Recognition (NER) tagsets (typically PER, ORG, LOC, MISC). These are too coarse-grained and exclude common nouns and nominal concepts, which constitute a large fraction of coreference mentions, leaving many clusters untyped.

Methodology

The authors propose a semantically-enhanced evaluation framework that overlays Concept and Named Entity Recognition (CNER) onto coreference outputs. This approach does not alter the coreference model itself but adds a diagnostic layer to interpret its outputs.

1. Labeling and Propagation Technique

The framework employs a two-step process to assign semantic labels to coreference clusters:

• Step 1: Mention Assignment: The system aligns coreference mentions with CNER-annotated spans using a token-level Jaccard similarity overlap function ($\Omega$). If the overlap exceeds a threshold ($\tau=0.5$), the mention is assigned the corresponding CNER label.
• Step 2: Category Propagation: For each coreference cluster, a semantic label is determined via majority voting over the assigned mentions. This dominant label is then propagated to all mentions within the cluster, including pronouns and previously unlabeled mentions.

2. Semantic Evaluation Metrics

Once clusters are semantically typed, the framework computes two stratified F1 scores:

• Typed Mention F1: Evaluates the quality of mention extraction for a specific semantic class, independent of clustering.
• Typed Link F1: Evaluates the quality of predicted coreference links (clustering) for a specific semantic class. To isolate linking performance, this metric is computed using gold-standard mentions as input.

3. Experimental Setup

The framework was evaluated on three distinct English datasets:

• OntoNotes: A multi-genre corpus with balanced entity types.
• LitBank: A literary corpus with restrictive guidelines, heavily skewed toward person (PER) mentions (83.1%).
• PreCo: A dataset based on preschool vocabulary, emphasizing everyday entities.

The authors evaluated three variants of the Maverick coreference model (an encoder-only neural architecture), each trained on one of the three datasets, and tested them both in-domain and out-of-domain.

Key Contributions

1. A Novel Evaluation Framework: The authors introduce a technique that merges CNER tags with coreference outputs, providing dense, granular semantic annotations for the majority of coreference mentions (covering both named entities and nominal concepts).
2. Fine-Grained Analysis: The framework reveals systematic, semantically interpretable error patterns that remain hidden under standard aggregate metrics. It demonstrates that standard NER-based evaluation is insufficient due to low coverage and coarse granularity compared to CNER's 29-category inventory.
3. Actionable Diagnostics for Data Augmentation: The authors demonstrate that these diagnostics can guide targeted, low-cost data augmentation. By identifying underrepresented semantic categories in a model's training data, they generated synthetic documents to specifically address these gaps.

Results

• Coverage and Granularity: The CNER-based approach achieved ~90% mention coverage across all datasets after propagation, significantly outperforming standard NER (which covered only ~22–53%). CNER provided a much richer label space (29 categories vs. 4), enabling the detection of performance variations across diverse semantic types.
• Revealing Systematic Weaknesses: The evaluation exposed that models trained on the person-centric LitBank dataset suffered severe performance drops when evaluated out-of-domain, particularly on categories absent from the training data (e.g., DISEASE, CULTURE, MONEY). Standard metrics masked these specific failures.
• Impact of Targeted Augmentation: The authors trained an augmented model on LitBank data supplemented with three synthetic documents containing underrepresented semantic classes.
  • Using unrestricted annotation (covering all CNER classes) for the synthetic data resulted in a +2.8 CoNLL-F1 point improvement and a ~9.5 point improvement in Mention F1 on out-of-domain datasets compared to the baseline.
  • This confirmed that semantically targeted diagnostics can directly guide data augmentation strategies to improve model generalization.

Significance and Claims

The paper claims that semantic grounding of mentions is essential for moving beyond aggregate statistics to truly interpretable evaluation. By addressing the "interpretability gap," the framework allows researchers to:

• Identify specific semantic categories where models fail.
• Understand whether failures are due to mention extraction or linking logic.
• Design cost-effective, targeted interventions (such as data augmentation) rather than relying on brute-force scaling.

The authors position this work as a step toward more robust coreference evaluation, emphasizing that while the current propagation technique is a scalable baseline, it enables a level of diagnostic depth previously unavailable. They acknowledge limitations, including the reliance on English CNER resources and the fact that ~10% of mentions (mostly pronoun-only clusters) remain unlabeled, but argue that the framework successfully demonstrates the value of explicit semantics in evaluating and improving coreference resolution.