🤖 machine learning

Do Lexical and Contextual Coreference Resolution Systems Degrade Differently under Mention Noise? An Empirical Study on Scientific Software Mentions

This paper presents a second-place ranking system for the SOMD 2026 shared task that compares fine-tuning-free lexical and contextual coreference approaches, revealing that while contextual models generally outperform lexical ones, their relative robustness to specific noise types and superior scalability make them the preferred choice for large-scale scientific software mention resolution.

Original authors: Atilla Kaan Alkan, Felix Grezes, Jennifer Lynn Bartlett, Anna Kelbert, Kelly Lockhart, Alberto Accomazzi

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Atilla Kaan Alkan, Felix Grezes, Jennifer Lynn Bartlett, Anna Kelbert, Kelly Lockhart, Alberto Accomazzi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian trying to organize a massive library of scientific papers. Your job is to figure out when different authors are talking about the exact same software tool, even if they describe it slightly differently.

For example, one paper might say "MATLAB," another might say "MATLAB version 9.0," and a third might say "The MathWorks software." Your goal is to group all these mentions together as one single "book" (or entity) in your catalog. This is called Coreference Resolution.

This paper is a report from a team at Harvard who entered a competition (SOMD 2026) to see who could do this job best. They tested two different strategies and discovered some surprising things about how these systems handle mistakes.

Here is the breakdown of their findings using simple analogies:

The Two Competitors

The team built two different "librarians" to do the sorting:

The "Fuzzy Matcher" (FM):
- How it works: This librarian is like a spell-checker. It looks at two names and asks, "Do these words look almost the same?" If "MATLAB" and "MATLAB 9" are 90% similar, it groups them. It doesn't read the whole sentence; it just compares the strings of text.
- The Vibe: Fast, literal, and relies on surface appearance.
The "Context Reader" (CAR):
- How it works: This librarian is like a deep thinker. It reads the sentence and the whole paragraph to understand the meaning. It uses a smart AI to turn the words into a "concept map." It knows that "Python" in a biology paper is different from "Python" in a finance paper, even if the word is spelled the same.
- The Vibe: Smart, contextual, and understands nuance.

The Big Surprise: The "Surface Regularity" Rule

The team found that for software names, the "Fuzzy Matcher" is actually incredibly good. Why? Because software names are very consistent. Unlike people (who can be called "Bob," "Robert," "Bobby," or "The CEO"), software usually keeps its name.

The Analogy: Imagine trying to find all mentions of "iPhone 15." You don't need a genius to tell you that "iPhone 15" and "iPhone 15 Pro" are related; you just need to see the words match. Because software names are so predictable, the simple spell-checker (Fuzzy Matcher) performed almost as well as the super-smart AI (Context Reader).

The Stress Test: What Happens When Things Go Wrong?

Real-world data is messy. Sometimes the computer that finds the software names makes mistakes. The team simulated two types of mistakes to see which librarian would break first:

1. The "Typos and Truncation" Test (Boundary Noise)

The Scenario: The computer accidentally cuts off the end of a name or adds an extra word. "MATLAB" becomes "MATLAB for" or "MATLA".
The Result: The Context Reader (CAR) was much tougher. It could still understand the meaning even if the word was slightly mangled. The Fuzzy Matcher (FM) got confused immediately because the letters didn't match perfectly.
Metaphor: If you whisper a name with a cold, the Context Reader (who knows you) still understands you. The Fuzzy Matcher (who only reads a transcript) thinks you said a different word.

2. The "Wrong Name" Test (Mention Substitution)

The Scenario: The computer finds a software mention but swaps the name entirely. It says "We used Python" when the paper actually meant "We used R."
The Result: Both librarians crashed hard. However, the Fuzzy Matcher actually handled this slightly better in the long run.
Why? The Context Reader was so focused on the context that when the name was swapped, the context got corrupted too, causing a total collapse. The Fuzzy Matcher, being simple, just failed to match and moved on.
Metaphor: If someone puts a fake ID on a person, the Context Reader (who trusts the ID) gets fooled. The Fuzzy Matcher (who just checks the name against a list) realizes the name doesn't match the list and stops trying.

The Speed vs. Size Dilemma

The team also looked at how fast these librarians work as the library gets bigger.

Small Library: The Fuzzy Matcher is a speed demon. It's lightning fast.
Massive Library: As the number of papers grows, the Fuzzy Matcher slows down drastically. It has to compare every single name against every other name (like a game of "Rock, Paper, Scissors" with 10,000 people).
The Winner: The Context Reader scales much better. It processes each paper independently, so even with millions of papers, it doesn't get bogged down in comparisons.

The Final Verdict: Which Librarian Should You Hire?

The paper concludes that there is no single "best" system. It depends on your situation:

If you have a small collection of papers and a very accurate tool to find the names: Use the Fuzzy Matcher. It's fast, cheap, and accurate enough.
If you have a massive library (millions of papers): Use the Context Reader. It will save you time in the long run, even if it takes a bit more computing power.
If your name-finding tool makes a lot of "boundary" mistakes (cutting off words): Use the Context Reader. It's more forgiving of typos.
If your name-finding tool swaps names around: Neither system is perfect. You need to fix the tool that finds the names in the first place.

In a nutshell: Software names are usually so consistent that a simple spell-checker works great. But if you are dealing with a massive amount of data or messy input, you need the smarter, more scalable AI approach. The key to success isn't just picking the smartest algorithm; it's understanding the quality of your data and the size of your library.

1. Problem Definition

The paper addresses the Cross-Document Coreference Resolution (CDCR) task specifically for scientific software mentions. Unlike general coreference resolution which often deals with pronouns and diverse nominal expressions, scientific software mentions are characterized by high surface regularity (e.g., proper names, versioned strings like "MATLAB 8.0", URLs, and acronyms).

The challenge involves partitioning mentions across a collection of scientific papers into clusters where each cluster represents a single underlying software entity. The authors frame this within the SOMD 2026 shared task, which includes three subtasks:

Gold-standard mentions: Evaluating resolution in isolation from detection errors.
Automatically predicted mentions: Simulating real-world conditions with upstream detection noise.
Large-scale predicted mentions: Testing computational efficiency on larger corpora.

The core research questions (RQs) are:

RQ1: Can a simple, unsupervised lexical baseline compete with contextual embedding approaches?
RQ2: How do these systems degrade under different types of mention noise (boundary errors vs. identity substitution)?
RQ3: What is the precision-speed trade-off between the two approaches at different scales?

2. Methodology

The authors propose and compare two fine-tuning-free, unsupervised approaches:

A. Fuzzy Matching (FM)

Mechanism: A lexical string-similarity method.
Algorithm: Uses the Ratcliff/Obershelp algorithm (via Python's difflib) to compute similarity scores based on the longest common substring.
Clustering: Mentions are linked if their similarity score exceeds a threshold ( $\theta$ ). Clusters are formed via transitive closure.
Characteristics: Agnostic to document context; relies entirely on surface form similarity.

B. Context Aware Representations (CAR)

Mechanism: Combines mention-level and document-level embeddings.
Model: Uses all-MiniLM-L6-v2 (a lightweight sentence transformer) to generate dense vector representations.
Representation Construction:
1. Mention-level ( $e_m$ ): Encodes the normalized mention string.
2. Document-level ( $e_d$ ): Encodes an aggregation of up to 10 sentences containing mentions from the same document to capture thematic context.
3. Fusion: The final representation is a weighted sum: $e = \alpha \cdot e_m + (1-\alpha) \cdot e_d$ (with $\alpha=0.6$ ).
Clustering: Uses agglomerative clustering with cosine distance and average linkage.

C. Noise Injection Protocol

To evaluate robustness (RQ2), the authors conducted a controlled study injecting two types of noise into the gold-standard training data:

Boundary Modification: Randomly extending or truncating mention spans (simulating span boundary errors).
Mention Substitution: Replacing a mention string with a different software name from the training set (simulating identity errors).

3. Key Contributions

Competitive Unsupervised Baselines: Two systems (FM and CAR) that achieved 2nd place across all three subtasks of the SOMD 2026 shared task without any task-specific fine-tuning.
Systematic Robustness Analysis: The first study to characterize how coreference systems degrade under specific types of mention noise in the context of scientific software.
Scalability Insights: A detailed analysis of the inference-time trade-off, revealing non-intuitive scaling behaviors between lexical and neural approaches.
Open Source: Release of code to support reproducibility and future research on this underexplored task.

4. Results

Performance on Official Test Set

Both systems achieved high CoNLL F1 scores (0.94–0.96).
CAR consistently outperformed FM by approximately 1 point (0.96 vs. 0.95) on the official test set.
Both systems significantly outperformed other supervised participants (System B), suggesting that the high lexical regularity of software names makes complex supervised models unnecessary for this specific domain.

Robustness to Noise (RQ2)

The systems exhibited complementary failure modes:

Boundary Noise: CAR was significantly more robust.
- CAR lost only 0.07 F1 points from clean to 100% noise.
- FM lost 0.20 F1 points.
- Reason: Dense embeddings are tolerant to minor character/token perturbations, whereas fuzzy matching relies on exact string structure.
Mention Substitution: FM degraded more gracefully.
- FM lost 0.52 F1 points.
- CAR lost 0.63 F1 points.
- Reason: CAR's document context is constructed from mention-bearing sentences. If the mention string is substituted, the document context is simultaneously corrupted, removing the disambiguation signal. FM, being purely lexical, does not suffer from this structural dependency.

Efficiency and Scaling (RQ3)

Small Scale (Subtask 1, ~743 mentions): FM was 7.4x faster than CAR (0.6s vs. 4.45s) with near-identical accuracy.
Large Scale (Subtask 2, ~21,500 mentions): The advantage reversed.
- FM's inference time increased 78x (superlinear scaling due to pairwise comparisons).
- CAR's inference time increased only 10x (approximate linear scaling).
- At large scale, both systems converged to similar inference times (~46s), but CAR maintained its accuracy edge.

5. Significance and Implications

Lexical Regularity is Key: The study confirms that scientific software mentions have high intra-cluster lexical similarity (avg. 0.881). This makes simple lexical matching a surprisingly strong baseline, challenging the assumption that complex neural architectures are always required.
System Selection Strategy: The choice of system should not be based on accuracy alone but on the deployment context:
- Small Corpora / High-Quality Detectors: Use Fuzzy Matching (FM) for speed.
- Large Corpora: Use Context Aware Representations (CAR) for better scalability.
- Noisy Boundaries: Use CAR (more robust to span errors).
- Noisy Identities: Improve the upstream mention detector rather than changing the coreference system, as both methods collapse under severe identity substitution.
Future Directions: The authors suggest that future improvements depend more on the quality of the upstream mention detector than on the coreference resolution algorithm itself. They also propose exploring document-level representations decoupled from mention strings to improve robustness against substitution errors.