💬 NLP

When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguity

This paper demonstrates that Mapper, a topological data analysis tool, reveals how fine-tuned language models restructure embedding spaces into modular, high-purity regions that maintain structural confidence even when human annotators disagree, offering a superior diagnostic framework for understanding model ambiguity compared to traditional visualization methods like PCA or UMAP.

Original authors: Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nisrine Rair, Alban Goupil, Valeriu Vrabie, Emmanuel Chochoy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, invisible room filled with thousands of people. Each person represents a sentence from the internet. In this room, people who say similar things stand close together, while those who say very different things stand far apart. This is how computer models "think" about language: they turn words into points in a massive, multi-dimensional space.

For years, scientists have tried to understand this room by taking a 2D photograph of it (using tools like UMAP or PCA). But the paper argues that these photos are like trying to understand a 3D sculpture by looking at its shadow. You lose the depth, the curves, and the hidden connections.

Here is what this paper discovered using a new tool called Mapper, which acts like a 3D topological map instead of a flat photo.

The Problem: When Humans Can't Agree

The researchers looked at a dataset of tweets about sensitive topics (like politics or race). They asked five different humans to label each tweet as "offensive" or "not offensive."

The "Easy" Tweets (A++): Everyone agreed. (e.g., "I hate you" is clearly offensive).
The "Hard" Tweets (A0): The humans couldn't agree. Some said it was offensive, others said it wasn't. This is called ambiguity.

Usually, when humans disagree, we assume the computer model will be confused, too. But the researchers wanted to see what the model was actually doing inside its brain when it faced these confusing tweets.

The Tool: Mapper (The "Connect-the-Dots" Map)

Instead of squashing the 3D room into a 2D photo, the authors used Mapper.

The Analogy: Imagine you are trying to map a complex cave system.
- Old Tools (UMAP/PCA): You take a photo from the entrance. You see a big blob of rock. You can't tell if there are separate tunnels or just one big cave.
- Mapper: You shine a light from different angles, slice the cave into overlapping layers, and then connect the dots where the slices overlap. You end up with a graph (a network of nodes and lines) that shows you exactly how the tunnels connect, where they branch off, and where they dead-end.

The Big Discovery: The Model "Forces" Order

The researchers found something surprising about how the model handles the "hard" tweets where humans disagreed.

Before Training (The Messy Room): When the model was first given the data, the "offensive" and "non-offensive" people were scattered everywhere. The map looked like a messy cloud.
After Training (The Organized Room): Once the model learned the task, it didn't just make two neat piles (one for offensive, one for not). Instead, it built modular neighborhoods.
- Even for the confusing tweets where humans disagreed, the model grouped them into specific, distinct regions.
- The Shock: Inside these regions, the model was extremely confident. It would look at a group of confusing tweets and say, "This whole neighborhood is 'Offensive'!" with 98% certainty.

The Metaphor:
Imagine a group of tourists arguing about whether a painting is "art" or "garbage."

The Humans: They are split 50/50.
The Model: It walks in, looks at the whole group, and confidently declares, "This whole group is Art!" It doesn't hesitate. It has created a "zone" for this group and assigned it a single label, ignoring the fact that the humans inside the zone are still arguing.

What This Means

The paper claims that the model doesn't just "guess" when it's confused. Instead, it reorganizes the space to make its own kind of sense.

It creates smooth, consistent regions where it feels safe to make a decision.
It ignores the "noise" of human disagreement and imposes its own structure.
This explains why models can be overconfident: they aren't looking at the individual confusion of the humans; they are looking at the "neighborhood" the data has been sorted into, and that neighborhood has a clear label.

Why Mapper is Better

The authors show that if you use the old "photo" tools (UMAP), you might think the offensive tweets are all in one big, neat circle. But Mapper reveals that they are actually in many different, disconnected islands that are only connected by thin bridges.

Old View: "Everything is separated nicely."
Mapper View: "It's a complex web of islands. The model is confident on each island, but the islands are scattered in a way that a simple photo hides."

Summary

The paper argues that when humans disagree on a label, the computer model doesn't get confused and wobble. Instead, it builds a new map where it groups those confusing examples into specific zones and confidently assigns them a single label. This "topological" view (looking at the shape and connections) reveals that the model is actually very structured, even when the data is messy. It's not that the model is failing to understand ambiguity; it's that it has found a way to resolve it by creating its own consistent, if sometimes overconfident, reality.

1. Problem Statement

Natural Language Processing (NLP) models are typically evaluated using scalar metrics (e.g., accuracy, F1-score) which fail to capture how models internally represent ambiguity, particularly in scenarios where human annotators disagree.

The Gap: While prior work has modeled ambiguity via soft labels or confidence estimates at the output level, there is limited understanding of how ambiguous instances are structurally encoded within the high-dimensional embedding space of fine-tuned models.
Limitations of Current Tools: Standard geometric visualization tools like PCA and UMAP often oversimplify high-dimensional structures. They rely on Euclidean distance assumptions, which can flatten complex, non-convex, and hierarchical relationships (common in hyperbolic embedding spaces), leading to misinterpretations of decision boundaries and neighborhood structures.

2. Methodology

The authors propose using Mapper, a tool from Topological Data Analysis (TDA), to analyze the geometry of text embeddings. Unlike dimensionality reduction techniques that project data into a lower dimension, Mapper constructs a graph-based summary that preserves connectivity and local structure.

Dataset and Models

Dataset: MD-Offense, a dataset of 10,753 English tweets annotated by five human annotators.
- Ambiguity Proxy: Instances are categorized by agreement levels: A++ (5/5 agreement, unambiguous), A+ (4/5), and A0 (3/5, high ambiguity/disagreement).
Model: RoBERTa-Large (selected for its strong performance across ambiguity levels). Both pretrained and fine-tuned versions were analyzed.
Embeddings: Extracted from the final [CLS] token of the last hidden layer.

The Mapper Algorithm Implementation

The authors implemented Mapper using the KeplerMapper library with the following configuration:

Filtering (Lens Function): High-dimensional embeddings ( $d=1024$ $d = 1024$ ) were projected into 1D intervals ( $k=1$ $k = 1$ ) using six distinct lens functions to capture different geometric properties:
- Centroid Projection: Alignment with the decision boundary.
- PCA: Global variance.
- Eccentricity: Distance from training data (outlier detection).
- L2 Norm: Activation magnitude.
- Random Projections: Baselines.
Covering: The projected space was covered by overlapping hypercubes (intervals) with a resolution of $r=40$ and an overlap of $\epsilon=0.3$ .
Clustering: Within each interval, data points were clustered using HDBSCAN (Hierarchical Density-Based Spatial Clustering) to avoid imposing a fixed number of clusters and to handle varying densities.
Graph Construction: Nodes represent clusters; edges connect nodes if they share data points (due to the overlapping cover).

Topological Metrics

To quantify the results, the authors defined three metrics:

Component Purity (CP): The average purity of nodes within a connected component (measuring class homogeneity).
Edge Agreement (EA): The proportion of edges connecting nodes with the same majority class label (measuring global consistency).
Majority Match (MM): The alignment between the model's predicted majority label and the ground-truth majority label at the component level.

3. Key Contributions

Topological Perspective on Ambiguity: The paper introduces Mapper as a diagnostic tool to visualize how fine-tuned models restructure embedding spaces, revealing that models do not simply collapse classes into linear clusters but organize them into modular, non-convex regions.
Discovery of "Structural Confidence": The study reveals a hidden tension: even in highly ambiguous data (A0) where human annotators disagree, the model organizes these instances into highly coherent, prediction-pure topological regions. The model exhibits "overconfidence" by assigning a single label to entire topological components, regardless of the underlying label noise.
Superiority over Geometric Tools: The authors demonstrate that Mapper uncovers structural details (fragmented regions, boundary collapses, and interwoven classes) that UMAP and PCA flatten or obscure.

4. Key Results

Fine-Tuning Restructures Topology: Fine-tuning transforms the embedding space from a fragmented state (in the base model) into a structured, modular topology.
- Purity Increase: Post-fine-tuning, >98% of connected components exhibit $\ge90\%$ prediction purity, even in the ambiguous A0 subset.
- Edge Consistency: Edge Agreement (EA) increases significantly after fine-tuning, indicating smoother transitions between neighboring clusters.
The "Overconfidence" Phenomenon:
- In the A0 subset (high human disagreement), the ground-truth labels within components are noisy (CP ~31.5%).
- However, the model's predictions within those same components are almost perfectly pure (CP ~98.0%).
- This indicates the model resolves ambiguity by imposing a coarse topological structure, grouping ambiguous instances into regions that align with its binary training objective, effectively "ignoring" the nuance of human disagreement.
Robustness: These findings hold across different lens functions (1D and 2D) and different model architectures (BERT, DeBERTa, DistilBERT, etc.), suggesting this is a fundamental behavior of fine-tuned transformers.

5. Significance and Implications

Diagnostic Tool: Mapper provides a powerful method for understanding how models resolve ambiguity, moving beyond scalar metrics to reveal the internal geometry of decision regions.
Challenging Assumptions: The results challenge the assumption that models fail on ambiguous data by guessing randomly. Instead, models exhibit systematic, region-level behavior, generalizing by carving out complex substructures that may not align with human consensus.
Future Directions: The authors suggest that these topological insights can inform proactive modeling strategies, such as:
- Selecting subsets for training based on topological purity.
- Designing fine-tuning regimes that account for topological uncertainty.
- Developing robustness evaluations specifically for ambiguous regions.
Reproducibility: All code and configurations are open-sourced, facilitating further research into topological NLP analysis.

In conclusion, the paper argues that topology explains the geometry of ambiguity: fine-tuned models do not merely learn to classify; they reorganize the embedding space into modular, high-confidence regions that often override the uncertainty present in human annotations.