💬 NLP

The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

This paper introduces the "Ghost Annotator" framework, which combines conformal prediction with collaborative filtering to analyze human label variation and reveal that larger language models exhibit increased confidence in predictions diverging from human consensus, thereby exposing structural demographic biases rooted in pretraining data.

Original authors: Mirko Lai, Alessandra Urbinati, Simona Frenda, Fabiana Vernero, Marco Antonio Stranisci

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mirko Lai, Alessandra Urbinati, Simona Frenda, Fabiana Vernero, Marco Antonio Stranisci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to decide what is "rude" or "harmful" in a conversation. You ask a group of 100 different humans to read a specific comment and rate it. Because people have different backgrounds, ages, and cultures, they don't all agree. Some think the comment is funny; others think it's hate speech. This disagreement is called Human Label Variation.

The paper introduces a new tool called the "Ghost Annotator" to figure out how well AI models (like the ones powering chatbots) understand these human disagreements and where they go wrong.

Here is how the paper works, explained through simple analogies:

1. The Problem: The Robot vs. The Crowd

Usually, when we test an AI, we ask: "Did the robot get the 'right' answer?" But in content moderation, there often is no single right answer. If 50 people say a comment is "mildly offensive" and 50 say it's "harmful," the AI is stuck.

The researchers wanted to know:

Does the AI get confused when humans disagree?
Does the AI have its own "personality" that makes it agree with some types of people and ignore others?

2. The Tool: The "Ghost" in the Machine

To solve this, the team created a framework using a statistical method called Conformal Prediction. Think of this as a "confidence meter."

The Ghost Prediction: Imagine the AI is taking a test. Usually, it picks an answer that matches what the humans picked. But sometimes, the AI picks an answer that no human picked. The researchers call this a "Ghost Prediction." It's like the AI is seeing a ghost—a label that doesn't exist in the human world.
The Ghost Annotator: This is the paper's main invention. Instead of just looking at the AI's answers, they turn the AI's behavior into a "fingerprint" (a mathematical vector). They call this fingerprint the Ghost Annotator. It represents the AI's unique "opinion" as if it were a human worker, even though it's a machine.

3. The Experiment: Comparing Fingerprints

The researchers took four different AI models (two small, two large, from two different companies) and tested them on four different datasets of social media posts (about hate speech, violence, and offensiveness).

They compared the AI's "Ghost Fingerprint" against the fingerprints of real human annotators, grouping the humans by demographics like age, gender, and where they are from (e.g., Sub-Saharan Africa, India, the US).

4. The Findings: What the Ghosts Revealed

Finding A: The "Confident Outsider"

The Analogy: Imagine a small, nervous student who admits, "I'm not sure about this answer," when the class is arguing. Now imagine a very confident, older student who says, "I know the answer is X," even when the whole class disagrees.
The Result: The researchers found that larger AI models (the confident students) were actually more confident when they gave answers that no human agreed with (Ghost Predictions). They were sure of their "ghost" answers, even when they were completely out of sync with human opinion. Smaller models were less confident in these weird situations.

Finding B: The "Demographic Blind Spot"

The Analogy: Imagine a group of judges. If you ask a judge from Group A to rate a movie, they might agree with the AI. But if you ask a judge from Group B, they might completely disagree. The paper found that the AI acts like a judge who is consistently "out of touch" with one specific group of people, no matter how many people from that group are in the room.
The Result: The AI models showed a consistent pattern of demographic misalignment. Specifically, the AI's "Ghost Fingerprint" was consistently the least similar to human annotators from Sub-Saharan Africa (SSA).
The Twist: This happened even though the SSA group was actually the largest group of human annotators in the study. The AI didn't ignore them because there were few of them; it ignored them because of a deep-seated bias.
The Cause: The researchers believe this bias comes from the pre-training data (the massive amount of text the AI read before being taught the specific task). It's like the AI learned its "worldview" from a library that didn't have enough books written by people from Sub-Saharan Africa. This bias is "structural," meaning it's baked into the foundation of the model, not just a mistake in the specific test.

5. Why This Matters (According to the Paper)

The paper argues that we can't just fix this by adding more diverse human labels to the training data. If the AI's "foundation" (pre-training) is already biased, adding a few more diverse voices at the end won't fix the "Ghost Annotator's" perspective.

The Ghost Annotator framework is a way to "X-ray" an AI model to see exactly which groups of people it is failing to understand, before we let it loose on the internet to moderate content.

In short: The paper built a tool to see the "personality" of AI models. They found that big, confident AI models often have a "blind spot" for specific cultures, and this blind spot is so deep that it comes from the very data the AI learned to read in the first place.

Technical Summary: The Ghost Annotator Framework

Problem Statement
Current research in Natural Language Processing (NLP) and content moderation predominantly focuses on model performance metrics, often overlooking uncertainty estimation and the nuances of Human Label Variation (HLV). HLV refers to the divergence in annotations among human raters, particularly regarding subjective phenomena like hate speech, offensiveness, and toxicity. While recent paradigms advocate for capturing these diverse perspectives to enhance fairness, significant challenges remain: findings on HLV are often hindered by mismatches between datasets and annotation schemes, and the increasing reliance on Large Language Models (LLMs) to generate annotated data has shifted focus away from uncertainty. Furthermore, existing methods for evaluating LLMs-as-annotators often fail to distinguish between genuine HLV and annotation errors, and LLMs have been shown to struggle in replicating the diversity of human opinions, often exhibiting biases toward specific demographic groups or over-flagging offensive content.

Methodology
The authors introduce the Ghost Annotator, a framework designed to model LLM behavior relative to human annotators using Conformal Prediction. The methodology proceeds through the following technical steps:

Conformal Prediction & Non-Conformity Scores (NCS): The framework utilizes Conformal Prediction to generate calibrated uncertainty estimates. Instead of a single aggregated score, the system computes NCS for individual model–annotator–instance interactions. The NCS quantifies how "unusual" a model's prediction is relative to a calibration distribution derived from human annotations.
Ghost Prediction Metric: To move beyond traditional accuracy-based metrics (which rely on aggregated ground truth), the authors define Ghost Prediction as the proportion of instances where a model predicts a label that is not present in any of the human annotations for that specific item. This metric quantifies the divergence between the model and the entire set of available human perspectives.
Annotator Representation (Collaborative Filtering Analogy): Inspired by Collaborative Filtering (CF) in recommender systems, the framework represents both human annotators and models as 3-dimensional vectors. These vectors are derived from the quartiles of the NCS distributions:
- Human Annotators: Represented by the distribution of NCS values obtained from all items they annotated.
- Ghost Annotator (Model): Represented by the distribution of NCS values computed specifically on instances where the model's prediction differs from all human annotations (Ghost Predictions).
Alignment Analysis: The authors compute cosine similarity between the Ghost Annotator vector and the vectors of specific socio-demographic groups of human annotators. This allows for the profiling of which human groups a model aligns with or diverges from.

Experimental Setup
The framework was evaluated across four content moderation datasets: Attitudes (Hate Speech), CADE (Acceptability), Disentangling (Offensiveness), and MHS (Violence). Four LLMs were tested, spanning two families (Qwen and Llama) and two size scales (1.5B/1B and 7B/8B parameters). The Disentangling dataset was specifically used for the demographic alignment analysis (RQ2) due to its balanced representation of socio-demographic traits (gender, age, and macro-region of origin).

Key Results

Uncertainty and HLV (RQ1): There is a coherence between model uncertainty and HLV. As disagreement among human annotators increases, model uncertainty (measured by NCS) correspondingly increases. However, a counter-intuitive finding emerged regarding Ghost Predictions: larger models exhibited higher confidence (lower uncertainty) when predicting labels that no human annotator selected, compared to smaller models. This suggests that larger models may encode stronger majority-culture norms, leading to confident but divergent predictions in regions where minority perspectives differ.
Demographic Misalignment (RQ2): The Ghost Annotator framework revealed a robust and consistent pattern of demographic misalignment. Regardless of model architecture (Meta vs. Alibaba), parameter scale (1B to 8B), or the dataset used to profile the model, the models systematically diverged from annotators in the 18-30, Male, Sub-Saharan Africa (SSA) demographic group. Conversely, they showed the highest alignment with the 30-50, Female, Oceania group.
Annotator Pool Size: The misalignment with the SSA group persisted despite this group being the largest demographic pool in the Disentangling dataset ( $n=169$ ). This indicates that simply increasing the headcount of diverse annotators does not resolve the misalignment, suggesting the bias is not a result of data scarcity in the annotation phase.

Significance and Claims
The paper claims that the Ghost Annotator framework provides a statistically guaranteed method (via Conformal Prediction) to evaluate and position LLMs within the space of human annotator diversity. Its primary significance lies in identifying that the observed demographic misalignment is likely a structural bias rooted in pre-training corpora rather than a specific artifact of fine-tuning or model architecture.

The authors argue that because different model families and sizes converge on the same misalignment patterns, the bias exists upstream of task-specific adaptation. Consequently, the paper suggests that dataset-level interventions (such as increasing annotator diversity) are insufficient to resolve these biases without corresponding changes to how models are pre-trained or adapted. The framework serves as a tool to uncover these structural divergences before models are deployed, offering a path toward more conscientious use of LLMs in creating and augmenting training datasets for content moderation.