LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task
The third edition of the LeWiDi shared task at NLPerspectives advances the development of disagreement-aware AI by expanding the benchmark to four diverse NLP tasks with ordinal labeling schemes, introducing both soft-label and perspectivist evaluation paradigms with novel metrics, and providing new resources and insights into modeling human judgment variation.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot how to understand human language. For a long time, we taught robots by showing them examples where everyone agreed on the answer, like "2 + 2 = 4." But in real life, humans often disagree. One person might think a joke is hilarious, while another thinks it's offensive. One person might see a sentence as a lie, while another sees it as a harmless opinion.
This paper describes the third edition of a "contest" (called LeWiDi-2025) designed to teach AI models how to handle these disagreements, rather than pretending they don't exist.
Here is a simple breakdown of what they did, using some everyday analogies:
1. The Goal: Teaching AI to Listen to the Crowd
Instead of forcing the AI to pick just one "correct" answer, the organizers wanted to see if AI could understand the whole picture of human opinion.
- The Old Way: If 10 people vote on a movie, and 6 say "Good" and 4 say "Bad," the old AI would just be told, "The answer is Good." It ignores the 4 people.
- The New Way: The AI is told, "Here is the full story: 6 people liked it, 4 didn't. Your job is to understand why they felt that way and predict the mix of opinions."
2. The Four "Games" (Datasets)
The contest gave participants four different types of puzzles to solve, each representing a different kind of human disagreement:
- The Sarcasm Game (CSC): Imagine reading a text message. Is it a sincere compliment or a sarcastic jab? Different people rate how "sarcastic" it is on a scale from 1 to 6.
- The Irony Game (MP): A short post and a reply. Is the reply being ironic? This was played in 9 different languages (like English, Spanish, and Arabic), showing that irony is a global problem.
- The Paraphrase Game (Par): Two questions are asked. Are they asking the same thing? Instead of a simple "Yes/No," people rated how similar they were on a scale from -5 to 5.
- The Logic Game (VEN): A statement and a fact. Does the fact prove the statement, contradict it, or have nothing to do with it? Here, people also had to write explanations for their answers.
3. The Two Ways to Play (Tasks)
Participants had to choose between two different ways to show their AI's understanding:
- Task A: The "Weather Forecaster" (Soft-Label)
Instead of guessing one specific answer, the AI acts like a weather forecaster. It doesn't just say "It will rain"; it says, "There is a 60% chance of rain, 30% chance of sun, and 10% chance of snow." The AI predicts the distribution of how humans would vote. - Task B: The "Mind Reader" (Perspectivist)
This is harder. The AI has to guess what a specific person would say. If you tell the AI, "Here is what Person X usually thinks," the AI must predict, "Based on Person X's history, they will likely give this answer." It's like predicting whether your grumpy uncle or your optimistic aunt would agree with a joke.
4. The New Scorecard (Metrics)
In the past, the judges used a simple ruler to measure how far off the AI was. But that ruler didn't work well for things like "sarcasm scales" or "multiple languages."
- The New Ruler: They invented new ways to measure success.
- For the "scales" (like 1 to 6), they used a metric that understands that being off by one point (saying 4 instead of 5) is better than being off by five points (saying 1 instead of 5).
- For the "mind reader" task, they measured how well the AI could mimic the specific biases and habits of individual people.
5. The Results: Who Won?
About 15 teams entered the contest. Here is what they found:
- The "Big Brain" Approach: The top teams used Large Language Models (LLMs)—the same kind of super-smart AI that writes essays and chats with you. These models were very good at looking at examples of how different people voted and copying that pattern.
- The "Small but Mighty" Approach: Some teams used smaller, specialized models. These actually did surprisingly well on the smaller, trickier datasets.
- The Secret Ingredient: The winning systems didn't just look at the text; they looked at who was writing the text. They used clues like the annotator's age, gender, or past voting history to make better guesses.
- The "Explanation" Bonus: On the logic game, teams that fed the AI the explanations people wrote (not just the answers) performed much better. It's like understanding why someone voted, not just how they voted.
6. The Catch (Limitations)
The authors are honest about what they didn't test:
- The "New Face" Problem: In the real world, you might meet a stranger whose opinion you've never seen before. In this contest, the AI was tested on people it had already "met" during training. We don't know yet if these models can handle a completely new person.
- Text Only: The contest was only about reading and writing. We don't know if these methods work for images, videos, or voice.
Summary
This paper is a report card for a contest that taught AI to stop pretending everyone agrees. By using new scoring methods and focusing on the messy reality of human disagreement, the winners showed that the best AI models are the ones that can look at a crowd, understand the different voices within it, and predict the mix of opinions rather than just picking a single winner.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.