💬 NLP

Multiperspectivity as a Resource for Narrative Similarity Prediction

This paper proposes leveraging an ensemble of 31 diverse LLM personas to embrace narrative multiperspectivity rather than treating it as a challenge, demonstrating that such an approach improves similarity prediction accuracy on the SemEval-2026 Task 4 dataset while revealing limitations in current single-ground-truth evaluation frameworks.

Original authors: Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj, Vera Schmitt

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Max Upravitelev, Veronika Solopova, Jing Yang, Charlott Jakob, Premtim Sahitaj, Ariana Sahitaj, Vera Schmitt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to decide which of two stories is more similar to a third "original" story. In the world of computers, this is usually treated like a math problem: there is one right answer, and the computer just needs to find it.

But this paper argues that storytelling isn't math; it's art. And in art, there is no single "right" answer. A literary critic might see a story about a woman as a tale of female empowerment, while a historian might see it as a critique of 19th-century society. Both are valid, but they lead to different conclusions about what the story "means."

The authors of this paper asked: What if we stopped trying to force computers to find the "one true answer" and instead let them argue like a panel of diverse experts?

Here is the breakdown of their experiment, explained with some everyday analogies.

1. The Setup: The "Super-Panel" of 31 Characters

Instead of asking one AI model to make the decision, the researchers created a team of 31 different AI "personas."

Think of this like hiring a jury for a trial, but instead of regular people, you hire:

The Experts (Practitioners): A Feminist Literary Critic, a Postcolonial Theorist, a Data Scientist, a Psychologist. These are the "serious" thinkers with specific rulebooks for analyzing text.
The Regulars (Lay People): A High School Student, a Football Player, a Pirate, a Taxi Driver. These are the "everyday" folks who just use their gut feeling.

They asked each of these 31 characters to vote on which story was more similar. Then, they took a majority vote (like a democratic election) to get the final answer.

2. The Surprise: The "Clueless" Crowd vs. The "Over-Thinkers"

The researchers expected the "Experts" to win. After all, they have fancy degrees and specific frameworks for analyzing stories, right?

Wrong.

The Individual Result: When you looked at each character alone, the "Regulars" (like the Football Player or the Tour Guide) were actually better at guessing the right answer than the "Experts." The experts often got bogged down in their own complex theories and missed the obvious similarities.
The Team Result: However, when you combined them all into a group vote, the Experts saved the day. Even though they were individually worse, their errors were different from each other. The Football Player might get confused by one thing, while the Feminist Critic gets confused by something else. When you mix them all together, their mistakes cancel each other out, and the group becomes incredibly smart.

The Analogy: Imagine trying to guess the weight of a giant pumpkin.

One expert might overthink the soil density and get it wrong.
Another might focus on the stem and get it wrong.
But if you take 30 people with different, slightly wrong guesses and average them, you often get a result that is shockingly close to the real weight. This is called the "Wisdom of Crowds."

3. The Twist: The "Gender" Trap

Here is where the paper gets really interesting. They noticed a strange pattern: whenever the AI personas started using words related to gender, feminism, or patriarchy (like "female protagonist," "gender roles," or "objectification"), their accuracy dropped.

Why?

Theory A: The benchmark (the test they were taking) was designed by people who didn't care about gender. So, when the AI started focusing on gender, it was looking at the "wrong" clues for this specific test.
Theory B: The AI was actually finding valid, deep meanings in the text that the test creators completely missed. The test said "Wrong," but the AI said, "Actually, this is a very important part of the story."

This suggests that our current tests for AI might be unfair. They might punish an AI for being too smart or too culturally aware, forcing it to ignore valid interpretations just to get a high score.

4. The Conclusion: Diversity is the Secret Sauce

The paper concludes that to make AI understand stories better, we shouldn't just try to make it "smarter." We should make it more diverse.

Don't just have one brain: Have a room full of different brains.
Don't fear the "wrong" perspective: Even if a specific perspective (like a feminist critique) gets the "test score" wrong, it adds valuable diversity to the group, which helps the whole team make better decisions in the long run.

In a nutshell:
If you want a computer to understand a story, don't ask it to be a robot. Ask it to be a room full of people: a poet, a plumber, a historian, and a teenager. Let them argue, let them vote, and let their differences create a smarter, more human-like understanding of the story.

1. Problem Statement

The paper addresses the challenge of narrative similarity prediction, a task recently formalized as SemEval-2026 Task 4. Unlike standard semantic similarity tasks, narrative understanding is inherently interpretive; different valid perspectives can yield divergent similarity judgments for the same text pair.

The Core Issue: Traditional semantic evaluation benchmarks encode a single "ground truth," treating interpretive multiperspectivity as noise or a challenge to be overcome.
The Hypothesis: Instead of suppressing diverse interpretations, the authors propose operationalizing multiperspectivity as a modeling component. They hypothesize that aggregating diverse interpretive perspectives (via an ensemble) can improve prediction accuracy, drawing on the Condorcet Jury Theorem (CJT), which suggests collective decisions improve with the addition of independently competent voters.

2. Methodology

The authors constructed a multi-persona ensemble system to test this hypothesis.

Dataset: SemEval-2026 Task 4, consisting of triplets (Anchor story, Candidate A, Candidate B). The goal is to predict which candidate is more similar to the anchor.
- Split: 200 development samples, 400 test samples.
LLM Personas: An ensemble of 31 distinct personas was created, divided into two categories:
- Practitioners [P] (18 personas): Characters with explicit analytical frameworks (e.g., Feminist Literary Critic, Postcolonial Critic, Computational Narratologist, Ethicist).
- Lay People [L] (13 personas): Intuitive "everyday" characters (e.g., High School Student, Football Player, Tour Guide).
Model Architecture:
- Three distinct open-weight models were used to ensure model-family diversity: Gemma 27B-it, Qwen3-14B, and gpt-oss-20b.
- Each persona was instantiated on all three models, resulting in 93 total agents ( $31 \text{ personas} \times 3 \text{ models}$ ).
Voting Configurations:
1. Individual: Single persona on a single model.
2. Model-Specific Majority: 31 personas on one model voting.
3. Cross-Model Majority: All 93 predictions combined into a single majority vote.
Evaluation: Accuracy of the binary choice against gold data.

3. Key Contributions

Operationalizing Multiperspectivity: The paper is the first to employ diverse LLM persona ensembles grounded in distinct interpretive frameworks (critical theory, feminist criticism, etc.) specifically for narrative similarity prediction.
Empirical Validation of CJT in LLMs: It demonstrates that while LLMs violate the strict independence assumption of the Condorcet Jury Theorem, aggregating diverse perspectives still yields significant performance gains, particularly when error correlations are low.
Error Diversity Analysis: The study reveals a counter-intuitive finding: Practitioner personas (who use complex frameworks) perform worse individually than Lay personas but produce less correlated errors. This lack of correlation allows them to outperform Lay personas when aggregated via majority voting.
Vocabulary-Accuracy Correlation: A systematic analysis reveals a negative correlation between the use of gender-focused/feminist vocabulary (e.g., "patriarchal," "female protagonist") and prediction accuracy, suggesting that valid interpretive dimensions may be penalized by current benchmark ground truths.

4. Results

Ensemble Size & Accuracy:
- Accuracy consistently increases with ensemble size.
- Cross-model majority vote achieved 75.8% accuracy on the development set and 0.705 on the test set (ranking 13th out of 47 teams in SemEval-2026).
- Individual persona accuracy ranged from ~63% to ~71%.
Practitioner vs. Lay Performance:
- Individual Level: Lay personas ([L]) generally outperformed Practitioners ([P]) (71.7% vs. 71.0%).
- Ensemble Level: The [P] group achieved higher majority vote accuracy (76.0%) compared to [L] (75.3%).
- Error Correlation: [P] personas had significantly lower pairwise error correlation ( $r=0.388$ ) compared to [L] ( $r=0.461$ ). This indicates that Practitioners make different types of mistakes, which cancel each other out in a vote, whereas Lay personas tend to make similar mistakes.
Optimization vs. Aggregation:
- An optimized subset of 8 personas (selected via exhaustive search) achieved 82.0% on the dev set but dropped to 0.705 on the test set.
- The simple majority vote of all 31 personas achieved nearly identical test performance (0.7025), suggesting that broad aggregation is more robust than targeted subset optimization for interpretive tasks.
Vocabulary Analysis:
- The presence of gender-distinctive terms (e.g., "female," "gender roles," "objectification") showed a statistically significant negative correlation with accuracy across all persona categories.
- For example, when the term "female" appeared, accuracy dropped from 66.6% to 56.5%.

5. Significance and Implications

Reframing Evaluation: The paper argues that in interpretive NLP tasks, diversity of perspective is a feature, not a bug. The "noise" of differing interpretations can be harnessed to improve robustness.
Benchmark Limitations: The negative correlation between feminist/gender vocabulary and accuracy suggests that current benchmarks may encode a specific, non-inclusive ground truth. Valid interpretations that focus on gender dynamics are systematically penalized, raising questions about how future benchmarks should handle interpretive plurality.
Ensemble Strategy: The findings support the use of large, diverse ensembles over small, "optimized" subsets for tasks where item difficulty and annotation variability are high.
Future Directions: The authors note that while majority voting works well, the potential of multi-agent debate (iterative interaction) remains unexplored and could further leverage these diverse perspectives.

In conclusion, the paper demonstrates that treating narrative similarity as a multi-perspective problem and aggregating diverse LLM personas leads to state-of-the-art performance, while simultaneously exposing limitations in how current benchmarks evaluate interpretive nuance.