SwanNLP at SemEval-2026 Task 5: An LLM-based Framework for Plausibility Scoring in Narrative Word Sense Disambiguation

1. Problem Statement

The paper addresses a gap in Natural Language Understanding (NLU): while Large Language Models (LLMs) perform well on standard Word Sense Disambiguation (WSD) tasks at the sentence level, their ability to handle narrative contexts remains underexplored.

• The Challenge: Traditional benchmarks often rely on isolated sentences where disambiguation is based on immediate syntactic or neighbor-word clues. However, real-world language understanding often requires multi-sentence context to resolve ambiguity.
• The Task: SemEval-2026 Task 5 introduces the AmbiStory dataset, which presents short narratives (pre-context, ambiguous target sentence, and optional ending). The goal is to predict a human-perceived plausibility score (1–5) for a specific word sense within that narrative. This requires simulating human reasoning rather than just selecting the "correct" sense, as human annotators often disagree on the plausibility of ambiguous cases.

2. Methodology

The authors propose a multi-faceted framework utilizing three distinct approaches to simulate human plausibility judgments:

A. Supervised Fine-Tuning (SFT) of Low-Parameter LLMs

The team fine-tuned smaller models (Gemma-4B and Qwen-4B) using Low-Rank Adaptation (LoRA) to simulate human annotation. They tested four specific reasoning strategies:

1. Single Annotator Simulation: Predicting a score as if from one human.
2. Five Annotator Simulation: Generating five distinct scores to capture variance and consensus.
3. Reasoning-Driven (Chain-of-Thought): The model first analyzes contextual cues and the narrative ending to determine plausibility before assigning a score.
4. Difficulty-Aware Analysis: The model first classifies the instance as "Human-Easy" or "Human-Ambiguous" (based on pre-computed difficulty tags) and then reasons about the sense accordingly.

• Key Insight: The authors used GLOSSGPT to pre-compute likely sense interpretations, using them as auxiliary reasoning cues rather than ground-truth labels to prevent over-reliance on a single sense.

B. Dynamic Few-Shot Learning with RAG for Large Models

Recognizing that small models struggle with "moderately plausible" ambiguous cases, the team applied a Retrieval-Augmented Generation (RAG) approach to large commercial models (GPT-4o, DeepSeek V3, Gemini).

• Data Categorization: Training data was split into three categories based on human annotation variance:
  • Ambiguous Context: High disagreement among annotators.
  • Human Easy (High Score): High agreement on high plausibility.
  • Human Easy (Low Score): High agreement on low plausibility.
• Retrieval Mechanism: During inference, the system retrieves relevant few-shot examples (K=1 or K=3) from a vector store (using BAAI/bge-small-en-v1.5 embeddings and FAISS) to guide the model's in-context learning.

C. Model Ensembling for Multi-Annotator Agreement

To better simulate the consensus of five human annotators, the team employed an ensemble approach where each model in the previous phases acted as an individual "annotator."

• Strategies Tested: Majority Voting, Equal-Weight Averaging, Performance-Based Weighted Averaging, Linear Regression Ensemble (LRE), Support Vector Regression (SVR), and XGBoost.
• Training: Ensemble models were trained on the development set using k-fold cross-validation to ensure robustness.

3. Key Contributions

• Narrative WSD Framework: A structured framework that moves beyond sentence-level WSD to evaluate plausibility in multi-sentence narrative contexts.
• Reasoning-Driven Fine-Tuning: Demonstrated that explicitly teaching low-parameter models to analyze "difficulty" and perform "Chain-of-Thought" reasoning significantly outperforms standard fine-tuning.
• Dynamic Few-Shot RAG: Showed that dynamically retrieving context-specific examples (categorized by ambiguity level) enhances the performance of large commercial models in simulating human judgment.
• Ensemble Simulation: Proved that aggregating predictions from multiple models (simulating multiple annotators) yields higher correlation with human judgments than single-model predictions.

4. Results

The team achieved 10th place on the leaderboard with an average score of 0.760.

• Low-Parameter Models:

  • Qwen-4B consistently outperformed Gemma-4B.
  • The Difficulty Analysis strategy yielded the best results for fine-tuned models (e.g., Qwen-4B achieved a Spearman correlation of 0.491 on the test set with this strategy).
  • Simulating five annotators directly via fine-tuning generally underperformed compared to single-annotator simulation, suggesting that forcing variance in small models introduces noise.

• Large Parameter Models (Dynamic Few-Shot):

  • GPT-4o was the strongest performer overall.
  • Dynamic few-shot prompting improved performance over zero-shot baselines.
  • Optimal K: Interestingly, 1-shot prompting performed best on the test set, while 3-shot was better on the development set, indicating a trade-off between generalization and overfitting to examples.
  • GPT-4o (3-shot) achieved a test Spearman correlation of 0.755 and Accuracy of 0.798.

• Ensembling:

  • Ensembling significantly improved results.
  • XGBoost achieved the highest Spearman correlation on the test set (0.724).
  • Linear Regression Ensemble achieved the highest Accuracy on the test set (0.797).
  • SVR performed best on the development set, suggesting non-linear relationships in the data.

5. Significance and Conclusion

• Human Alignment: The study confirms that LLMs can closely replicate human plausibility judgments, particularly when guided by dynamic few-shot prompting and ensemble methods.
• Reasoning is Key: The results highlight that "reasoning-aware" approaches (identifying difficulty, using CoT) are superior to direct score prediction, especially for smaller models.
• Limitations: Despite high performance, models still struggle with consistency in highly ambiguous or context-sensitive scenarios where human judgment is inherently uncertain. The gap between "approximating" human reasoning and achieving perfect "human-aligned" nuance remains.
• Future Direction: The work suggests that future WSD systems should prioritize structured reasoning mechanisms and ensemble strategies that mimic multi-annotator consensus rather than relying on single-model deterministic outputs.