💬 NLP

What Makes Words Hard? Sakura at BEA 2026 Shared Task on Vocabulary Difficulty Prediction

This paper presents two models for vocabulary difficulty prediction developed for the BEA 2026 Sakura shared task: a high-accuracy black-box LLM that achieved top results and an explainable model that offers insights into factors like spelling and test construction while outperforming baseline encoders.

Original authors: Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang, Hitomi Yanaka

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang, Hitomi Yanaka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to figure out which English words are the hardest for your students to spell and use correctly. You have a giant list of words, and for each one, you know exactly how many students got it right or wrong in a test. Your goal is to build a computer program that can look at a word and guess its "difficulty score" before you even give it to a student.

This paper is about two different teams (or approaches) that tried to build the best possible "difficulty predictor" for a competition called the BEA 2026 Shared Task. Here is how they did it, explained simply.

The Two Main Approaches

The authors tried two very different strategies, like a master chef using two different recipes.

1. The "Black Box" Chef (The High-Accuracy Model)

This approach is like hiring a super-smart, mysterious chef who can taste a dish and instantly know exactly how salty it is, but you can't ask why they think that.

How it works: They took a massive, pre-trained AI (a Large Language Model or LLM) and taught it to predict difficulty scores.
The Secret Sauce: Usually, when you teach an AI to guess a number (like 3.5), you force it to pick a whole number (like 3 or 4) and then punish it for being wrong. The authors found a better way: they taught the AI to think in "probabilities." Instead of saying "It's a 3," the AI learned to say, "It's 60% likely to be a 3 and 40% likely to be a 4." This is called using soft targets.
The Result: This method was incredibly accurate. It won the "Open Track" of the competition (where you can use any tools you want), beating almost everyone else. It's the "smartest" model, but it's a bit of a mystery because it's hard to see exactly which specific rule it used to make its decision.

2. The "Explainable" Detective (The Transparent Model)

This approach is like hiring a detective who solves the case but writes down every single clue they used. You know exactly why they think a word is hard.

How it works: Instead of letting the AI guess blindly, the authors gave the model a checklist of specific features:
- Spelling Difficulty: How hard is it to spell?
- Frequency: How often do learners actually write this word?
- Similarity: Does the word look like the student's native language (e.g., Spanish or German)?
- Confusion: Does the word have multiple meanings that might trick the student?
The Result: This model wasn't quite as accurate as the "Black Box" chef, but it was still very good. More importantly, it told the researchers why it made its guesses. It used a tool called SHAP (which is like a magnifying glass) to show which clues mattered most.

What They Discovered About "Hard Words"

By looking at the "Detective's" notes, the authors found some surprising things about what makes a word difficult in these tests:

Spelling is King: For speakers of Spanish and German, the biggest hurdle wasn't knowing the meaning of the word; it was spelling it correctly. If a word is long or looks weird, it gets a high difficulty score.
The "Trick" Factor: Sometimes, the test itself is tricky. The authors found that some test questions were designed in a way that confused even the smartest AI. For example, a test might give a clue that points to a word that almost fits, but not quite. The authors called this "trickiness." It's not that the English word is hard; it's that the puzzle is poorly designed.
Native Language Matters:
- For Chinese speakers, knowing the word's "CEFR level" (a standard grade for English proficiency) was the biggest clue.
- For German and Spanish speakers, how similar the English word looks to their native word was a huge factor.

The Big Takeaway

The paper shows that to predict how hard a word is, you need two things:

Raw Power: A massive AI that can learn patterns from millions of examples (the "Black Box").
Human Insight: Understanding that difficulty isn't just about the word itself; it's also about how the word is spelled and how the test question is written (the "Detective").

The authors made their code available so others can try to build even better "difficulty predictors" in the future. They proved that while a "Black Box" AI is the most accurate, understanding the "Detective's" clues helps us understand why students struggle with certain words.

Technical Summary: What Makes Words Hard? (BEA 2026 Shared Task)

Problem Definition

The paper addresses the Vocabulary Difficulty Prediction task for English learners, specifically utilizing the British Council's Knowledge-based Vocabulary Lists (KVL). The goal is to predict the difficulty of English words given a learner's first language (L1: Chinese, German, or Spanish). Unlike traditional lexical complexity prediction (LCP) which focuses on reading comprehension, this task focuses on productive knowledge: the ability to write an English word with correct spelling given an L1 equivalent, an L1 context, and a clue (first letter and word length).

The difficulty scores are derived from a Generalized Linear Mixed Model (GLMM), representing the log-odds of a learner responding correctly. The challenge lies in predicting these continuous values accurately while understanding the factors driving difficulty, particularly given that the dataset includes test items where difficulty may be influenced by factors beyond the inherent difficulty of the word itself (e.g., spelling complexity or test item construction).

Methodology

The authors propose two distinct modeling approaches: a high-accuracy "black-box" model and an explainable feature-based model.

1. Fine-Tuning with Soft Targets (Black-Box Model)

The core innovation is a technique to fine-tune Large Language Models (LLMs) and Masked Language Models (MLMs) for continuous value prediction using cross-entropy loss with soft targets.

Standard Approach: Typically, continuous values are discretized (e.g., rounded to integers) and treated as hard targets for cross-entropy loss, or regression heads with Mean Squared Error (MSE) are used. The authors argue these methods lose precision or are not standard for LLMs.
Proposed Method:
- Training: The model is prompted to predict values on a discrete scale (e.g., $S = \{1, 2, 3, 4, 5\}$ ). Instead of assigning probability mass to a single target token, the continuous target $y$ is expressed as a probability-weighted sum of its two nearest integer points ( $a$ and $a+1$ ). The loss function is the cross-entropy between the model's predicted distribution and this soft target distribution.
- Inference: The model outputs a token probability distribution over the discrete scale. The final continuous prediction $\hat{y}$ is computed as the probability-weighted mean of the scale values.
Implementation: The authors fine-tuned various models (GLM-4, Qwen2.5, Ministral-3, mmBERT, XLM-RoBERTa) using 4-bit quantization and QLoRA. They employed linear stacking ensembles of these models to maximize performance in the Open Track.

2. Explainable Model

To interpret the factors influencing difficulty, the authors trained an XGBoost regressor using a set of hand-crafted and LLM-extracted features.

Features:
- Production/Reception Frequency: Log-frequencies from learner corpora (Lang-8) and spoken corpora (TUBELEX, BNC).
- Lexical Properties: Word length, CEFR level, and L1 similarity (Levenshtein distance).
- LLM-Extracted Features: Using GPT-5.2 and DeepSeek-V3.2 to rate Spelling Difficulty, Lexical Ambiguity (polysemy/homonymy in the specific test context), and L1 Calque status (morpheme-for-morpheme translation). These features were converted to continuous values using the G-Scale method (temperature scaling + probability-weighted mean).
Analysis: The authors used SHAP (SHapley Additive exPlanations) to quantify the impact of each feature on the prediction.

3. Additional Features and "Trickiness"

In the Open Track, the authors introduced a "trickiness" feature. They prompted LLMs (GPT-4.1-mini/nano) to solve the test items. The probability of the LLM answering incorrectly served as a proxy for the "trickiness" of the test item construction (e.g., misleading L1 contexts or clues), distinct from the inherent difficulty of the English vocabulary.

Key Results

Open Track Performance

The fine-tuned LLM ensemble (using soft targets) achieved the top results in the Open Track, with a mean Root Mean Square Error (RMSE) of 0.710.
The best individual models (GLM-4-32B, Qwen2.5-32B, Ministral-3-14B) achieved RMSEs between 0.751 and 0.753.
Adding external features (frequency, trickiness) to the fine-tuned LLMs provided marginal improvements, suggesting the LLMs themselves captured most of the signal.
The results approached the "Statistical Optimum" (the theoretical limit of performance given the noise in the gold standard data), which was an RMSE of 0.277.

Closed Track Performance

The explainable model (XGBoost with LLM-extracted features) achieved a mean RMSE of 1.067, outperforming the official baseline (fine-tuned XLM-RoBERTa, RMSE 1.218).
The closed_max model (adding a fine-tuned MLM as a feature to the explainable model) achieved the best closed-track result with an RMSE of 0.921.
The authors noted that the "traditional" feature-only model (without LLM-extracted features) performed on par with the baseline, while the inclusion of LLM-based features (spelling difficulty, ambiguity) provided the significant boost.

Ablation Studies

Soft Targets vs. Standard Loss: Replacing the soft-target cross-entropy with standard discretized cross-entropy or MSE regression heads resulted in significant performance drops (e.g., RMSE increase from 0.753 to 0.832 for the LLM).
Model Size: Performance generally improved with model size up to 32B parameters. Notably, the 14B Ministral-3 model performed comparably to 32B models, outperforming smaller MLMs of similar parameter counts.
Language Specificity: Training on single-language data yielded results comparable to multi-language training for most cases, though multi-language training was slightly superior.

Analysis and Insights

The SHAP analysis revealed several key drivers of difficulty in the KVL dataset:

Production Frequency: The most consistent high-impact feature across all L1s, aligning with the task's focus on written production.
Spelling Difficulty: The most important feature for German learners and second most for Spanish. It was less critical for Chinese learners, likely due to the lack of orthographic interference from Chinese characters.
Lexical Ambiguity: High impact, suggesting that test items often rely on less common senses of words or homonyms, which confuses learners.
Test Item Construction ("Trickiness"): The analysis of "trickiness" and local SHAP explanations indicated that difficulty scores are often inflated by the specific construction of the test item (e.g., L1 contexts that suggest a different English word, or clues that are ambiguous) rather than the intrinsic difficulty of the word.

Significance and Claims

The paper claims the following contributions:

Methodological Novelty: It introduces a novel technique for fine-tuning LLMs and MLMs to predict continuous values using cross-entropy loss with soft targets, demonstrating that this approach outperforms standard discretization or regression head methods for this specific task.
State-of-the-Art Performance: The proposed models achieved the top results in the BEA 2026 Shared Task, surpassing previous baselines and demonstrating that LLMs can effectively model vocabulary difficulty when trained with the appropriate loss function.
Interpretability: The explainable model provides empirical evidence that spelling difficulty and test item construction (beyond just the word's productive difficulty) are significant factors in the KVL dataset. This suggests that the dataset's difficulty scores may reflect the specific constraints of the test format (productive spelling) as much as the lexical complexity itself.
Limitations: The authors modestly note that the soft-target method was tested only on this specific dataset and its applicability to other tasks or noisier data remains uninvestigated. They also acknowledge that the "trickiness" insights cannot be empirically validated without access to individual learner responses, which are not public.