Technical Summary: Modeling L1 Influence on English Vocabulary Difficulty

Problem Statement

The acquisition of second-language (L2) vocabulary is a fundamental challenge, yet the difficulty of learning specific words varies significantly depending on the learner's native language (L1). While intrinsic lexical properties (e.g., frequency, concreteness) affect all learners, L1-specific factors such as orthographic overlap and cognate status create divergent learning curves. Existing computational models for lexical difficulty often fail to account for individual learner backgrounds or are limited to single language pairs. This paper addresses the gap in modeling English vocabulary difficulty for learners with distinct typological backgrounds: Spanish, German, and Chinese.

Methodology

Data

The study utilizes the Knowledge-based Vocabulary Lists (KVL), a crowd-sourced corpus containing lexical-difficulty test responses from over 100,000 English learners. The dataset covers three L1 groups (Spanish, German, Chinese) with approximately 6,091 training items, 677 development items, and 748 test items per language.

• Task Format: Learners translate an L1 word into English given the first letter and the number of characters (e.g., German Kabel $\to$ c _ _ _ _).
• Target Variable: Item difficulty is estimated via a Rasch model, producing a continuous logarithmic score where higher values indicate easier words.

Feature Engineering

The authors extract 24 features per item, organized into four distinct groups to capture different cognitive mechanisms:

1. Familiarity (11 features): Captures exposure effects, including log-transformed word frequency, contextual diversity, age of acquisition (AoA), percentage of annotators who knew the word, and CEFR proficiency levels (both CEFR-J and EFLLex corpus distributions).
2. Meaning (5 features): Approximates semantic complexity using the $\ell_2$ norm of fastText embeddings, mean hypernym depth, sense count (from WordNet), part-of-speech (POS) dominance ratio, and a binary flag for disambiguation requirements.
3. Surface (7 features): Describes orthographic form, including target/source word length, syllable count, letters-per-phoneme ratio (orthographic transparency), context sentence length, and the initial letters of the English clue and L1 translation.
4. Transfer (1 feature): Computes the cosine similarity between character $n$-gram TF-IDF vectors of the English word and its L1 translation. This serves as a proxy for orthographic cognateness. Notably, this feature yields zero similarity for Chinese due to script differences.

Model Architecture

• Primary Model: CatBoost (gradient-boosted decision trees) is trained separately for each L1. This choice allows for the modeling of non-linear interactions and feature importance while maintaining interpretability.
• Baselines:
  • Ridge Regression: A linear model trained on the same 24 features to test the necessity of non-linear interactions.
  • Transformer-based: A fine-tuned pretrained language model (Skidmore et al., 2025) trained directly on difficulty scores without handcrafted features.
• Evaluation: Models are evaluated using Root-Mean-Square Error (RMSE) and Pearson's $r$ on held-out test sets. Cross-L1 generalization is tested by evaluating models trained on one L1 against the test sets of the others.
• Interpretability: SHAP (Shapley Additive exPlanations) values are used to attribute predictions to individual features. Feature importance is aggregated by group to determine the relative contribution of familiarity, meaning, surface, and transfer mechanisms.

Key Results

Prediction Accuracy

The CatBoost models achieved the lowest RMSE and highest Pearson correlation across all three L1s compared to both the linear and transformer baselines.

• Spanish: CatBoost RMSE = 1.24 (vs. 1.26 Transformer, 1.30 Linear).
• German: CatBoost RMSE = 1.12 (vs. 1.26 Transformer, 1.20 Linear).
• Chinese: CatBoost RMSE = 1.04 (vs. 1.14 Transformer, 1.07 Linear).
  The results indicate that non-linear feature interactions and well-chosen psycholinguistic features outperform both linear combinations and raw transformer representations for this specific task.

Feature Importance and L1 Differences

SHAP analysis revealed distinct difficulty profiles based on the learner's L1:

1. Familiarity: This was the dominant feature group across all three languages. However, the specific drivers differed:
   • Spanish/German: Primarily driven by Age of Acquisition (AoA) and Frequency.
   • Chinese: Primarily driven by Proficiency Levels (CEFR-J and EFLLex span), suggesting a more curriculum-driven acquisition pattern.
2. Transfer (Orthographic Overlap): This feature acted as a critical differentiator.
   • For Spanish and German, transfer was the strongest predictor overall (mean $|SHAP| \approx 0.51-0.52$), particularly for less familiar words. It provided a "second route" to easiness via cognates.
   • For Chinese, this feature contributed zero importance due to the lack of shared script.
3. Surface vs. Meaning: Surface features (length, initial letters) were significantly more important for Chinese learners than for Spanish or German learners. Conversely, meaning features contributed minimally across all groups, likely because the KVL task tests form recall (spelling) rather than semantic understanding.

Cross-L1 Generalization

• Spanish $\leftrightarrow$ German: Models trained on one language generalized reasonably well to the other (e.g., Spanish-trained model on German test set: RMSE 1.18 vs. 1.12 within-German). This suggests shared cognitive strategies for typologically related languages.
• Chinese: Transfer involving Chinese was markedly worse. A Chinese-trained model failed to generalize to Spanish/German (RMSE > 1.5), and models trained on Spanish/German performed worse on Chinese than the native Chinese model, though they still outperformed baselines. This confirms that the difficulty mechanisms for Chinese learners are qualitatively distinct.

Significance and Contributions

The paper claims several contributions to the field of computational linguistics and educational technology:

1. L1-Tailored Modeling: It demonstrates that vocabulary difficulty is not a universal metric but is shaped by specific L1-L2 interactions. By training separate models, the authors capture these nuances, outperforming agnostic models.
2. Interpretability: The use of SHAP values allows for a decomposition of why a word is difficult. The study identifies two distinct routes to easiness for Spanish/German learners (familiarity OR orthographic transfer) versus a unified reliance on familiarity and surface cues for Chinese learners.
3. Curriculum Design Implications: The findings suggest that curricula for Spanish and German learners could leverage cognates to accelerate vocabulary acquisition, whereas curricula for Chinese learners should prioritize frequency and structured proficiency levels.
4. Scalability: The authors provide an interactive demo and an extended vocabulary (4,000–7,000 additional words per language) derived from Wiktionary, demonstrating that these models can generate difficulty estimates beyond the original KVL dataset.

The authors maintain a modest stance, noting that their feature importance analysis describes model prediction structures rather than direct cognitive processes. They acknowledge limitations regarding the specific task format (spelling recall) and the limited language coverage, suggesting that future work could explore phonological similarity and calques to capture additional variance.