Entropy of Ukrainian

1. Problem Statement

The paper addresses the lack of empirical data regarding the entropy (unpredictability and information density) of the Ukrainian language. While Claude Shannon's 1951 experiment established a methodology for measuring language entropy via human prediction, and subsequent studies have quantified this for English and Hebrew, no such study had been conducted for Ukrainian.

Existing estimates for Ukrainian relied on statistical N-gram models (Babenko and Sushko, 2012), which yielded an entropy of 1.25–1.40 bits per character (bpc) for $N=9$. However, N-gram models suffer from data sparsity as context length increases and cannot capture long-range dependencies inherent in natural language. The authors aim to replicate Shannon's human-guessing experiment to provide a more accurate, context-independent upper bound on Ukrainian entropy.

2. Methodology

The study replicates Shannon's experiment using a volunteer-based online platform, adapting the methodology from Ren et al. (2019) (who studied English).

• Experimental Design:

  • Participants: 184 volunteers recruited via Telegram and word-of-mouth.
  • Task: Participants were presented with sentences from Ukrainian news articles (Ukrainska Pravda). The first 70 characters were revealed, and participants had to guess the next character sequentially until the correct one was entered.
  • Constraints: To maintain engagement, the experiment started at position 70 rather than 0. Sentences were filtered to be between 120–200 characters.
  • Data Collection: 323 users registered; 184 started sessions. The study collected 17,023 correct guesses (observations) from 501 completed sessions and 192 incomplete sessions.

• Entropy Calculation:

  • The true entropy $H$ is bounded by $H_{lower} \le H \le H_{upper}$.
  • Upper Bound ($H_{upper}$): Calculated based on the distribution of the number of guesses required to find the correct character.
  • Lower Bound ($H_{lower}$): Calculated based on the "least surprising" distribution (assuming a character guessed on the $i$-th try has probability $1/i$).
  • Formula: $H_{upper} = -\sum_{i=1}^{K} q_i \log_2 q_i$, where $q_i$ is the fraction of characters guessed correctly on the $i$-th try.

• Data Processing & Filtering:

  • Outlier Removal: A binomial analysis was used to remove "suspicious" sessions (4.3%) where performance was statistically improbable (likely due to cheating or looking up answers online).
  • Trimming: To mitigate the "volunteer effect" (lower engagement than paid workers), the authors aggressively trimmed the bottom 65% of performing sessions. This was chosen to minimize the confidence interval width while retaining natural variance.
  • Pooling: Data from positions 70 to 110 were pooled to calculate the final entropy, as context beyond 70 characters offered diminishing returns for guess accuracy.

3. Key Contributions

1. First Human-Based Entropy Measurement for Ukrainian: This is the first study to apply Shannon's guessing experiment to the Ukrainian language.
2. Refined Upper Bound: The study establishes a new upper bound of 1.201 bpc for Ukrainian, which is lower than previous N-gram estimates (1.25–1.40 bpc), indicating that longer contexts significantly reduce unpredictability.
3. LLM Benchmarking: The authors compared human performance against state-of-the-art Large Language Models (LLMs) on the same corpus, providing a baseline for model efficiency vs. human intuition.
4. Open Source: The code, data, and experimental interface are published to facilitate future research.

4. Key Results

Human Performance

• Upper Bound: 1.201 bpc (95% Confidence Interval: [1.10, 1.19]).
• Lower Bound: 0.5987 bpc.
• Redundancy: The language exhibits approximately 76.4% redundancy.
• Comparison to English: The entropy of Ukrainian (1.201 bpc) is remarkably similar to English (1.22 bpc), despite Ukrainian having a larger alphabet (33 letters + space vs. 26 + space). The authors suggest that less frequent letters in Ukrainian do not significantly increase overall entropy.

Large Language Model (LLM) Performance

The authors tested several pre-trained models (Gemma-3, Llama-3.1, Qwen, Mistral, and the Ukrainian-specific Lapa model) on the same dataset.

• Best Performers: Gemma-3 (27B) achieved 0.717 bpc, and Lapa (12B) achieved 0.734 bpc.
• Comparison: LLMs significantly outperform humans (lower BPC is better).
• Overfitting Concern: While LLMs approach the human lower bound (0.599 bpc), the authors note this may indicate overfitting to the specific news corpus rather than capturing the true entropy of the entire language. The gap between the human upper bound (1.201) and LLM results suggests the true entropy lies somewhere in between.

5. Significance and Limitations

Significance:

• Information Density: Confirms that Ukrainian has a similar information density to English, validating cross-linguistic compression and modeling assumptions.
• Model Evaluation: Provides a rigorous human baseline for evaluating Ukrainian NLP models. The fact that top LLMs are approaching human lower bounds suggests they are highly effective at modeling Ukrainian syntax and semantics, though potential overfitting must be investigated.
• Methodological Insight: Highlights the challenges of volunteer-based experiments (engagement, cheating) and the necessity of aggressive data trimming to obtain reliable entropy estimates.

Limitations:

• Sample Size: The dataset (17k observations) is smaller than comparable English studies (Ren et al. had ~172k), leading to wider confidence intervals.
• Context Window: The experiment started at position 70, preventing the calculation of an entropy curve from position 0 and making extrapolation to infinite context difficult.
• Dataset Bias: The corpus consisted of only 5 news articles, which may not represent informal speech, regional dialects, or creative writing, potentially underestimating the true entropy of the full language.
• Volunteer Bias: Unpaid volunteers were less engaged than paid workers, potentially inflating the entropy estimate (making the language appear more unpredictable than it is).

Conclusion

The paper successfully quantifies the entropy of Ukrainian as approximately 1.201 bpc, refining previous estimates and demonstrating that Ukrainian and English share similar information densities. While modern LLMs outperform humans in prediction tasks, the study underscores the need for larger, more diverse datasets and better controls to distinguish between true language modeling and overfitting to specific domains.