💬 NLP

Accurate and Efficient Statistical Testing for Word Semantic Breadth

This paper proposes a GPU-accelerated, Householder-aligned permutation test that accurately distinguishes genuine differences in word semantic breadth from directional variations, thereby reducing Type-I error by 32.5% and achieving a 23x speedup over CPU baselines.

Original authors: Yo Ehara

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yo Ehara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Measuring How "Wide" a Word's Meaning Is

Imagine you are trying to measure how many different "lives" a word lives.

A narrow word (like "colitis") is like a specialist who only ever works in one specific room. It always means the same thing.
A broad word (like "mark") is like a traveler who visits many different countries. It can mean a tattoo, a grade on a test, a target, or a sign.

In the world of AI, we can see these "lives" by looking at the word's contextual embeddings. Think of these as a cloud of tiny dots floating in a 3D space.

If the word is narrow, the dots cluster tightly together in one spot.
If the word is broad, the dots are spread out all over the place.

The goal of this paper is to build a better ruler to measure how "spread out" (dispersed) these dots are, so we can tell if one word is truly broader than another.

The Problem: The "Confused Crowd" Mistake

The author found a major flaw in how people usually try to compare these clouds of dots.

The Analogy:
Imagine two groups of people standing in a large park.

Group A (The "Broad" Word): They are spread out across the whole park, but they are all standing near the North Gate.
Group B (The "Narrow" Word): They are huddled tightly together, but they are standing near the South Gate.

If you want to know if Group A is more spread out than Group B, a naive test might just say, "Hey, Group A covers more ground!" But wait—Group A is spread out because they are near the North Gate, and Group B is tight because they are near the South Gate. The difference in their location (North vs. South) is messing up the measurement of their spread.

In the paper's technical terms, this is called a Type-I Error. The test falsely claims there is a difference in "breadth" when the real difference is just a difference in "direction" (where the group is standing). It's like thinking a crowd is chaotic just because they are standing in a different part of the room.

The Solution: The "Magic Mirror" (Householder Alignment)

To fix this, the author invented a clever trick using a mathematical tool called a Householder reflection.

The Analogy:
Imagine you have a giant, magical mirror (the Householder matrix).

You take Group A (the North Gate crowd) and reflect them in the mirror.
Suddenly, Group A is no longer at the North Gate; they have been moved to stand right next to Group B at the South Gate.
Crucially, the mirror didn't change how spread out the people were; it just moved the whole group to the same starting line.

Now, both groups are standing in the exact same spot. If you shuffle the people between the two groups (a process called permutation testing), you are only testing how spread out they are, not where they are standing. This gives you a fair, accurate comparison.

The Results: Smarter and Faster

The author tested this new method and found two big wins:

It's More Accurate: By using the "magic mirror" to align the groups first, the test stopped making false alarms. It reduced the number of false "significant differences" by 32.5%. It became much better at telling the difference between a word that is genuinely broad and one that just happens to be in a different part of the map.
It's Much Faster: Doing these shuffles and calculations on a standard computer (CPU) is slow, like trying to sort a deck of cards by hand. The author wrote a version that runs on a GPU (the powerful graphics chips used for gaming and AI). This made the process 23 times faster. It's like switching from sorting cards by hand to using a high-speed machine.

Why Does This Matter?

This isn't just about math; it helps people who build dictionaries and language resources.

Dictionary Makers: If they are trying to decide if a word needs a new definition (a new "sense"), they can use this test. If the test says the word is truly broader than expected, they know they need to write more definitions for it. If the test says the difference is just a fluke, they can save time and not over-complicate the dictionary.
Language Learners: The author suggests that words with a "wider" spread of meanings might be harder for students to learn because they appear in so many different situations.

Summary

The paper fixes a broken ruler used to measure word meanings. The old ruler got confused by where the words were standing. The new ruler uses a "magic mirror" to move everyone to the same spot before measuring, ensuring the results are fair. Plus, it runs 23 times faster, making it practical for real-world use.

Technical Summary: Accurate and Efficient Statistical Testing for Word Semantic Breadth

Problem Statement
The paper addresses a critical flaw in using non-parametric permutation tests to compare the "semantic breadth" (contextual diversity) of two word types based on contextualized token embeddings. While dispersion-based statistics (e.g., the spread of token vectors on a unit sphere) serve as effective proxies for semantic breadth, naive permutation testing yields inflated Type-I error rates.

The core issue arises because word embeddings often differ not only in dispersion but also in their mean direction (semantic orientation). In a standard permutation test, randomly shuffling labels between two groups with different mean directions creates "permuted" groups that span a wider area of the embedding space than the original groups. This artificially inflates the estimated dispersion of the permuted groups, leading the test to falsely reject the null hypothesis (claiming a significant breadth difference) even when the true dispersions are identical. This is particularly problematic in the "small-difference regime," where statistical testing is most needed to distinguish genuine effects from sampling noise.

Methodology
The authors propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. The methodology consists of three main stages:

Householder Mean-Direction Alignment:
Before conducting the permutation test, the method aligns the mean direction of the two word types. Given two sets of unit-normalized embeddings, $X$ and $Y$ , with mean directions $\hat{\mu}_X$ and $\hat{\mu}_Y$ , a Householder reflection matrix $H$ is constructed:
$H = I - 2 \frac{vv^\top}{v^\top v}, \quad \text{where } v = \hat{\mu}_X - \hat{\mu}_Y$
This matrix satisfies $H\hat{\mu}_X = \hat{\mu}_Y$ . Applying $H$ to all vectors in $X$ rotates the cloud to share the same mean direction as $Y$ while preserving norms and relative geometry. This ensures that subsequent permutations only shuffle labels within a common directional region, preventing the artificial inflation of dispersion.
Dispersion Proxy and Test Statistic:
Semantic breadth is quantified using the Mean Resultant Length (MRL), $r(X) = \|\frac{1}{n}\sum x_i\|_2$ . A smaller $r$ indicates higher dispersion (broader meaning). The authors define a breadth proxy $v(X)$ as an inverse concentration measure derived from $r(X)$ . The test statistic is the log-volume difference between the aligned set $X'$ and the unaltered set $Y$ :
$T_{obs} = \log v(X') - \log v(Y)$
GPU-Accelerated Permutation Inference:
To address the computational intensity of permutation testing, the authors introduce a vectorized, GPU-oriented implementation.
- Vectorization: Instead of looping through permutations, the method uses a sign matrix $S \in \{+1, -1\}^{B \times N}$ to represent $B$ permutations. Group sums are computed via a single General Matrix Multiplication (GEMM) operation: $U = SX$.
- Memory Efficiency: Permutations are processed in chunks to manage memory, updating the exceedance count for p-values on the fly without storing all intermediate statistics.
- Efficiency: The Householder application is optimized to require only a dot product and scaled vector subtraction per vector, avoiding explicit $d \times d$ matrix operations.

Key Contributions

Identification of a Failure Mode: The paper identifies that mean-direction differences in embedding spaces act as a confounder in naive permutation tests, systematically inflating Type-I error and undermining the validity of significance claims for semantic breadth.
Householder-Aligned Test: A novel statistical procedure that uses a single Householder reflection to remove the mean-direction nuisance factor. This allows the permutation test to target dispersion differences specifically, yielding calibrated p-values.
Scalable Implementation: A GPU-accelerated implementation of the permutation test that achieves a 23 $\times$ speedup over a CPU baseline, making large-scale resampling feasible for embedding-set statistics.

Experimental Results
The authors evaluated the method using ModernBERT embeddings on the British National Corpus (BNC) and BERT-large on the BCCWJ (Japanese) corpus, comparing against WordNet sense counts as a gold standard.

Type-I Error Reduction: In the small-difference regime (ranking gaps of 1–10), the proposed method reduced Type-I error by 32.5% compared to the naive baseline. For example, at a ranking gap of 10, the error rate dropped from 4.0% (baseline) to 2.67% (proposed), while maintaining or improving precision in detecting genuine sense differences.
Calibration: The aligned test demonstrated more conservative behavior, keeping rejection rates closer to the nominal significance level ( $\alpha = 0.05$ ) under the null hypothesis, whereas the baseline showed elevated false positive rates in challenging scenarios.
Performance: The GPU implementation reduced the time per permutation from 1.588 ms (CPU) to 0.069 ms, enabling the execution of 20,000 permutations in approximately 1.4 seconds.

Significance and Claims
The paper claims that its primary contribution is not the estimation of the number of senses per se, but the provision of a well-calibrated testing procedure. This procedure enables researchers and lexicographers to:

Distinguish between words that are plausibly well-covered by current sense granularities and those requiring additional sense splitting.
Perform reliable statistical inference in the "small-difference regime" where naive tests fail due to directional confounds.
Scale permutation testing to large vocabularies and corpora through efficient GPU acceleration.

The authors note that while the method assumes unimodal and isotropic distributions (similar to von Mises-Fisher models), it effectively captures observable regularities in word-type token clouds. They position this work as a foundational step for objective, corpus-driven lexicography and vocabulary selection, avoiding spurious detections caused by semantic direction mismatches.