💬 NLP

Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts

This paper proposes a novel framework to disentangle grammatical gender from semantic bias in contextual embeddings for gendered languages like Spanish, demonstrating that unweighted controlled contexts combined with centroid estimators effectively isolate grammatical gender directions while preserving semantic distinctions.

Original authors: Huanping Xiao, Yingji Li

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huanping Xiao, Yingji Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot librarian (a "language model") that has read almost everything written in Spanish and French. This robot is great at understanding words, but it has a confusing problem: it mixes up two very different types of "gender."

The Grammar Gender: In languages like Spanish and French, every object has a gender just because of grammar rules. A "table" is feminine, and a "chair" is masculine, even though neither is a boy or a girl. This is like a uniform the object must wear.
The Social Gender: This is the bias we worry about. The robot has learned from real-world books and articles that "nurses" are usually women and "engineers" are usually men. This is the robot's opinion based on society.

The problem is that the robot's brain (its "embeddings") mixes these two together. It's hard to tell if the robot thinks a "table" is feminine because of grammar rules or because it thinks tables are "soft" and "feminine" (which is a bias).

The Big Goal

The authors of this paper wanted to teach the robot how to separate these two things. They wanted to find the "pure" direction in the robot's brain that represents only the grammar rules, without the social bias. Once they find that pure direction, they can "turn it off" for objects that shouldn't have a gender, without accidentally deleting the important gender information for jobs (like "doctor" vs. "nurse").

How They Did It: The "Clean Room" vs. The "Messy Room"

To figure out the pure grammar direction, the researchers tried two different ways of asking the robot about words:

The Messy Room (Natural Contexts): They asked the robot to look at sentences taken from real Wikipedia articles. This is like asking a student to study in a noisy cafeteria. The robot sees the word "table" surrounded by other words about furniture, art, or history. These extra words "contaminate" the answer with social bias.
The Clean Room (Controlled Templates): They created a special, boring, repetitive sentence template for every word. For example: "The [word] was mentioned in the text." They did this for thousands of words. This is like putting the student in a silent, white room with no distractions.

The Surprise Finding:
The researchers thought that maybe they could use math to "clean up" the messy Wikipedia sentences. But they discovered that the Clean Room was far superior. The "boring" sentences gave them the purest, most accurate map of the grammar rules. Trying to fix the messy sentences with math only made things slightly better, but never as good as just using the clean sentences to begin with.

The Tools: How to Draw the Map

Once they had the data, they needed to draw a line (a vector) that separates "masculine" grammar from "feminine" grammar. They tried three different tools:

The Average (Centroid): They just took all the "masculine" words, averaged them, took all the "feminine" words, averaged them, and drew a line between the two centers.
The Strict Teacher (SVM & LDA): These are complex math tools that try to draw a perfect line that separates every single example perfectly, like a strict teacher grading a test.

The Winner:
Surprisingly, the simple Average (Centroid) method worked the best. The complex "Strict Teachers" got confused by the noise and made mistakes. The simple average was robust enough to ignore the chaos and find the true grammar direction.

The Result: A Balanced Solution

The researchers created a new way to test their work. They wanted to make sure that when they removed the "grammar gender" from objects like "tables" and "chairs," they didn't accidentally erase the "social gender" from words like "actor" and "actress."

Their method worked perfectly:

It successfully removed the unnecessary grammar gender from inanimate objects (so the robot stops thinking a "table" is inherently feminine).
It kept the important social gender distinctions for jobs intact (so the robot still understands the difference between a male and female doctor).

In a Nutshell

This paper is like a guide for cleaning a dirty window. The authors found that trying to wipe the window while it's still covered in rain and mud (natural text) doesn't work well. Instead, you need to take the glass out, put it in a clean room (controlled templates), and wipe it there. They also found that a simple, gentle wipe (averaging) works better than trying to scrub every single spot with a complex tool. This allows us to fix the robot's bias without breaking its ability to understand human jobs and roles.

Technical Summary: Estimating Grammatical Gender Directions in Contextual Embeddings under Controlled and Natural Contexts

1. Problem Statement

Pre-trained contextual language models (e.g., BERT, XLM-R) encode pervasive social gender stereotypes. While existing debiasing methods effectively address this in English (a language without grammatical gender), they face critical limitations when applied to grammatically gendered languages like Spanish and French. In these languages, two distinct gender signals coexist within embeddings:

Mandatory Morphosyntactic Grammatical Gender: An inherent linguistic property where inanimate nouns are fixed as masculine or feminine and require agreement with articles and adjectives.
Social Semantic Gender Bias: Unwanted stereotypes learned from real-world corpora.

Current approaches conflate these two signals. Furthermore, prior disentanglement work has been limited to static word embeddings, leaving contextual representations unexplored. A significant challenge is semantic contamination: even semantically neutral inanimate nouns absorb contextual gendered descriptions from natural text, making it difficult to isolate the pure grammatical gender direction required for effective debiasing.

2. Methodology

The authors propose a contamination-aware framework to disentangle grammatical gender from semantic bias in contextual embeddings. The methodology consists of four main components:

A. Data Construction

Inanimate Nouns: Balanced datasets of Spanish and French inanimate nouns were constructed (approx. 6,000 per language) by filtering out human, occupational, and proper nouns.
Context Sources: Two distinct context types were created for each noun:
- Controlled Templates: Semantically neutral, fixed templates designed to minimize topic, lexical, and co-occurrence noise, serving to extract pure grammatical signals.
- Natural Contexts: Sentences from Wikipedia used to quantify the degree of semantic contamination present in real-world usage.
Evaluation Lexicon: A separate set of 100 masculine-feminine occupation pairs per language was created to test whether debiasing preserves legitimate semantic gender distinctions.

B. Contamination Quantification
The framework characterizes semantic contamination as the degree to which inanimate noun embeddings align with a semantic gender axis (derived from male/female attribute word sets). Using a WEAT-style scoring framework, the authors compute a contamination score for each noun. To ensure robustness, two normalization strategies are applied:

Mean-based Normalization: Corrects for global shifts in contamination.
Median-based Normalization: Provides robustness against outliers and skewed distributions.

C. Grammatical Gender Direction Estimation
Three estimators are implemented to identify the grammatical gender direction ( $d_g$ ) from the noun embeddings:

Centroid Estimator: Computes the difference between the mean embeddings of masculine and feminine sets (distributional shift approach).
Linear Discriminant Analysis (LDA): Models gender as a supervised separation problem to maximize class separability.
Linear Support Vector Machine (SVM): Uses the normal vector of the separating hyperplane as the gender direction.

D. Evaluation Framework
A dual-objective metric system is proposed to balance two conflicting goals:

Inanimate Noun Reduction Score ( $R_{noun}$ ): Measures the suppression of grammatical gender leakage on inanimate nouns after orthogonal projection.
Occupational Preservation Score ( $P_{occ}$ ): Measures the retention of semantic gender distinctions in occupation terms after projection.
Trade-off Score: The harmonic mean ( $T_{harm}$ ) of the above two scores is used as the primary metric to rank methods, penalizing extreme imbalances.

3. Key Results

Experiments were conducted on four pre-trained encoders (BETO, mBERT, XLM-R for Spanish; CamemBERT, mBERT, XLM-R for French) using final-layer embeddings.

Estimator Performance: The Centroid estimator consistently outperformed discriminative baselines (LinearSVM and LDA) across all metrics. It achieved the highest harmonic trade-off score (0.870), suggesting that grammatical gender in contextual embeddings manifests more as a global distributional shift than a complex classification boundary.
Context Source Impact: Controlled, unweighted contexts yielded the purest grammatical gender directions.
- Controlled-unweighted settings achieved the highest reduction in grammatical leakage ( $R_{noun} \approx 0.98$ ) while maintaining strong semantic preservation.
- Natural contexts introduced variability that weakened the clarity of the estimated signal.
- Weighting strategies (mean/median) provided only marginal improvements in specific cases and often reduced the noun-level reduction performance, acting more as a conservative adjustment than a primary performance driver.
Debiasing Efficacy: The orthogonal projection method successfully removed grammatical gender components from inanimate nouns without disrupting the general lexical semantic structure, as validated by semantic stability tests.

4. Contributions

The paper claims the following primary contributions:

Pioneering Contextual Disentanglement: It is the first work to attempt grammatical-semantic gender disentanglement specifically for contextual language models, moving beyond the static embeddings used in prior literature.
Formalization of Semantic Contamination: The authors formally define and quantify semantic contamination in inanimate noun representations for contextual models, introducing contamination-aware weighting strategies to suppress biased distributional artifacts.
Novel Evaluation Framework: The paper proposes a dual-objective evaluation framework (Leakage Reduction + Semantic Preservation) with a harmonic trade-off metric, addressing the lack of balanced standards for debiasing in grammatically gendered languages.

5. Significance and Limitations

Significance:
The study demonstrates that controlled templates are superior to natural corpora for isolating grammatical gender signals, effectively eliminating corpus noise. It establishes that simple centroid-based aggregation is more robust than discriminative classifiers for extracting these directions in high-dimensional, noisy contextual spaces. This provides a reliable foundation for multilingual gender bias research by separating formal grammatical rules from inherent semantic bias.

Limitations and Future Directions:

Typological Scope: The findings are restricted to Spanish and French (Romance languages with binary gender systems). The authors note it is unclear if these linear disentanglement behaviors generalize to languages with case-gender interactions, neuter systems, or non-binary gender structures.
Model Specificity: Results are based on frozen final-layer embeddings of specific transformer architectures. The behavior of disentanglement across different layers or model architectures remains an open question.
Weighting Strategies: The current mean/median weighting schemes are acknowledged as suboptimal. The authors suggest future work should explore adaptive paradigms, such as adversarial weighting or causal interventions, to better model semantic contamination.

The authors conclude that while weighting strategies offer minor adjustments, the most effective approach for estimating grammatical gender directions remains the use of unweighted controlled contexts combined with centroid-based estimation.