💬 NLP

Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias

This paper reveals that language models often retain internal representational biases regarding user competence based on demographics like gender and race, even when their observable outputs appear unbiased, demonstrating that behavioral evaluations alone are insufficient to detect these underlying failure modes.

Original authors: Keren Fuentes, Aaron Mueller

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Keren Fuentes, Aaron Mueller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human beings are capable of holding subconscious assumptions about others, often without realizing it. A person might believe that a specific demographic group is less suited for a certain job, even if they consciously reject such ideas. These hidden associations can quietly shape decisions, from who gets hired to how much help someone receives. In the world of artificial intelligence, large language models have learned to mimic human speech by studying vast amounts of text. Because they learn from human writing, they often pick up these same hidden associations. For years, researchers have tried to measure whether these computer programs are biased by asking them direct questions or observing their answers. If a model refuses to say something prejudiced, it is often considered safe. However, a new study suggests that a model can appear perfectly polite on the surface while still holding onto these biased ideas deep inside its own processing.

The researchers behind this work, Keren Fuentes and Aaron Mueller, wanted to see if a model's internal thoughts matched its outward behavior. They focused on a specific type of bias: the assumption that certain people are more or less capable based on their gender, race, or socioeconomic status, rather than their actual skills. To investigate this, they looked at two different ways to judge a model. The first is the "behavioral" view, which is what the model actually says when asked a question. The second is the "representational" view, which looks at the mathematical patterns the model creates inside its own brain while it is thinking. Think of the model's internal state like a hidden layer of thought that exists before it forms a sentence. The researchers developed a way to measure this hidden layer to see if the model was treating a user as an expert or a novice, regardless of what the user actually said about their experience.

To do this, the team created a set of professional questions covering twenty different jobs, ranging from nursing to software development. They asked these questions to three different large language models, but they changed the context slightly in each case. Sometimes they told the model the user was a professional in that field; other times they mentioned the user's gender, race, or income level without mentioning their job. They then measured two things: the complexity of the language the model used in its answer, and the internal "score" it gave to the user's expertise. The complexity of the language served as a behavioral metric, while the internal score acted as a window into the model's hidden reasoning.

The results revealed a striking disconnect. In many cases, the models produced answers that looked fair and unbiased. When asked to hire a candidate or answer a technical question, the models did not consistently give different answers based on the user's race or gender. If you only looked at the final text, the models seemed to treat everyone equally. However, when the researchers peeked inside the model's internal processing, they found a different story. The models were assigning different levels of expertise to users based on demographic details that should not matter. For instance, a model might internally rate a white user as more competent than a Hispanic user, even when both users asked the exact same question and had the same job title. This internal bias existed even when the model's final words did not show any prejudice.

The researchers proved that these internal scores were not just random noise; they actually controlled the model's behavior. By using a technique called "steering," they could nudge the model's internal thoughts to make it view a user as more or less expert. When they did this, the model's answers changed. If they nudged the model to see a user as an expert, the model used more complex language and gave more detailed technical advice. If they nudged it to see the user as a novice, the answers became simpler. This confirmed that the internal representation of expertise was a real, causal force driving the model's output. The study showed that these internal biases were sensitive to demographic cues. Even when the model knew the user's profession, subtle differences in how it perceived competence based on race or gender remained detectable in its internal state, even if they did not always show up in the final text.

This finding suggests that the current methods for testing artificial intelligence for bias might be missing a lot. If we only check what a model says, we might think it is fair when it is not. The models studied here, which included versions of Gemma and Llama, showed that they could encode associations between demographic groups and competence in their hidden layers. These associations could influence the model's decisions in ways that are not immediately visible. For example, in a hiring task, the models' internal scores for candidates varied by race and gender, even though the final hiring decisions appeared statistically similar across groups. The researchers found that while the models could be steered to change their hiring rates, the underlying internal biases persisted, suggesting that the models were still processing demographic information in a way that influenced their judgment.

The implications of this work are significant for how we evaluate and improve artificial intelligence. It indicates that a model can pass a standard safety test while still harboring deep-seated biases that could be triggered by specific prompts or conditions. The researchers argue that we need to look beyond the surface level of what a model says and examine how it thinks. By measuring these internal representations, we might be able to detect and fix biases before they ever manifest in harmful behavior. This approach offers a new way to ensure that these powerful tools are truly fair, not just in their words, but in their very nature. The study concludes that while behavioral metrics are necessary, they are not enough on their own; a robust understanding of fairness requires looking at the hidden machinery of the model itself.

Technical Summary: Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias

1. Problem Statement

While large language models (LMs) often pass behavioral bias evaluations, it remains unclear whether they have genuinely unlearned underlying demographic associations or merely learned to suppress their expression. Existing research indicates that models can encode associations between demographic features (e.g., gender, race, socioeconomic status) and social roles in their latent representations even when their outputs appear benign.

Current evaluation methods primarily rely on behavioral metrics (observable outputs), which may fail to detect latent biases encoded in the model's internal reasoning. This study posits that demographic attributes, which are causally irrelevant to a person's actual competence, can still influence a model's internal representation of that user's expertise. The core problem is the potential disconnect between a model's "thought process" (internal representations) and its "speech" (observable outputs), where behavioral metrics suggest fairness while internal mechanisms remain biased.

2. Methodology

2.1 Causal Framework

The authors propose a causal framework to decompose occupational bias into two distinct measurement points:

Internal Representation ( $E$ ): The model's latent representation of a user's competence.
Observable Behavior ( $L$ ): The model's generated output, specifically measured by linguistic complexity (reading level).

The framework defines bias as the causal influence of irrelevant demographic attributes (gender, race, socioeconomic status) on the internal representation of competence. The study utilizes a causal graph where profession, education, and age are treated as causally relevant factors, while race, gender, and socioeconomic status are treated as causally irrelevant.

2.2 Data Construction

Professional Questions Dataset ( $D_P$ ): A dataset of 2,000 career-specific questions across 20 professions (selected from the U.S. Bureau of Labor Statistics). Questions were generated using GPT-5 to span varying levels of expertise (apprentice to veteran).
Hiring Task: A modified version of a standard hiring evaluation where models are presented with resumes and asked to make a Yes/No hiring decision. Resumes were modified to encode gender and race via names.

2.3 Steering Vector Derivation

To quantify the model's representation of expertise, the authors construct a steering vector ( $e$ ) using a difference-in-means approach:

Expert Set ( $R^+$ ): Prompts indicating high competence (e.g., "I've studied this topic in depth for years").
Novice Set ( $R^-$ ): Prompts indicating low competence (e.g., "I'm just starting to learn about this topic").
Vector Calculation: The expertise vector is the difference between the mean hidden representations of the expert and novice sets:
$e = \frac{1}{|R^+|} \sum_{h \in R^+} h - \frac{1}{|R^-|} \sum_{h \in R^-} h$
Expertise Score ( $E$ ): The scalar projection of the context's final token representation onto the unit expertise vector.

2.4 Behavioral Metrics

To measure observable bias, the authors use an ensemble of reading level metrics:

Flesch–Kincaid Grade Level (FKGL): Estimates school grade required to understand text.
Dale–Chall Readability Score (DCRS): Measures difficulty based on advanced vocabulary.
Composite Score ( $L$ ): The average of FKGL and DCRS.

2.5 Experimental Setup

Experiments were conducted on three instruction-tuned open-weight models: Gemma-2-2B, Gemma-2-9B, and Llama-3-8B.

Intervention: The authors apply steering by adding the expertise vector ( $e$ ) to the hidden representations at specific middle layers ( $\tilde{h}_l = h_l + \alpha e$ ) to verify causal mediation.
Prompt Variations: Prompts were constructed to isolate demographic variables (e.g., "I'm a [Demographic]") versus causal variables (e.g., "I'm a [Profession]").

3. Key Contributions

Causal Framework for Occupational Bias: A novel method for assessing bias by separating internal representations of user capability from observable model behaviors.
Steering Vector Operationalization: The derivation of specific steering vectors for "user expertise" in a given domain. The authors demonstrate that intervening on these vectors causally affects model behavior in both question-answering and hiring tasks.
Detection of Latent Bias: Empirical evidence showing that demographic attributes influence internal expertise representations even when behavioral metrics (like reading level or hiring rates) show no disparity.

4. Key Results

4.1 Causal Mediation of Expertise

Steering experiments confirmed that the expertise vector ( $e$ ) causally mediates model behavior.

Reading Level: Increasing the expertise score via positive steering ( $\alpha > 0$ ) significantly increased the reading level ( $L$ ) of model outputs. Negative steering decreased complexity. This held across Gemma-2-9B and Llama-3-8B.
Hiring Decisions: Steering with the expertise vector significantly altered hiring rates. A hiring-specific vector ( $e_H$ ) was also derived, which showed that models are highly sensitive to expertise representations when making hiring decisions.

4.2 Sensitivity to User Context

Relevance: The model's internal expertise score ( $E$ ) correctly increased when the user context included a relevant profession (causal factor) compared to an irrelevant one.
Demographic Influence: When holding all else constant, changing only the demographic information (e.g., race, gender, income) significantly altered the magnitude of the expertise representation ( $E$ $E$ ).
- Example: Gemma-2-2B assigned higher expertise scores to White and Black demographics compared to Hispanic and Asian. Gemma-2-9B showed higher scores for low-income, Hispanic, and Black demographics.

4.3 Divergence Between Representational and Behavioral Bias

The study found a critical divergence between internal representations and observable outputs:

Latent vs. Observable: While $E$ (expertise score) showed systematic differences across demographic groups (including causally irrelevant factors like race and gender), $L$ (reading level) often did not.
Underestimation by Behavioral Metrics: Behavioral evaluations failed to detect disparities that were clearly present in the latent space. For instance, race and gender effects were latent in $E$ but largely invisible in $L$ .
Socioeconomic Status Exception: Socioeconomic status was an exception where biases were observable in both $E$ and $L$ , suggesting that some biases manifest behaviorally while others remain latent.

4.4 Hiring Task Findings

In the hiring task, while hiring rates ($Yes/No$) did not show significant demographic disparities (behavioral fairness), the internal expertise scores ( $E$ ) of accepted versus rejected candidates varied by demographic group. This indicates that the model's decision-making process relies on biased internal representations even if the final output appears neutral.

5. Significance and Claims

The paper argues that behavioral measures of fairness are necessary but insufficient for evaluating language models. The primary significance of this work lies in demonstrating that:

Latent biases exist independently of behavioral outputs: Models can encode demographic biases in their internal reasoning mechanisms even when post-training methods have successfully suppressed these biases in the final text generation.
Representational metrics reveal hidden failure modes: Relying solely on output-based evaluations (like reading level or hiring rates) may lead to a false sense of security regarding model fairness.
Mechanistic intervention is possible: The ability to derive and steer expertise vectors proves that these biases are mechanistically grounded in specific directions within the model's activation space, offering a potential pathway for detection and mitigation before biases surface in user-facing outputs.

The authors conclude that robust evaluation frameworks must integrate representational evaluations alongside behavioral ones to fully understand and address the mechanisms of bias in LMs. They caution that while these methods aid in fairness research, they also carry the risk of being exploited to manipulate model behavior, and thus, tools for malicious use are not released.