💬 NLP

Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model

This paper demonstrates that in large language models, the properties of demographic identity being readable, faithfully structured, and causally used are three dissociable phenomena, revealing that while specific attention heads can encode survey-like group differences with high fidelity, this encoding does not necessarily translate into the model's actual causal usage for generating responses.

Original authors: Fathin Difa Robbani

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fathin Difa Robbani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are increasingly being asked to act as stand-ins for human beings in social science research. Instead of spending months and millions of dollars interviewing thousands of real people to understand how different groups feel about politics, religion, or economics, researchers can simply ask a computer program to simulate those responses. The hope is that these digital surrogates can capture the subtle differences between a young Democrat and an older Republican, or between a Hindu and a Christian, just as a real survey would. But a growing body of evidence suggests these simulations are failing. The computer's answers tend to be too uniform, too agreeable, and too similar to one another, regardless of the demographic label attached to the prompt. The central mystery has been whether this failure happens because the model simply does not know the differences between these groups, or because it knows the differences but chooses not to use them when it speaks.

To solve this, a researcher set out to look inside the "brain" of a large language model to see where demographic information actually lives. The investigation focused on a specific model, Mistral-7B, and asked three distinct questions. First, is the information about who is answering readable from the model's internal signals? Second, is that information arranged in a way that mirrors the real world, where the distance between two groups in the computer's mind matches the distance between them in reality? And third, does the model actually use this internal map when it generates an answer? The study treated these three questions as separate properties, realizing that a model could possess a faithful internal map without ever consulting it to form an opinion.

The researchers began by mapping the model's internal landscape. They created prompts for 169 different intersectional groups, such as young Asian Democrats or middle-aged Hindu Republicans, and examined the model's internal state at the very end of its processing. They compared the model's internal arrangement of these groups against real survey data from the Pew Research Center, which provided the ground truth of how these groups actually differ in their opinions. The results revealed that the standard way of reading the model's mind—looking at the final output layer—was misleadingly weak. It suggested the model had a blurry, indistinct view of demographic differences. However, when the researchers looked deeper, at the specific components that make up the model's processing, they found a much clearer picture.

The information was not lost; it was just hidden in specific, narrow channels. The study discovered that the model's internal geometry is dominated by individual attention heads, which are specialized sub-processors within the model. In five out of six types of demographic attributes tested, the best single attention head provided a much more accurate map of group differences than the model's entire final output layer. One specific attention head, located in the 11th layer of the model, stood out as remarkably faithful. It held a representation of all six demographic types—age, education, income, religion, political party, and political ideology—that closely matched the real-world structure of how these groups differ. This single component carried a map where the relationships between groups were arranged with a fidelity that reached roughly 70% of the theoretical limit of what could be measured, even after accounting for the fact that the survey data itself contains some noise.

Yet, finding a faithful map inside the model did not mean the model was using it. The researchers then tested whether this accurate internal arrangement actually influenced the model's answers. They performed a causal experiment where they swapped the demographic identity in the prompt and watched how the model's predicted opinions shifted. The result was stark: changing the entire identity of the person in the prompt moved the model's predicted answers by less than 2% of their total error. The model produced slightly different, but equally wrong, answers for different groups. Even more surprisingly, the researchers found that the location with the most faithful map was not the location that drove the answers. The clearest causal pathway, where changing the internal state actually moved the model's prediction toward the truth, was found in a demographic type that had one of the weakest internal maps. Conversely, the most faithfully arranged type showed no detectable causal effect when the researchers tried to intervene on it.

This disconnect suggests that the failure of these models to simulate populations is not simply a lack of knowledge. The model does hold a detailed, accurate map of how different groups differ, particularly regarding political and economic factors, while its understanding of race and religion remains weak and unstable. However, the model does not consult this map when it speaks. The internal geometry is faithfully arranged but causally disconnected from the final output. The study also found that simply reading this internal map directly, using a mathematical probe on that single faithful attention head, produced results that were 21% to 31% closer to real survey truth than the model's own generated answers. However, even this superior read-out could not predict the specific order of opinions for individual questions any better than the model's own flawed answers.

The implications of these findings are significant for anyone trying to use artificial intelligence to understand human society. The research demonstrates that the ability to simulate a population cannot be assumed just because a model can be prompted to act like a specific person. The model possesses the knowledge, but it is not using it in the way researchers hope. Furthermore, the study warns that demographic attributes are not interchangeable; while the model has a strong grasp of political and economic identities, its understanding of racial and religious identities is fragile and prone to breaking under slight changes in how a question is asked. The work concludes that treating the model's internal knowledge and its external behavior as the same thing is what keeps the debate over whether these models can simulate populations unresolved. The model has the map, but it is not following the route.

Technical Summary: Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model

Problem Statement

Large Language Models (LLMs) are increasingly deployed as simulated survey respondents ("silicon sampling") to estimate population opinion distributions. However, behavioral evidence indicates these outputs are excessively homogeneous and unfaithful to real inter-group differences. A critical gap exists in the literature: interpretability research localizes demographic information within models but validates only against the model's own behavior (first-order), while simulation research uses external ground truth (real survey distributions) but treats the model as a black box. Consequently, it remains unknown whether the failure of LLMs to simulate populations stems from a lack of internal knowledge (the map is missing) or a failure to utilize that knowledge (the map exists but is not used).

Methodology

The author bridges this gap by holding two instruments simultaneously: internal interpretability tools and external population ground truth.

Ground Truth Construction: Using 15 waves of the Pew American Trends Panel (ATP), the author constructed a Representational Dissimilarity Matrix (RDM) for 169 intersectional demographic cells (e.g., Asian × Hindu, Age 18–29 × Democrat) across six attribute types. The distance metric is the mean Wasserstein distance between real response distributions for shared questions.
Internal Representation Mapping: The study analyzes Mistral-7B at 1,089 locations per prompt variant: the residual stream after every layer, every attention head output (32 layers × 32 heads), and every FFN block output.
Fidelity Measurement (RSA): The core metric is the Spearman correlation between the model's internal RDM at a specific location and the external survey RDM. This "second-order" comparison tests whether the geometry of group representations mirrors real inter-group opinion structures, rather than just decoding individual group opinions.
Causal Intervention: To determine if the model uses these representations, the author employs activation patching (causal tracing). They intervene at specific layers and heads, swapping activations between demographic personas to measure the causal effect on the model's predicted opinion distribution.
Statistical Rigor: The study employs strict corrections for selection bias (winner's curse) using max-statistic permutation tests, held-out template validation, and split-half validation. Causal inference uses cluster-robust statistics (sign-flip permutations over 12 donor-recipient pairs) to account for item-level dependencies.

Key Contributions

First Fidelity Map: The first systematic map of demographic identity in an LLM scored against real survey ground truth across layers, heads, and prompt variants.
Identification of a General Geometry Head: The discovery of a single attention head (L11 H16 in Mistral-7B) that is significantly faithful to real opinion structures across all six attribute types as a fixed location, without per-type re-selection.
Quantification of the Knowledge-Behavior Gap: Evidence that replacing the entire demographic identity in a prompt moves the model's predicted distribution by less than 2% of its total error, and that this movement is often directionally uninformative.
Dissociation of Properties: The demonstration that "Readable" (decodable), "Faithfully Arranged" (geometrically accurate), and "Causally Used" (driving the output) are three distinct properties that do not necessarily co-occur.
Cross-Model Replication: Replication of the fidelity map phenomenon (including the specific weakness of race-based attributes) across three checkpoints of a second model family (Qwen3-8B/OSim), showing that 10 billion tokens of human-simulation training barely alter the internal map.

Key Results

1. The Standard Read-Out Understates the Model

The standard "last-token residual stream" read-out yields weak fidelity ( $\rho \approx 0.17$ pooled) and is highly anisotropic. However, attention-head read-outs dominate the residual stream in five of six attribute types, with selection-corrected fidelity reaching up to $\rho = 0.63$ (roughly 70% of the measurement-reliability ceiling). This high-fidelity structure survives partialling out lexical similarity, confirming it is not merely an artifact of word embeddings.

2. A Single Head Encodes General Demographic Geometry

Head L11 H16 (Layer 11, Head 16) is a fixed location that is significantly faithful across all six attribute types (political, socio-economic, and racial/religious).

Heterogeneity: While political and socio-economic structures are deeply encoded, race-based types (especially Race × Religion) are consistently weak and prompt-fragile across all locations.
Stability: This phenomenon replicates in a second model family at a family-specific address (L33 H9), suggesting it is a structural property of the architecture rather than a training artifact.

3. Causal Use Does Not Follow Fidelity

Despite the existence of faithful internal maps, the model barely uses this information to generate answers:

Identity Invariance: Swapping the full demographic identity in the prompt shifts the model's output by less than 2% of its prediction error.
Dissociation: The type with the clearest causal pathway (Race × Political Ideology) is one of the least faithful types. Conversely, the most faithful types (e.g., Age × Party) show no correction-surviving single-layer causal effect.
Intervention Risks: Direct steering through a single "faithful" head (L11 H16) can actively push predictions away from the truth for low-fidelity types (e.g., Race × Religion), indicating that faithfulness does not guarantee utility as a steering lever.

4. Direct Read-Out vs. Model Output

A linear probe on the single faithful head (L11 H16, 128 dimensions) recovers group distributions 21–31% closer to survey truth than the model's own generated answers. However, this probe recovers almost none of the per-question ordering of groups (correlation near zero), performing no better than the model's own answers in ranking specific items. Thus, while the internal map is more accurate on average, it lacks the resolution for item-level simulation.

Significance and Claims

The paper argues that the debate "Can LLMs simulate populations?" is unresolved because it conflates three dissociable properties:

Readable: Information is present and decodable.
Faithfully Arranged: The internal geometry mirrors real-world opinion structures (specifically at attention heads).
Causally Used: The model actually utilizes this geometry to generate answers.

The author claims that faithful structure exists internally (property 2), but the model fails to use it (property 3) to generate diverse outputs. This suggests that the "Sim2Real" gap is not solely due to a lack of representation quality (as the map exists) but also due to a failure in causal throughput.

Practical Implications:

Fine-tuning limitations: Surface fine-tuning (e.g., SubPOP) may be "re-teaching" what the model already encodes, explaining transfer failures to unseen surveys.
Steering risks: Treating a faithful location as a steering lever is dangerous; the most consistent causal effect found was a harmful drift away from truth in low-fidelity types.
Simulation limits: Current LLMs under natural prompting are not reliable substitutes for human respondents, particularly for race-based attributes where internal structure is weakest and most fragile.
Read-out utility: While probing internal states yields better aggregate distributions than asking the model, it is not a substitute for survey data due to a lack of item-level discrimination.

The study concludes that the failure of LLMs to simulate populations is a dual mechanism: insufficient causal throughput where the map is good, and a lack of a good map where the structure is weak (particularly for race).