Technical Summary: Topics as Proxies for Sociodemographics

Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes scenarios, including legal, medical, and financial advice. Prior research indicates that users from different sociodemographic groups receive disparate outcomes from LLMs, even when their conversation histories contain no explicit demographic markers. While these disparities are statistically significant, their exact magnitude and underlying mechanisms remain unclear. Specifically, it is unknown whether models drive these differences by accurately inferring user sociodemographics from conversational context or by reacting to other linguistic features that happen to correlate with those demographics. This paper investigates whether LLMs can infer user sociodemographics from a single conversation history and identifies the primary drivers of outcome disparities in high-stakes advice generation.

Methodology

The study employs a multi-faceted experimental design using three open-weight LLMs (Llama 3.1 8B, Gemma 3 12B, and Qwen3.6 27B) and a frontier reasoning model (Kimi K2.6).

Datasets:

• Conversational Histories: Two datasets were utilized: the PRISM dataset (8,011 conversations with 1,500 participants from 75 countries) and the Community Alignment Dataset (26,521 English conversations from 1,220 participants in India and the US). Both include user sociodemographic annotations (e.g., age, gender, ethnicity, education, region).
• High-Stakes Questions: The Sociolinguistic Bias Benchmark (SBB) provided 250 questions across five domains: Benefits, Medical, Legal, Political, and Salary. These questions were appended to the conversational histories to generate model responses.

Experimental Procedures:

1. Outcome Disparity Measurement: The authors measured the magnitude of differences in model outputs (e.g., salary recommendations, binary yes/no advice) across sociodemographic groups using ANOVA tests.
2. Demographic Inference Evaluation:
   • Prompting: Kimi K2.6 was explicitly prompted to infer user demographics from conversation histories in JSON format.
   • Linear Probing: Linear classifiers were trained on the latent representations (last token positions) of Gemma and Llama to predict user demographics, testing the internal linear separability of these features.
3. Predictive Factor Analysis: The authors computed a wide range of (psycho)linguistic features for each conversation, including:
   • Conversation Topics: Derived from opening prompts or clustering.
   • Psycholinguistic Features: LIWC-22 dictionary metrics, emotions (BERT-based), sentiment (RoBERTa-based), politeness, concreteness, and readability (Flesch score).
   • Linguistic Metrics: Perplexity, token counts, and diversity metrics.
   • Regression Modeling: ElasticNet regression was used to determine which features (sociodemographics vs. linguistic/contextual) best predicted model outcomes.
4. Controlled Comparison: A 2x2 design compared model outcome differences between users of the same/different demographic groups discussing the same/different topics to isolate the effect of topic versus identity.

Key Contributions and Results

1. Minimal Magnitude of Disparities
While statistically significant differences in outcomes exist between sociodemographic groups, their magnitude is minimal. Across all models and datasets, significant salary differences between the most distinct groups averaged $342.20 (max $882.70). For other domains, differences ranged from 0.66 to 2.22 percentage points. This corresponds to fewer than one or at most two questions out of 50 being answered differently between the most distinct groups. These differences are substantially smaller than those observed relative to the model's baseline prediction without any conversational context.

2. Poor Demographic Inference Capabilities
LLMs struggle to accurately infer user sociodemographics from conversation histories:

• Prompting: When explicitly asked to infer demographics, Kimi K2.6 failed to outperform majority-class baselines in five of seven categories, largely due to over-predicting majority classes (e.g., predicting "Male," "White," "18-34 years old"). It never predicted users aged 55+.
• Linear Probing: While probes trained on latent representations outperformed random and majority baselines in later model layers, performance remained low (Macro F1 scores around 0.3 for unbalanced classes and 0.7 for balanced classes). This suggests sociodemographics are not clearly linearly represented in the model's internals.

3. Conversation Topics as Primary Drivers
Regression analysis revealed that conversation topics are the strongest predictors of LLM behavior, far outweighing user sociodemographics.

• Topic features consistently held the highest importance coefficients across all models and domains.
• Specific topics correlated with specific outcomes: e.g., conversations about "luxury travel" led to higher salary recommendations, while "job search" or "budget constraints" led to lower ones.
• In the 2x2 comparison, differences in outcomes between users discussing different topics were consistently higher than differences between users of different demographic groups discussing the same topic.

4. Topics as Proxies
The study finds that conversation topics act as proxies for sociodemographics. Since users from different backgrounds often select different topics (e.g., younger users discussing job searches more frequently), the model's reaction to the topic inadvertently creates demographic disparities. However, the model is reacting to the topic, not the inferred identity.

Significance and Claims

The paper claims that the primary driver of demographic bias in high-stakes LLM advice is not the model's ability to infer and discriminate based on user identity, but rather the model's sensitivity to conversational context, specifically topic.

• Subtlety of Bias: The findings highlight that demographic disparities in high-stakes scenarios arise indirectly and subtly through topic selection rather than explicit demographic signaling. This makes such biases difficult to detect and mitigate.
• Limitations of Current Mitigation: A simple prompt-based mitigation strategy (instructing the model to ignore user characteristics) was found to be largely ineffective, reducing disparities only in the political domain while leaving benefits and salary domains largely unchanged.
• Future Direction: The authors argue for a shift in understanding user identity in LLMs, moving beyond static sociodemographics to the complex interplay of conversational context. They call for future research to better understand and mitigate the effects of conversational context, particularly the role of topics as proxies for identity, in high-stakes applications.

The paper concludes that while outcome disparities exist, they are driven more by what users talk about than who they are, challenging the assumption that LLMs are directly inferring and acting upon demographic identities in realistic conversational settings.