Technical Summary: DAIQ – Auditing Demographic Attribute Inference from Neutral Questions in LLMs

1. Problem Definition

Current evaluations of Large Language Models (LLMs) regarding social bias primarily rely on prompts containing explicit demographic signals (e.g., names, pronouns, dialects). This approach overlooks a critical failure mode: whether models infer sensitive demographic attributes from demographically neutral questions that contain no explicit cues.

The paper argues that such inference constitutes epistemic overreach. When models attribute user identity based on topic, tone, or phrasing without evidence, they rely on learned population priors and social heuristics rather than user-provided information. This behavior raises significant concerns for privacy and fairness, as inferred demographics can silently condition downstream responses, leading to "silent personalization" driven by speculation rather than intent.

2. Methodology: The DAIQ Framework

The authors introduce Demographic Attribute Inference from Questions (DAIQ), a diagnostic audit framework designed to measure demographic inference under epistemic uncertainty.

Task Definition

In the DAIQ task, an LLM is presented with a natural language question containing no explicit demographic information. The model is explicitly prompted to reason about and infer the likely demographic attributes of the question's author, providing both a prediction and a justification.

• Normative Behavior: Since the input lacks evidence, the correct response is abstention (e.g., "unknown" or "cannot be determined").
• Failure Mode: Any specific demographic attribution reflects a reliance on spurious correlations learned during pre-training.

Dataset and Scope

• Source: The study utilizes the AccessEval benchmark, specifically filtering for 212 Neutral Queries (NQs) across six real-world domains: Education, Finance, Healthcare, Hospitality, Media, and Technology.
• Attributes: Five demographic attributes are evaluated:
  1. Gender (Male/Female)
  2. Race/Ethnicity (Black/White)
  3. Socioeconomic Status (Low/High)
  4. Geographic Location (Urban/Rural)
  5. Educational Background (Low/High)
• Models: The framework evaluates 18 instruction-tuned LLMs, including open-source families (Phi, LLaMA, Qwen, InternLM, Cohere Aya, Mistral) and closed-source models (Anthropic Claude, OpenAI GPT, xAI Grok).
• Metrics:
  • Response Rate (RR): The proportion of queries where the model assigns a specific demographic value rather than abstaining. Ideally, RR should be zero.
  • Value-Specific RR: Analyzes asymmetry in attribution (e.g., bias toward "Male" over "Female").

3. Key Results

3.1 Pervasive Unintended Inference

The study finds that many models infer demographics from neutral questions at high rates.

• Proprietary Models: Models like GPT-4.1, Claude-3.5 Haiku, and Grok-4 exhibited response rates near 100%, inferring demographics for almost all queries.
• Open-Source Models: Susceptibility varied; for instance, Qwen2.5-3B and Llama-3.3-70B showed high inference rates, while aligned models like Phi-4-mini showed lower rates for gender/race but still inferred socioeconomic and educational attributes.
• Scale vs. Alignment: Response rates did not monotonically increase with model size. Demographic caution appeared driven more by alignment choices and training data than by scale alone.

3.2 Directional Defaults and Stereotypes

Inferred demographics were not random; they exhibited strong directional defaults toward socially dominant categories:

• Gender: A pronounced skew toward Male attribution.
• Race: Overwhelming default to White; Black inference was rare.
• Socioeconomic Status: High status was inferred far more often than low status.
• Geography: Nearly exclusive inference of Urban locations.
• Education: High educational attainment was inferred in over 95% of queries for some models (e.g., Qwen2.5-72B), often exceeding rates for gender or race.

3.3 Qualitative Analysis of Reasoning

The paper analyzed the rationales provided by models, revealing:

• Gendered Rationales: Female attributions were anchored in caregiving, empathy, and planning, while male attributions were linked to technical authority and analytical language.
• Asymmetric Justification: Male predictions were often left unexplained or justified by "neutral assumptions," whereas female predictions required explicit social or occupational rationales.
• Meta-Awareness Failure: Several models explicitly flagged their reasoning as speculative or biased yet proceeded to assign demographics regardless, indicating that awareness alone does not prevent stereotype-driven inference.

3.4 Consequential Impact (Silent Personalization)

The study demonstrated that inferred demographics act as latent conditioning variables.

• Using semantic similarity analysis, the authors showed that model responses conditioned on inferred gender were systematically closer to the model's own inferred gender than to alternative values.
• This confirms that demographic inference alters the tone, framing, and content of responses, effectively enabling silent personalization without user input.

4. Mitigation Strategy

The authors tested an abstention-oriented prompting strategy.

• Result: This approach substantially reduced unintended demographic inference across models without requiring fine-tuning.
• Residual Effects: While gender and race inferences were nearly eliminated, residual inference persisted for educational background (specifically high education), suggesting this attribute is perceived as more indirectly inferable or less constrained by the prompt guardrail.

5. Significance and Claims

The paper claims that current bias evaluations are incomplete because they presuppose the presence of demographic cues. The significance of DAIQ lies in:

1. Redefining the Audit Scope: It shifts the evaluation standard from "how models respond to demographic information" to "whether models should infer it at all."
2. Identifying Epistemic Overreach: It highlights that inferring sensitive attributes from neutral inputs is a distinct failure mode that reinforces disparities and privacy risks.
3. Demonstrating Latent Conditioning: It provides empirical evidence that inferred demographics actively shape downstream generations, creating silent personalization.
4. Practical Mitigation: It shows that simple prompt engineering (abstention-oriented prompts) can serve as an effective, model-agnostic guardrail to reduce these inferences.

The authors conclude that abstention under uncertainty must be treated as a first-class evaluation criterion for LLMs deployed in high-stakes domains.