Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias

1. Problem Statement

Current research on Large Language Model (LLM) personalization and bias relies heavily on demographic cue-based evaluation. Researchers typically use a single signal (e.g., a specific name, a dialect, or an explicit label) to represent a demographic group (e.g., "Black" or "Female") and assume these cues are interchangeable operationalizations of the same underlying identity-conditioned behavior.

The paper challenges this core assumption, asking:

1. Do different cues intended to signal the same demographic group produce consistent within-group model behaviors?
2. Do different cues yield consistent conclusions about inter-group disparities (bias)?

The authors argue that treating cues as interchangeable may lead to unstable, contradictory, or misleading conclusions about how LLMs adapt to user identity or exhibit bias.

2. Methodology

The study employs a large-scale, controlled experimental design involving 14.8 million prompts across three high-stakes domains: Healthcare, Salary, and Legal advice.

• Models Evaluated: Three distinct LLMs were tested:
  • LLaMA-3.1 8B (Open-weights)
  • OLMo2-7B (Fully open-source)
  • GPT-5.2 (Frontier model via API)
• Demographic Attributes: Race (Black vs. White) and Gender (Male vs. Female) within a U.S. context.
• Cue Types (Operationalizations): The study compares four distinct methods of signaling identity:
  1. Names: Implicit signals using first names associated with specific race/gender groups (drawn from three distinct datasets: Elder & Hayes, Rosenman et al., Tzioumis).
  2. Dialect: Implicit signals via translation of prompts into African American Vernacular English (AAVE).
  3. Dialog History: Implicit signals via prepending conversation histories from specific demographic groups (using CAD and PRISM datasets).
  4. Explicit Attributes: Direct third-person labels (e.g., "[MEMORY: User is a Black male]").
• Experimental Design:
  • Prompts were constrained to elicit binary (Yes/No) or numeric (Salary) responses to ensure comparability.
  • Counterfactual Analysis: For every base prompt, the authors generated variants using different cues to measure the deviation from a no-cue baseline.
  • Mechanism Analysis: The authors investigated why cues differ by measuring:
    • Cue-Group Association Strength: How strongly the model infers the demographic category from the cue (via a forced-choice race prediction task).
    • Bundled Linguistic Features: How the cue alters structural properties (e.g., sentence length, complexity) using Flesch-Kincaid grade levels.

3. Key Contributions

1. Partial Cross-Cue Convergence: The study demonstrates that signals for the same demographic group induce only moderately correlated behavioral shifts. While cues of the same type (e.g., different name lists) are highly consistent, different types of cues (e.g., Name vs. Dialect) yield divergent behavioral patterns.
2. Cue-Dependent Disparities: Estimates of bias (inter-group differences) are unstable. Both the magnitude and direction of group differences vary significantly depending on the cue used. For instance, a model might show a bias favoring White users when cued by names but show no bias or even a reverse bias when cued by dialect.
3. Mechanistic Explanation: The authors identify two primary drivers of this variation:
   • Association Strength: Cues vary in how reliably they trigger the model's internal demographic inference (e.g., Explicit labels trigger near-perfect inference; Dialog history triggers very weak inference).
   • Linguistic Bundling: Cues introduce confounding linguistic features (e.g., Dialect changes syntax/lexicon; Dialog history increases prompt length). These features independently influence model outputs, decoupling the "demographic effect" from the "linguistic effect."

4. Key Results

• Weak Group Differentiation: The choice of how identity is cued has a larger effect on model behavior than which group is cued. Within a single cue type, Black and White prompts often elicit nearly identical responses ($r \approx 0.98$), whereas the same group prompted with different cues shows much lower similarity ($r \approx 0.66$).
• Inconsistent Bias Conclusions:
  • Name-based cues and Dialog history often show minimal inter-group disparity.
  • Dialect and Explicit cues frequently show significant disparities, sometimes in opposite directions depending on the model and task.
  • Example: In legal advice tasks, GPT-5.2 suggested more positive outcomes for Black prompts when cued by explicit labels, but the opposite when cued by other methods.
• Regression Findings:
  • Inferred Race (what the model thinks the user is) is a stronger predictor of behavior than the Actual Cued Race.
  • Readability (Flesch-Kincaid grade level) is a robust, independent predictor of model responses. Cues that alter readability (like Dialect or Dialog History) significantly shift outcomes, suggesting models respond to linguistic signals rather than stable demographic categories.

5. Significance and Recommendations

Significance:
The paper fundamentally shifts the understanding of demographic conditioning in LLMs. It argues that "demographic effects" are not stable, cue-invariant parameters but are signal-conditioned adaptations. Models respond to specific linguistic patterns (names, dialects, discourse structures) rather than abstract demographic categories. Consequently, current evaluation practices that rely on a single cue risk producing non-replicable or misleading findings regarding personalization and bias.

Recommendations:

1. Multi-Cue Evaluation: Researchers must use multiple, ecologically valid cues to assess robustness. Relying on a single cue (e.g., only names) provides an incomplete picture.
2. Mechanism-Aware Auditing: Evaluations should move beyond treating cues as one-dimensional indicators. Researchers must audit and model the components bundled with cues (e.g., measuring how strongly a cue triggers demographic inference and what linguistic features it introduces) to distinguish between true demographic bias and linguistic artifacts.

Conclusion:
Demographic cue selection is a substantive methodological choice, not a neutral technical detail. To make robust, interpretable claims about LLM behavior, the field must adopt multi-cue, mechanism-aware evaluation frameworks that acknowledge the complexity of how identity is signaled in real-world interactions.