Sociodemographic Biases in Educational Counselling by Large Language Models

1. Problem Statement

As Large Language Models (LLMs) are increasingly integrated into educational settings for tasks ranging from personalized tutoring to administrative decision support, there is a critical need to understand whether these models inherit or mitigate sociodemographic biases found in human decision-making. While educational psychology has long documented that teachers' expectations and recommendations are influenced by students' race, gender, socioeconomic status (SES), and immigration background (even when academic performance is equivalent), it remains unclear how LLMs behave in similar high-stakes counseling scenarios. Specifically, the study addresses the lack of systematic understanding regarding:

• The extent of differential recommendations LLMs make across demographic groups.
• How the precision of student descriptions (information density) moderates the magnitude of these biases.
• Whether LLM bias patterns align with or diverge from documented human biases.

2. Methodology

The study employed a large-scale experimental design using a vignette-based approach to isolate demographic variables.

• Experimental Design:
  • Models Tested: Six frontier LLMs: GPT-5.2, GPT-5-nano (OpenAI), DeepSeek Chat v3.2, DeepSeek Reasoner v3.2 (DeepSeek), Grok 4.1 Fast (xAI), and Gemini Flash 3.5 (Google).
  • Variables:
    1. Demographic Conditions (15 levels): A neutral control ("a student"), 8 race-gender combinations (e.g., African American male/female, Asian male/female, etc.), 4 SES levels (low-income, working-class, middle-class, affluent), and 2 immigration backgrounds (first-generation, immigrant family).
    2. Information Density (9 levels): Ranging from "Minimal" (demographic only) to "Context Only," "Vague" (subjective descriptors), "Neutral/Positive/Negative Metrics" (GPA, attendance), and "Contradictory" (metrics vs. descriptors).
    3. Decision Contexts (10 scenarios): Including college counseling, course placement, gifted identification, discipline referrals, and parent conferences.
  • Scale: The study generated 243,000 model responses (6 models × 15 demographics × 9 density levels × 10 contexts × 10 vignette variations × 3 questions per context).
  • Prompting: Models were prompted to act as experienced school counselors and forced to choose a single option (A–D) on a 1–4 scale for specific questions (e.g., "College recommendation," "Discipline severity").
  • Analysis: Bias was quantified as $\Delta = \bar{x}_{demographic} - \bar{x}_{control}$. Statistical significance was assessed using t-tests with Benjamini–Hochberg FDR correction ($q < 0.05$). Internal consistency was validated using Cronbach's $\alpha$.

3. Key Contributions

1. Dataset Creation: A publicly available dataset of 243,000 LLM responses to K-12 counseling scenarios, systematically varying demographics and information density.
2. Bias Quantification: Demonstration that LLMs exhibit systematic sociodemographic biases that both align with and diverge from human biases.
3. Information Precision as a Moderator: Establishment that information quality is a critical factor; vague descriptions amplify bias, while concrete metrics significantly reduce it.
4. Model Heterogeneity: Documentation of substantial variance in bias profiles across different models, suggesting bias is a result of specific training choices rather than an inevitable property of LLMs.

4. Key Results

A. Extent and Patterns of Bias (RQ1)

• Prevalence: All six models exhibited measurable biases. Approximately 14.7% of demographic-context combinations showed statistically significant effects.
• Socioeconomic Bias (Dominant Pattern): Low-income and working-class students received significantly lower recommendations for college and graduation pathways compared to affluent students (e.g., a gap of nearly 1 full scale point in graduation planning under minimal information).
• Racial Bias:
  • Asian Students: Consistently received favorable treatment (the "model minority" stereotype), particularly in college counseling and parent engagement.
  • African American Students: Showed positive bias in college counseling (contrary to typical human underestimation), but negative bias in extracurricular/gifted contexts.
  • Hispanic Males: Showed negative bias in extracurricular and gifted identification.
• Immigration Status: First-generation and immigrant students generally received slightly positive treatment, though immigrant students faced negative bias in gifted identification under minimal information.

B. Alignment with Human Biases (RQ2)

• Alignment: LLMs replicated human biases regarding socioeconomic expectations and the Asian "model minority" stereotype.
• Divergence:
  • Positive Bias for Underserved Groups: Unlike human teachers who often underestimate African American potential, LLMs showed positive bias for African American females in college counseling.
  • Vague-Negative Compensation: In conditions where students were described with negative subjective language but no metrics, LLMs often rated demographic groups higher than the control group, suggesting a "benefit of the doubt" heuristic not typically seen in human annotators.

C. The Role of Information Precision (RQ3)

• Magnitude Reduction: This is the study's most striking finding. Vague descriptions produced a mean absolute bias of $|\Delta| = 0.19$, while concrete metrics reduced it to $|\Delta| = 0.07$—a nearly threefold reduction.
• Mechanism: When objective data (GPA, attendance) is absent, LLMs appear to fill informational gaps with demographic associations.
• Reversal of Bias: In "Effort Attribution" questions, vague negative descriptions led to working-class/immigrant students being blamed for lack of effort. However, when neutral metrics were provided, the bias reversed, and these same groups received more sympathetic attributions (external factors).

D. Model Heterogeneity (RQ4)

• GPT-5.2: Emerged as the least biased model overall, with low absolute bias values and negative z-scores (less bias than average) across most conditions.
• DeepSeek Reasoner: Showed the strongest positive racial bias (favoring all racial groups) but reversed this for socioeconomic groups (disfavoring affluent students). Notably, it failed to reduce bias even when concrete metrics were provided.
• Grok 4.1 Fast: Exhibited the most pronounced socioeconomic stratification, heavily penalizing low-income students while favoring affluent ones.
• Gemini Flash: Showed uniformly non-negative bias, offering the highest positive treatment to African American students and strong compensatory responses to negative descriptions.

5. Significance and Implications

• System Design over Model Selection: The study argues that system architecture is as important as model selection. Educational AI systems must be designed to elicit rich, objective student data (metrics) rather than relying on minimal demographic descriptors. Minimal information queries, often perceived as "neutral," actually maximize demographic bias.
• Bias is Not Inevitable: The substantial heterogeneity between models indicates that bias is a consequence of specific training data and choices (e.g., RLHF), not an inherent flaw of the transformer architecture. This supports the need for fairness auditing as a standard component of model selection for high-stakes applications.
• Complexity of AI Fairness: The divergence between LLM and human biases (e.g., the "vague-negative compensation" effect) suggests that AI does not simply mirror societal prejudice. These patterns could represent beneficial over-correction or new forms of unfairness, requiring careful consideration in deployment.
• Future Directions: The authors call for structured-input systems to reduce bias, further investigation into the mechanisms of compensation effects, and cross-cultural studies beyond U.S. contexts.

Conclusion: The paper concludes that while LLMs currently exhibit significant sociodemographic biases, these can be substantially mitigated through the provision of detailed, individualized student information. Relying on vague descriptions in AI-driven educational counseling is a high-risk strategy that amplifies disparities.