Identifying High-Confidence Social Biases in LLMs for Trustworthy Conversational Tutoring Agents

Technical Summary: Identifying High-Confidence Social Biases in LLMs for Trustworthy Conversational Tutoring Agents

Problem Statement
Conversational tutoring agents powered by Large Language Models (LLMs) offer scalable, personalized educational support but carry significant risks regarding social biases. While LLMs are known to perpetuate stereotypes and exhibit overconfidence in their judgments, existing evaluations often rely on general-purpose benchmarks (e.g., StereoSet) rather than naturalistic educational contexts. In educational settings, particularly for second language (L2) learners, high-confidence but incorrect or biased judgments can mislead students who lack the critical resources to identify such errors. There is a critical gap in understanding how LLM confidence interacts with bias detection specifically within conversational tutoring scenarios, where the stakes for learner outcomes are high.

Methodology
To address this gap, the authors propose a novel dataset generation method and a two-part experimental framework:

1. Dataset Generation (Biased Conversational Tutoring Dataset):

   • Source: The authors collected 4,210 authentic student–AI tutor conversations from an online language learning platform (EFL learners).
   • Preprocessing: Data was stratified to balance gender representation (55% female, 45% male) and filtered for depth (minimum 12 turns), resulting in 1,844 dialogues. All personally identifiable information was removed.
   • Regeneration: Using DeepSeek-V3.1, the authors regenerated these dialogues to preserve original linguistic features (discourse structure, interaction style, proficiency) while injecting a single controlled turn.
   • Bias Injection: The injected turns were derived from the StereoSet benchmark, categorized as stereotypical, anti-stereotypical, or neutral, and aligned with specific target groups and contexts.
   • Validation: Two expert annotators reviewed the generated dialogues using the Context Association Test (CAT) protocol, resulting in a final dataset of 1,727 valid conversations (610 stereotypical, 554 anti-stereotypical, 563 neutral).

2. Experimental Evaluation:

   • Models Evaluated: Three state-of-the-art LLMs: GPT-5.1, Gemini 2.5 Pro, and Claude Sonnet 4.5.
   • Experiment #1 (Bias Detection & Confidence): Models were tasked with classifying text/dialogue as stereotypical, anti-stereotypical, or neutral. They were prompted to self-verify their judgments and report confidence via two methods: (1) an overall confidence score (0–100) and (2) class-specific confidence distributions. Performance was measured using accuracy, precision, recall, and F2 scores. A confidence-weighted bias score was introduced to quantify the risk of high-confidence errors.
   • Experiment #2 (Reasoning & Feedback Coherence): The least biased model (Claude Sonnet 4.5) was prompted to provide reasoning explanations and generate student feedback for its incorrect classifications. Human raters evaluated the coherence between the model's expressed confidence, its reasoning, and its feedback using a 3-point Likert scale.

Key Contributions

1. Methodological Innovation: The paper proposes a method for creating student-tutor dialogue datasets that evaluate stereotypical biases under naturalistic instructional conditions, bridging the gap between static benchmarks and dynamic educational interactions.
2. Uncertainty Quantification Analysis: The study investigates the role of model confidence in LLM-based tutoring, specifically analyzing how confidence correlates with stereotypical biased reasoning and the resulting feedback provided to students.
3. Risk Identification: The authors establish a direct connection between the confidence of models' incorrect judgments and the amplification of bias in feedback, highlighting the specific risks of overconfident, biased behavior in educational agents.

Results

• Performance Degradation: Bias detection performance dropped significantly when moving from the StereoSet benchmark to the conversational tutoring dataset. There was a mean relative degradation of 16.3% across key metrics (Precision, Recall, F2, Accuracy). The best-performing models (Sonnet and Gemini) saw a 19% degradation in metrics when transitioning to the tutoring context.
• Overconfidence: All models exhibited high absolute confidence even when their judgments were incorrect. For instance, 34% of errors by Claude Sonnet 4.5 occurred with "very high" confidence (≥90%), and 38% occurred with "high" confidence (≥80%).
• Calibration Issues: Models showed poor calibration, particularly regarding anti-stereotypical statements. Claude Sonnet 4.5 had a very low recall (0.177) for anti-stereotypes and was more confident in its incorrect anti-stereotype predictions (0.826) than in its correct ones (0.730).
• Coherence: Despite being incorrect, the models demonstrated high internal coherence. In Experiment #2, 75% of reasoning explanations and 68% of generated feedback were rated as "highly coherent" with the model's expressed confidence, suggesting that overconfidence reinforces biased reasoning and feedback.
• Self-Correction Failure: When prompted to reconsider their judgments, models revised their responses in only 0.1% of cases, indicating a rigid adherence to initial biased judgments.

Significance and Claims
The paper claims that current state-of-the-art LLMs are not only overconfident but also internally consistent in expressing this confidence across their reasoning and feedback mechanisms. This behavior is particularly dangerous in educational contexts, where learners may lack the background knowledge to detect implicit stereotypes. The authors argue that relying solely on general-purpose benchmarks leads to a systematic underestimation of potential harms in educational AI.

The study emphasizes that traditional uncertainty frameworks (aleatoric and epistemic) are insufficient for capturing the interpretive ambiguity of social biases. Instead, the authors advocate for the integration of hermeneutic uncertainty—acknowledging the legitimacy of plural interpretations—to foster more responsible AI tutoring agents. The findings suggest that without mechanisms for calibrated uncertainty communication, LLM-based tutoring agents risk amplifying social biases and misleading learners, necessitating evaluations grounded in realistic educational settings rather than abstract benchmarks.