🤖 AI

BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

This paper introduces BiasTrace, an annotation scheme that links specific reasoning behaviors to biased LLM outputs, revealing that subtle reasoning patterns rather than explicit language drive bias and enabling improved detection and inference-time mitigation.

Original authors: Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos, Karin Sevegnani, Emine Yilmaz

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Varsha Ramineni, Hossein A. Rahmani, Jerome Ramos, Karin Sevegnani, Emine Yilmaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out why a super-smart robot friend keeps making unfair or silly mistakes. In the world of Artificial Intelligence, these robots are called Large Language Models (LLMs). They are like digital encyclopedias that can write stories, solve math problems, and chat with you, but they sometimes learn bad habits from the books and websites they read. These bad habits are called "biases"—like thinking that only certain types of people are good at specific jobs, or assuming things about people based on how they look.

For a long time, scientists have been trying to fix these robots by checking their final answers. If the robot says something mean or wrong, they try to teach it not to do that again. But this is a bit like only looking at the final score of a soccer game to understand why a team lost. You miss all the interesting plays, the bad passes, and the confused moments that happened during the game. Recently, these robots have gotten a new superpower: they can "think out loud" before they give an answer. They write down a step-by-step plan, called a "reasoning trace," showing how they got to their conclusion. This paper is all about watching that thinking process to see where the robot actually goes wrong.

The researchers behind this study, Varsha Ramineni and her team, decided to stop just looking at the final answer and start investigating the robot's "thought process." They created a new tool called BIASTRACE. Think of BIASTRACE as a special pair of glasses that lets you see the invisible habits a robot has while it's thinking. Instead of just asking, "Is the answer wrong?", they asked, "What was the robot doing in its head that made it go wrong?"

They found something surprising. They expected that robots would make biased mistakes because they used "bad words" or explicitly stated stereotypes in their thinking. But the data showed that wasn't the main culprit. Instead, the biggest predictor of a biased, unfair answer was something they called "overthinking."

Imagine a robot trying to solve a riddle. If it gets stuck, it might start doubting itself, going back and forth, and second-guessing every single step. "Maybe the answer is A? No, wait, maybe it's B? But what if it's C?" The researchers found that when a robot gets into this loop of excessive doubt and confusion, it is much more likely to grab onto a stereotype just to make a decision, even if the evidence doesn't support it. It's like a person so nervous about making the right choice that they accidentally pick the wrong one just to stop thinking.

The team tested this on thousands of questions about social issues, like whether a wealthy neighborhood or a poor one has more drug problems. They watched how different robots (like Qwen and GPT) thought through these questions. They discovered that even when a robot didn't use any mean words, it could still reach a biased conclusion just by getting confused and overthinking the situation. In fact, when the robots were told to "be careful of bias," they sometimes got more confused and overthought the problem even more, which accidentally made the bias worse in some cases.

So, what did they do with this discovery? They used their new "glasses" (BIASTRACE) to build a better way to catch these mistakes. Instead of just asking a robot judge, "Is this answer fair?", they asked it to look for specific thinking habits, like "Is the robot overthinking?" or "Is it making up facts not in the story?" This new method was much better at spotting the dangerous answers before they happened.

Finally, they tried to use this to fix the problem in real-time. They set up a system where the robot generates eight different answers, and a helper checks the thinking process of each one. If the helper sees that the robot is "overthinking" or making bad assumptions, that answer gets tossed out. The robot then picks the best answer from the remaining, "clean" thoughts. This simple trick made the robots both smarter and fairer, reducing their biased mistakes significantly without making them worse at answering questions.

The main takeaway is that bias in AI isn't always about the robot saying something mean; often, it's about the robot getting confused and overthinking until it grabs onto a stereotype to feel sure of itself. By watching how the robot thinks, not just what it says, we can build better tools to catch these errors and make AI a little more fair for everyone.

Technical Summary: BIASTRACE – Linking Reasoning Behaviours to Biased Outputs in LLMs

Problem Statement

Large Language Models (LLMs) exhibit social biases that can lead to inaccurate and discriminatory inferences, posing significant risks in high-stakes domains such as healthcare, employment, and criminal justice. While prior research has made strides in measuring and mitigating bias, it predominantly focuses on the final outputs of models. This approach suffers from a fundamental gap: it offers limited understanding of the internal mechanisms that produce biased outcomes. Existing methods often train models to avoid explicitly biased language, yet biases frequently emerge implicitly through other pathways. Furthermore, current evaluations of reasoning capabilities largely assess final answer correctness or detect explicitly biased language, overlooking subtle reasoning behaviors that may drive discriminatory conclusions even when the final language appears neutral.

Methodology

1. BIASTRACE Annotation Scheme

The authors introduce BIASTRACE, a fine-grained annotation scheme designed to label specific reasoning behaviors in model-generated traces and link them to biased outcomes. Developed through iterative inspection of model traces and informed by cognitive science literature (e.g., Kunda, 1990; Kahneman, 2011), the scheme categorizes behaviors into four groups:

Bias-Specific Behaviors:
- Group Assumption: Attributing traits or likelihoods to a demographic group without contextual justification.
- Bias Acknowledgement: Explicitly recognizing the potential for bias or stereotyping within the thought process.
Meta-Cognitive Behaviors:
- Meta Reflection: Questioning the intent, scope, or framing of the task (e.g., suspecting a "trick" question).
Out-of-Context Knowledge:
- Outside Demographic Knowledge: Utilizing external knowledge about social groups not present in the context.
- Outside Topical Knowledge: Utilizing external knowledge about the specific topic not present in the context.
Reasoning Under Uncertainty:
- Overthinking: Exhibiting excessive doubt, complexity, or repeatedly revisiting the same answer options (more than three times).

2. Data Generation and Annotation

Dataset: The study utilizes the BBQ (Bias Benchmark for QA) dataset, which contains logically structured multiple-choice questions across nine demographic categories (e.g., race, gender, socioeconomic status) in both ambiguous and disambiguated contexts.
Models: Reasoning traces were generated by Qwen3-8B, Qwen3-14B, and GPT-OSS-120B under various prompt conditions (simple vs. guided) and reasoning effort levels.
Scale: A total of 250,976 annotated traces were produced.
Annotation Strategy: To scale the labeling process, the authors employed an LLM-as-a-Judge approach. The judge model (DeepSeek-V3.2) was validated against human annotations, achieving strong agreement (Cohen's $\kappa$ = 0.64–0.84) on key labels like Overthinking and Group Assumption. Labels with low agreement were excluded to ensure reliability.

3. Experimental Design

The study addresses two primary Research Questions (RQs):

RQ1: Do reasoning behaviors beyond explicitly biased language contribute to biased outputs, and which are most predictive?
- Method: Logistic regression was performed on annotated traces to identify predictors of biased outcomes (defined as incorrect answers that align with stereotypes).
RQ2: Can reasoning behaviors be leveraged to more reliably identify traces leading to biased outcomes?
- Method: A new evaluation prompt incorporating BIASTRACE behaviors was designed and compared against baseline methods (0–5 ordinal scales, binary 0/1 labels, and a Fairness Reward Model) using Pearson correlation with biased outcomes.
Mitigation: The authors tested an inference-time mitigation strategy where reasoning traces flagged as biased by the BIASTRACE prompt were filtered before applying majority voting.

Key Results

1. Predictive Power of Reasoning Behaviors (RQ1)

Overthinking is the Strongest Predictor: Contrary to the assumption that explicit biased language is the primary driver, Overthinking was found to be the strongest predictor of biased outputs (Odds Ratio = 2.34 in ambiguous contexts). This effect was amplified under guided prompts.
Implicit vs. Explicit: Group Assumption (explicit bias) was not a significant predictor in the overall model, though it was a significant predictor specifically in ambiguous contexts (Odds Ratio = 4.88, p < .001). This indicates that inferring unstated group attributes in ambiguous settings is a key pathway for biased outputs, even if the behavior does not drive bias across all contexts. Outside Demographic Knowledge significantly increased the odds of bias regardless of context.
Complex Interactions: Biased outcomes were concentrated in specific combinations of behaviors. For instance, the combination of Overthinking, Group Assumption, and Outside Demographic Knowledge resulted in bias rates reaching 0.30, with a lift exceeding 40 times the baseline.

2. Improved Bias Evaluation (RQ2)

Superior Correlation: The BIASTRACE-based evaluation prompt demonstrated significantly higher correlation with biased outcomes ( $r = 0.08$ to $0.41$) compared to baseline methods ( $r = 0.01$ to $0.19$).
Generalization: These findings generalized to models not involved in the training of the annotation scheme (e.g., GPT-OSS-120B).
Downstream Application: On the COMPAS recidivism benchmark, the BIASTRACE prompt showed the highest correlation ( $r = 0.38$ ) with per-sample fairness contributions, outperforming baselines in identifying samples that most impact group fairness.

3. Inference-Time Mitigation

Filtering Effectiveness: Applying majority voting only to traces flagged as "unbiased" by the BIASTRACE prompt (Maj-BIASTRACE) improved both accuracy and reduced bias rates across multiple model families (Qwen3 and Llama-3) compared to using all samples or a single sample.
Quantitative Gains: For Llama-3-8B-Instruct, the bias rate dropped from 13.36% (Single) and 12.36% (Maj-All) to 7.36% (Maj-BIASTRACE), while accuracy increased to 77.27%.

Significance and Claims

The paper argues that bias in LLMs cannot be fully understood by examining final outputs or explicitly biased language alone. Instead, bias often arises from subtle reasoning processes, particularly behaviors like overthinking and the misuse of external knowledge.

Novelty: BIASTRACE provides a new lens for investigating bias by decomposing reasoning into specific behaviors linked to cognitive science concepts, moving beyond coarse bias labels.
Practical Utility: The work demonstrates that reasoning-level annotations improve bias detection and can be operationalized as an effective inference-time intervention to reduce biased outputs without compromising accuracy.
Future Directions: The authors suggest that understanding these reasoning patterns is critical for developing more robust bias detection and fairer model behaviors. They propose future work in incorporating reasoning-aware training objectives and establishing which reasoning-bias relationships generalize across different models and languages.

The authors maintain a modest stance, acknowledging that their analysis studies observable behaviors and does not claim Chain-of-Thought (CoT) traces are a faithful representation of internal model computation. They also note limitations regarding the scope of models (open-source, English-only) and the structured nature of the BBQ dataset.