💬 NLP

Subjective Multi-Bias Detection with Large Language Models

This paper presents a project that utilizes Large Language Models to detect and classify three types of subjective biases—framing, epistemological, and demographic—in text, leveraging the WIKIBIAS dataset of over 4,000 Wikipedia edit pairs to identify improper attitudes and misrepresentations.

Original authors: Ruiyu Li, Zhiying Zhu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruiyu Li, Zhiying Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are scrolling through a news feed or reading a Wikipedia article, and you stumble upon a sentence that feels just a little bit off. Maybe it makes a specific group of people sound like villains without saying it outright, or it uses fancy words to make a shaky claim sound like absolute fact. This is the world of subjective bias. It's not just about lying; it's about the subtle, sneaky ways writers can twist the truth or hide their opinions inside "facts." Detecting this is a huge challenge for computers because, unlike a math problem with a clear right or wrong answer, bias often hides in the tone, the choice of words, and the unspoken assumptions.

For years, scientists have tried to teach computers to spot these tricks using models like BERT, which is like a very smart student who has read millions of books but still sometimes misses the "vibe" of a sentence. More recently, a new generation of Large Language Models (LLMs) has arrived. Think of these as super-powered AI brains that don't just read words but understand the complex dance of human language, culture, and context. The big question researchers are asking is: Can these giant AI brains finally catch the sneaky, subjective biases that older models keep missing?

This paper is the story of a team of researchers who decided to put these new AI brains to the test. They took on a tricky dataset called WIKIBIAS, which contains over 4,000 pairs of sentences from Wikipedia edits. These pairs are like "before and after" snapshots, where someone has tweaked a sentence to add a specific type of bias. The researchers were looking for three specific kinds of tricks: framing bias (using one-sided words to push a point of view), epistemological bias (subtle language that makes a story seem more or less believable than it is), and demographic bias (words that assume things about a person's gender, religion, or background).

First, they tried the old school method: a standard BERT model. It was like asking a diligent but slightly myopic student to grade the papers. The result? It got a score of 0.33 on a scale called "macro-F1" (a way to measure how well the model does across all categories). The model struggled, especially with the trickier, less common types of bias, often confusing them or missing them entirely. It was clear that the old tools weren't sharp enough for the job.

So, the team built something new: EnsembleLlama. Instead of relying on just one AI, they created a "team of experts." They took a powerful open-source model called LLaMA2 (specifically the 7-billion parameter version) and gave it a special job. Instead of trying to guess all three bias types at once—which was confusing the model because some biases are rare—they trained three separate "mini-experts." One expert only looked for framing bias, another only for epistemological bias, and the third only for demographic bias. They then stacked these three experts together, letting them vote on the final answer.

The results were a game-changer. By using this team approach, the new model jumped its score from 0.33 all the way up to 0.59. That's a 26% improvement over the old baseline. It didn't just get better at the easy stuff; it significantly improved its ability to catch the rare and difficult biases that the old model kept missing. For example, when the old model saw a sentence about a dead IRA volunteer and thought it was just "framing," the new team correctly identified it as "epistemological bias."

However, the researchers are careful not to call this a perfect solution. They admit that while their model is much better, it still isn't perfect. There are cases where the AI gets confused, like when a sentence about a school founded by parents was mislabeled as having "no bias" when it actually had a "framing bias." They suggest that the model's success is still tied to how well it understands the text and the knowledge it has been fed. They also note that simply stacking the experts helped, but there might be even smarter ways to combine them in the future.

In short, this paper shows that by breaking a hard problem into smaller pieces and using a powerful new AI brain to tackle each piece, we can catch subjective bias much better than before. It's a big step forward, but the journey to perfectly understanding human bias in text is still ongoing.

Technical Summary: Subjective Multi-Bias Detection with Large Language Models

Problem Definition
This project addresses the pervasive challenge of detecting subjective bias in text, defined as improper attitudes or statements that contradict actual truth, often leading to misconceptions and social tensions. Specifically, the work focuses on identifying three distinct types of subjective bias within the WIKIBIAS corpus, a dataset derived from Wikipedia edits containing over 4,000 sentence pairs:

Framing Bias: The use of one-sided words or phrases conveying a specific point of view.
Epistemological Bias: Subtle linguistic features that affect the believability of the text.
Demographic Bias: Word or phrase usage based on presuppositions regarding specific demographic factors (e.g., gender, religion).

The task is complicated by significant class imbalance, where minor bias types (epistemological and demographic) are frequently misclassified, and the inherent difficulty of the task, which sometimes requires deep text understanding even for human annotators.

Methodology
The authors first established a BERT-base model as a baseline. This 110-million-parameter transformer was fine-tuned on the WIKIBIAS training set for three epochs. However, the baseline achieved a macro-F1 of only 0.33, struggling particularly with the imbalanced distribution of bias types.

To overcome the limitations of BERT's modeling capacity and the data imbalance, the authors proposed EnsembleLlama, a novel approach leveraging Large Language Models (LLMs) and ensemble learning:

LLM Backbone: Instead of using decoder-only models like GPT-3.5 or LLaMA2-70B for in-context learning (which is resource-intensive), the team fine-tuned the open-source LLaMA2-7B model. They froze the pre-trained LLaMA2 weights and appended linear layers (MLP classifiers) to the end of the architecture for classification. This design leverages LLaMA2's superior language understanding while remaining computationally efficient.
Prompt Engineering: Specific system prompts were designed to explain the concept of each bias type and provide examples (e.g., defining framing bias with an example sentence) to guide the model during inference.
Ensemble Strategy for Imbalance: Recognizing that training a single multi-class classifier failed to address class imbalance effectively, the authors re-formulated the problem. They trained three separate LLaMA-MLP classifiers, where each expert focuses exclusively on detecting one specific bias type (Framing, Epistemological, or Demographic).
Stacking: The outputs of these three specialized experts were stacked to produce the final multi-label detection results. This approach allowed each classifier to handle the specific distribution of its target class without being dominated by the majority "no bias" class or other bias types.

Key Contributions

Novel Architecture: The development of a fine-tuned LLaMA2-MLP classifier specifically adapted for subjective bias detection, moving beyond standard BERT-based approaches.
Ensemble Solution: The implementation of a stacked ensemble of specialized experts to directly tackle the class imbalance problem inherent in the WIKIBIAS dataset, rather than relying solely on loss function adjustments like focal loss.
State-of-the-Art Performance: The proposal of a method that significantly outperforms existing baselines on a challenging, fine-grained bias detection benchmark.

Results
Evaluated on the WIKIBIAS test set, the EnsembleLlama model demonstrated substantial improvements over the BERT baseline:

Macro-F1 Score: Increased from 0.33 (Baseline) to 0.59 (EnsembleLlama), an improvement of 0.26 (26 percentage points).
Class-Level F1: Significant gains were observed across all categories:
- Framing Bias (F-class): 0.49 $\rightarrow$ 0.75
- Epistemological Bias (E-class): 0.25 $\rightarrow$ 0.52
- Demographic Bias (D-class): 0.25 $\rightarrow$ 0.49
Training Efficiency: The model was trained on two NVIDIA 80GB H100 GPUs, with three epochs taking approximately one hour. No overfitting was observed during training.

Significance and Claims
The paper claims that the proposed EnsembleLlama approach achieves new state-of-the-art results on the WIKIBIAS benchmark. The authors emphasize that their method successfully mitigates the impact of dominant bias types on minor ones, thereby improving the detection of underrepresented bias categories.

However, the authors maintain a modest stance regarding the model's limitations. They acknowledge that performance remains dependent on the LLaMA2 model's inherent text understanding and knowledge base. Error analysis revealed that the model still struggles with:

Distinguishing between framing and epistemological biases in complex sentences.
Detecting multiple biases within a single sentence simultaneously.
Identifying subtle biases that require deep domain knowledge or single-word level analysis.

Consequently, the authors frame their work as a significant step forward in bias detection granularity but identify future directions involving more advanced ensemble algorithms, data augmentation, and the potential integration of domain-specific knowledge to further refine detection at the single-word edit level.

Technical Summary: Subjective Multi-Bias Detection with Large Language Models

More like this