Technical Summary: BiasLab – A Multilingual Dual-Framing Framework for LLM Bias Measurement

Problem Statement

Large Language Models (LLMs) are increasingly deployed in workplace and Human Resources (HR) contexts to support critical decisions regarding hiring, job design, and organizational policy. However, existing methods for evaluating LLM bias suffer from significant methodological fragmentation. Current approaches often rely on single-prompt formulations, heterogeneous output formats, and English-centric datasets, which fail to capture the robustness of model preferences across different linguistic and framing contexts. Furthermore, distinguishing between intrinsic bias (internal representations) and extrinsic bias (output-level behavior dependent on prompting) remains challenging. There is a lack of standardized, reproducible instruments that allow practitioners to quantify directional output-level bias across multiple models and languages before deployment.

Methodology: The BiasLab Framework

The paper introduces BiasLab, an open-source, end-to-end evaluation system designed to quantify directional preferences in LLMs through a multilingual dual-framing approach. The framework operates through four primary stages:

1. Dual-Framing and Mirrored Assertions:

   • For any given topic and two competing targets (Target A and Target B), BiasLab generates two semantically opposed prompt variants.
   • Affirmative Framing: Asserts Target A is the correct/preferred side.
   • Reverse Framing: A strict mirror of the affirmative prompt, replacing only the surface form of Target A with Target B while maintaining identical lexical and syntactic structures.
   • This symmetry ensures that observed preference shifts reflect the model's response to the targets themselves rather than phrasing artifacts.

2. Multilingual Prompt Construction with Randomized Wrappers:

   • The system supports 12 languages spanning five language families and six writing systems.
   • To ensure robustness against prompt templating artifacts, the framework employs randomized wrapper perturbations. Each prompt includes a randomly sampled prefix (meta-instruction) and a suffix (strict fixed-choice constraint).
   • These wrappers are translated to be semantically equivalent across languages and sampled independently across iterations.

3. Automated Evaluation and Normalization:

   • The system queries 10 diverse LLMs (from US, Chinese, and European developers) across 30 iterations per framing direction per language, generating 43,200 total responses.
   • Normalization: Raw model outputs (which may include refusals, hedging, or verbose explanations) are mapped to a unified categorical scale (Strongly Agree to Strongly Disagree) using an LLM-based judge. This step handles non-verbatim responses and refusals, labeling them as "Neutral" if the model abstains.
   • Polarity Alignment: Scores are aligned such that agreement with the affirmative framing and agreement with the reverse framing (after negation) both contribute to a unified directional score toward Target A.

4. Scoring and Visualization:

   • Responses are scored ordinally (+2 to -2). The final bias score is the mean of polarity-aligned scores.
   • The framework distinguishes between "no preference due to balanced agreement" and "no preference due to abstention" by explicitly reporting neutrality rates.

Key Contributions

• A Standardized Measurement Instrument: BiasLab provides a model-agnostic, language-inclusive framework for measuring output-level bias, addressing the fragmentation in current evaluation practices.
• Dual-Framing Detection of Asymmetry: By utilizing mirrored prompt pairs, the framework reveals a recurring asymmetric pattern invisible to single-frame designs: models often reject disfavored claims more strongly than they endorse their opposites (a "protective" rather than "assertive" bias).
• Multilingual Robustness: The study demonstrates that directional preferences are consistent across 12 languages, suggesting that observed biases are not merely English-centric artifacts but reflect shared model-level characteristics.
• Open-Source Implementation: The full framework, including probe generation, scoring, and visualization modules, is publicly available to support replication and extension.

Results

The framework was applied to six workplace-relevant topics: (1) Gender in leadership, (2) Employment gaps, (3) Age in hiring, (4) Remote vs. office work, (5) Four-day vs. five-day work weeks, and (6) AI-assisted vs. human-only hiring.

• Consistent Directional Preferences: All ten evaluated models exhibited consistent directional preferences across every topic:
  • Gender: Preference for female managers over male managers.
  • Employment Gaps: Preference for candidates with employment gaps being equally capable as those without.
  • Age: A lean toward older workers in hiring contexts.
  • Work Arrangements: Preference for remote work over office work and four-day work weeks over five-day weeks.
  • AI in Hiring: A strong, unanimous preference for AI-assisted hiring over human-only hiring.
• Framing Asymmetry: A significant finding was the "protective" bias pattern. Models were more willing to reject a disfavored claim (e.g., "Male managers are better") than to assert its opposite ("Female managers are better"). This asymmetry was particularly evident in topics involving protected attributes (gender, age) and employment gaps.
• Model Variability: While direction was consistent, effect sizes varied. For instance, GLM-4.7 Flash showed high neutrality rates (41–55%), while GPT-5.2 and Gemini 3 Flash showed strong effect sizes on several topics.
• Self-Referential Bias: Topic 6 (AI-assisted hiring) produced the most uniform signal, with all models favoring AI-assisted hiring. The authors note this is a self-referential topic where models evaluate their own class of technology.

Significance and Claims

The paper positions BiasLab not as a tool to definitively label all preferences as "bias" in a moral sense, but as a standardized instrument for measurement and transparency.

• Context-Dependent Interpretation: The authors argue that whether a preference constitutes "bias" depends on the topic. For legally protected attributes (gender, age), directional preferences map onto equal-employment standards and represent compliance-relevant bias. For other topics (work weeks, remote work), these are better described as systematic preferences that lack an external fairness ground truth but still represent a non-neutral stance.
• Practical Implications for HR: The study claims that LLMs consulted in HR do not provide neutral summaries of evidence but rather consistent directional leans. Organizations should treat these outputs as instruments requiring pre-deployment evaluation.
• Regulatory Alignment: The framework supports emerging regulatory requirements (e.g., EU AI Act, NYC Local Law 144) by providing a reproducible method for bias auditing and model comparison.
• Caution on Self-Referential Topics: The authors specifically caution against relying on LLMs to evaluate the adoption of AI in hiring, given the uniform pro-AI preference observed in the study.

The paper concludes that BiasLab offers a practical safeguard for organizations to profile model preferences and a measurement standard for ongoing audit, emphasizing that understanding the direction and asymmetry of model outputs is critical for safe deployment in consequential decision-making contexts.