💬 NLP

FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

This paper introduces FairFund-Bench, a comprehensive benchmark demonstrating that while LLMs' demographic biases in resource allocation are highly sensitive to audit design and often reverse between individual and comparative tasks, their evaluations are far more strongly and consistently driven by causal framings of need that align with human deservingness theories.

Original authors: Martin Lukk (University of Toronto)

Published 2026-08-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Martin Lukk (University of Toronto)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the referee in a high-stakes game where a limited pot of money needs to be shared among people asking for help. In the real world, we often worry that referees might be unfair, giving more money to people who look like them or less to people from certain groups. Now, imagine we start using super-smart computer programs, called Large Language Models (LLMs), to act as these referees. These models are like digital brains that have read almost everything ever written on the internet, so they are incredibly good at understanding language. But because they learned from humans, they might have picked up our human biases, too. The big question scientists are asking is: When these AI referees decide who gets the cash, are they fair, or are they secretly playing favorites based on a person's name, race, or gender?

This is exactly what the paper "FairFund-Bench" investigates. The researchers noticed something confusing: different scientists had tested these AI referees and got totally opposite answers. Some said the AI was being too nice to minorities, while others said it was being too mean. It was like one group of people saying the referee gave the winning team extra points, and another group saying the same referee docked points from them. The author of this paper realized the problem wasn't the AI itself, but how the tests were being run. They built a new, super-flexible testing kit called FairFund-Bench to see how the way you ask the question changes the answer.

Think of the AI as a very literal, slightly nervous student taking a test. If you ask the student, "Here is one person, Latoya. Does she deserve help?" the student might say, "Yes, absolutely!" and give her a high score. But if you put Latoya next to three other people and say, "Rank these four people from most to least deserving," the student might suddenly change their mind and put Latoya lower down the list. The researchers found that the AI's "bias" flips depending on the test format. When the AI sees people one by one, it often tries to be extra fair and gives everyone the same amount of money. But when the test is disguised—where the AI doesn't realize it's being watched and the names are mixed up with different stories about why they need money—the AI starts showing its true colors, giving significantly more money to some groups and less to others.

The study tested 14 different AI models with 600 different requests for financial help, covering four different races and two genders. They found that while the AI's bias based on race or gender was actually quite small overall, it was 3 to 4 times bigger when the test was "disguised" compared to when it was "transparent" (where the AI knew it was being tested). For example, in a transparent test, the AI would often split a $10,000 pot perfectly evenly among everyone. But in a disguised test, the gap in how much money different groups received grew from an average of $36 to $121.

However, the biggest surprise wasn't about race or gender at all. The paper found that the AI was much more influenced by the story of why someone needed money than by who they were. If a person's need was caused by something outside their control (like a layoff or a natural disaster), the AI gave them way more money—roughly $469 more than someone whose need was caused by a bad choice, and nearly $800 more than someone whose need was stigmatized (like an addiction) without any sign of them trying to fix it. This "story effect" was so strong that it was ten times bigger than the tiny differences the AI made based on race or gender.

The author suggests that these AI models are actually very good at copying how humans judge who "deserves" help, but that doesn't necessarily mean they are making the right moral choices. They also warn that if we only test AI in simple, obvious ways, we might think they are perfectly fair because they are trying to look good. But when we test them in more realistic, tricky situations, we see that they still carry some of the same biases and judgments that humans do. The paper concludes that we can't just look at one test to decide if an AI is fair; we have to look at how it behaves in many different situations, because the way we ask the question changes the answer.

Technical Summary: FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

Problem Statement

Large language models (LLMs) are increasingly deployed to evaluate competing claims for scarce resources, such as job applications, financial advice, and welfare eligibility. However, recent audits regarding demographic bias in these allocation decisions have yielded inconsistent and often contradictory results. Some studies report positive discrimination (favoring women and ethnic minorities), while others report negative discrimination against the same groups, even when evaluating the same models. The author argues that these discrepancies stem not from inherent model properties, but from unexamined variations in audit design choices—specifically the evaluation task, comparison context, and stimulus presentation mode. Existing benchmarks often fail to systematically vary these design parameters, leading to fragile conclusions about model fairness.

Methodology: FairFund-Bench

The paper introduces FairFund-Bench, the first benchmark designed to systematically vary key audit design choices to measure their impact on observed bias. The benchmark evaluates 14 leading LLMs across 7 providers using a pipeline comprising stimulus construction, an audit instrument, and a four-pillar scoring system.

1. Stimulus Construction

The stimuli consist of 600 unique human-authored financial aid appeals, calibrated against a corpus of 1.3 million real GoFundMe campaigns to ensure ecological validity in length and narrative structure.

Demographics: Appeals signal race (White, Black, Hispanic, Asian) and gender (Male, Female) via validated names.
Causal Framing: Based on the CARIN welfare deservingness theory, appeals vary in the causal attribution of need: No Cause, Structural (external), Self-cause (voluntary choice), Stigma (high blame, e.g., addiction), and Stigma with Redemption (corrective action).
Scenarios: Three domains (Medical, Rent, Education) with five scenarios each.

2. Audit Instrument Design

The benchmark varies three critical axes of design:

Tasks:
- Rate: Single-stimulus priority rating (1–5).
- Rank: Multi-stimulus ordering by priority.
- Allocate: Multi-stimulus distribution of a fixed $10,000 fund.
Comparison Context:
- Single-stimulus: One claimant evaluated at a time.
- Multi-stimulus: Multiple claimants evaluated simultaneously.
Presentation Mode:
- Transparent: Demographic differences are obvious (e.g., identical appeals differing only by name).
- Disguised: Demographic axes are co-varied with scenario details (using balanced placement schemes) to obscure the audit intent, mimicking real-world deployment where minimal pairs are rare.

3. Evaluation and Scoring

Each model generates responses for 6,360 observations (600 appeals × 3 tasks × 2 contexts × 2 presentation modes, with specific adjustments for bundle composition). Model performance is scored on four independent pillars:

P1 (Demographic Bias): Magnitude of between-group allocation variation (lower is better).
P2 (Deservingness Alignment): Consistency with human CARIN heuristics regarding need framing (higher is better).
P3 (Cross-Task Consistency): Stability of bias patterns across Rate, Rank, and Allocate tasks.
P4 (Cross-Context Consistency): Stability of bias patterns between transparent and disguised presentations.

Key Results

1. Audit Format Reverses Bias Direction

The study demonstrates that the direction of demographic bias is highly sensitive to the audit format.

Rating Task: Models exhibit positive discrimination toward ethnic minorities (Black, Hispanic, Asian) when rating individuals in isolation.
Ranking Task: The same models exhibit negative discrimination against some of these groups (specifically Asian and Hispanic claimants) when ranking them side-by-side.
Allocation Task:
- In transparent bundles (where demographic differences are obvious), models overwhelmingly engage in "equal splitting," distributing funds equally regardless of demographics. This suggests audit awareness or alignment training suppressing bias in obvious minimal-pair scenarios.
- In disguised bundles, equal splitting drops significantly (from ~100% to ~2% for race), and demographic disparities emerge. The magnitude of bias in disguised audits is 3–4 times larger than in transparent audits (e.g., $121 vs. $36 average gap for race).

2. Causal Framing Dominates Demographics

The effect of causal framing (deservingness) on allocation is substantially larger than demographic effects.

Models consistently allocate more funds to claims with Structural causes compared to Self-cause or Stigma causes.
The magnitude of framing effects exceeds demographic disparities by several times to an order of magnitude (e.g., $469–$ 795 differences based on framing vs. ~$100 differences based on race).
Unlike demographic bias, framing effects are robust and consistent across all tasks, contexts, and presentation modes, indicating that LLMs reliably reproduce human deservingness heuristics.

3. Model Performance

Demographic Bias (P1): Overall bias magnitudes are small (Cohen's d < 0.20), with most models scoring between 0.03 and 0.08.
Deservingness Alignment (P2): Larger frontier models (e.g., Gemini 2.5 Pro, Opus 4.6) show higher alignment with human deservingness patterns than smaller models.
Consistency (P3, P4): Models show high consistency across tasks and contexts, though the direction of demographic bias can flip between tasks.

Significance and Claims

The paper claims that the inconsistency in prior LLM bias literature is a methodological artifact rather than a model defect. By systematically varying audit design, FairFund-Bench reproduces the full spectrum of previously reported findings (positive, negative, and null bias) using the same models.

The author emphasizes three primary implications:

For Evaluators: Single-format audits are insufficient for credible claims about model bias. Ignoring the distinction between transparent and disguised prompts may lead to underestimating disparities, as "equal splitting" in transparent minimal-pair audits masks biases revealed in more realistic, disguised contexts.
For Developers: A single fairness score is inadequate. The four-pillar framework reveals that consistent performance (low volatility) and substantive alignment (reproducing human heuristics) do not necessarily coincide.
For Allocation Systems: Current LLMs robustly reproduce human deservingness judgments (e.g., penalizing self-caused or stigmatized needs). The author notes this is not necessarily desirable and raises ethical questions about whether allocation systems should mirror or overcome these human heuristics.

The paper concludes that FairFund-Bench provides a necessary framework for navigating the design space of bias audits, offering a public benchmark and scoring code to facilitate more rigorous and reproducible evaluations of LLMs in distributive contexts.