Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers

This paper presents the first systematic comparison of fairness between reasoning and non-reasoning rerankers, finding that reasoning models neither improve nor harm fairness compared to traditional approaches, as they largely preserve the fairness characteristics of their input rankings.

The Gist

Imagine you are the head chef at a busy restaurant. Your job is to serve customers the best dishes (search results) based on what they order (search queries).

For a long time, your kitchen had a simple rule: "Serve the tastiest dish first." This is like a standard search engine. It looks at a list of 500 possible dishes and picks the ones that taste the most like what the customer asked for.

Recently, a new type of chef has arrived: the "Reasoning Chef." Before serving a dish, this chef doesn't just taste it; they pause, think, and write a little note explaining why it's the best choice. "This soup is great because it has fresh herbs, and the customer loves herbs," they might think. These reasoning chefs have proven to be incredibly good at finding the tastiest dishes.

But here is the big question the paper asks: Does this new "thinking" process also make the menu fairer?

The Big Question: Does Thinking Make It Fairer?

"Fairness" in this context means making sure the menu isn't just about the most popular dishes. It means ensuring that if a customer asks for "soup," they get a mix of soups from different regions (USA, Sweden, Finland, etc.), not just 9 out of 10 soups from the USA, even if the Swedish soups are just as tasty.

The researchers wanted to know: Does the "Reasoning Chef" naturally start serving a more diverse menu, or do they just serve the same old favorites, only with a longer explanation?

The Experiment: A Taste Test

The researchers set up a massive taste test using a special library of recipes (the TREC 2022 Fair Ranking dataset). They compared two types of chefs:

1. The Old School Chef (Non-Reasoning): Picks the best dish instantly based on taste.
2. The Thinking Chef (Reasoning): Pauses, writes a justification, and then picks the dish.

They tested them in different scenarios:

• The Raw Ingredients: Starting with a basic list of dishes.
• The Rewritten Order: Sometimes customers give a messy order ("sailing, boat, ocean"). The researchers rewrote it into a clear sentence ("Tell me about sailing and types of boats") to see if clearer instructions helped.

The Surprising Results

Here is what they found, broken down simply:

1. The Thinking Chef is Great at Taste, But Not at Diversity
The "Reasoning Chefs" were fantastic at finding the most relevant dishes. If you asked for "sailing," they found the best sailing articles. However, they did not make the menu more fair.

• If the initial list of dishes was mostly from the USA, the Reasoning Chef served a list that was still mostly from the USA.
• If the list was diverse, the Reasoning Chef kept it diverse.
• The Analogy: Imagine a librarian who is asked to find the "best" books. If the library only has books by men, the librarian (even a super-smart, thinking one) will still only recommend books by men. The librarian isn't ignoring women; they just can't recommend books that don't exist in the library.

2. The "Thinking" Didn't Change the Outcome
Whether the chef thought deeply about why a dish was good or just picked it quickly, the final result was almost identical in terms of fairness.

• The Metaphor: It's like two people sorting a deck of cards. One person sorts them by looking at the card and thinking, "This is a King, it's important." The other just looks and says, "King." If the deck only has Kings and Queens, both people will end up with the same pile. The "thinking" didn't magically turn a Queen into a King or bring in a new suit.

3. The Real Problem is the "Library," Not the "Chef"
The study found that the biggest unfairness came from Geography (where the content is from). Even when the chefs were given a "perfect" list of relevant documents (an "Oracle" list where relevance was 100% perfect), they still struggled to be fair regarding geography.

• Why? Because the information about where a document is from isn't always written in the text. If a recipe doesn't say "Made in Poland," the chef (AI) can't guess it.
• The Lesson: You can't fix a lack of diversity by just making the chef smarter. You have to diversify the ingredients in the kitchen first.

4. Clearer Orders Help Everyone
The researchers found that when customers gave clear, natural language orders (e.g., "Tell me about sailing") instead of messy keywords (e.g., "sailing boat ocean"), everyone did a better job. Both the Old School and Thinking chefs found better results. This suggests that how we ask questions matters more than how the computer thinks.

The Bottom Line

Does reasoning make search more fair?
No, not yet.

The "Thinking" AI models are amazing at finding the right answer, but they don't automatically know to find a fair mix of answers. They are like a mirror: if you feed them a biased list of information, they will reflect that bias back at you, even if they write a long, thoughtful essay about why they chose it.

What needs to happen?
To get fair search results, we can't just rely on smarter algorithms. We need to:

1. Fill the library: Make sure the internet (the data) has content from all over the world and all different groups.
2. Ask better questions: Use clear, natural language when searching.
3. Build fairness into the training: We need to teach these AI chefs explicitly, "Hey, if two dishes taste the same, please serve one from a different country," rather than hoping they figure it out on their own.

In short: Thinking helps you find the best answer, but it doesn't help you find a fair answer unless you teach it to care about fairness first.

Technical Summary

Here is a detailed technical summary of the paper "Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers."

1. Problem Statement

Search engines play a critical role in information access, yet the introduction of Reasoning Large Language Models (LLMs) as rerankers raises a new question: Do these models, which generate intermediate reasoning steps to improve relevance, also improve ranking fairness?

While reasoning rerankers (e.g., Rank1, Qwen3-Reranker) have demonstrated superior performance in relevance metrics (nDCG), their impact on group fairness—specifically the equitable distribution of exposure across sensitive demographic attributes (e.g., geography, gender, occupation)—remains underexplored. There is a concern that reasoning processes, shaped by pretraining data and internal biases, might amplify unfairness by generating confident but skewed justifications, or conversely, that the deliberative nature of reasoning could help mitigate bias.

2. Methodology

Dataset and Setup

• Dataset: The study utilizes the TREC 2022 Fair Ranking Track dataset, derived from English Wikipedia and Wikidata.
• Attributes: Eight sensitive demographic attributes were evaluated: age, alphabetical order, gender, languages, occupation, popularity, source geography, and subject geography.
• Metrics:
  • Relevance: Normalized Discounted Cumulative Gain at 10 (nDCG@10).
  • Fairness: Attention-Weighted Rank Fairness (AWRF), which measures the Jensen-Shannon divergence between the actual exposure distribution of protected groups and a target fair distribution (combining relevant document demographics and global population statistics).
  • Combined Score: $M1 = AWRF \times nDCG$.

Experimental Design

The authors conducted a systematic comparison across four initial retrieval settings:

1. BM25 with original keyword queries.
2. BM25 with rewritten queries (generated via GPT-4o-mini to simulate natural language).
3. Qwen3-Embedding-8B dense retrieval with rewritten queries.
4. Reciprocal Rank Fusion (RRF) of BM25 and Qwen3.
5. Oracle Ranking: A baseline where documents are reordered to achieve near-perfect relevance (nDCG=0.9) to isolate fairness behavior from relevance limitations.

Models Evaluated

Six rerankers were compared, split into two categories:

• Reasoning Rerankers (R): Rank1-7B, Qwen3-Reranker-8B, ReasonRank-7B. These models generate intermediate reasoning chains or justifications before scoring.
• Non-Reasoning Rerankers (N): RankZephyr-7B, RankLLaMA-7B, MonoT5-0.3B. These models predict relevance scores directly without explicit reasoning steps.

Statistical Analysis

The study employed paired t-tests to detect differences and Two One-Sided Tests (TOST) to test for statistical equivalence (within a tolerance $\delta=0.05$) between rerankers and initial retrieval baselines.

3. Key Contributions

1. First Systematic Comparison: This is the first study to systematically compare fairness outcomes between reasoning and non-reasoning rerankers in information retrieval.
2. Stability of Fairness: The authors demonstrate that current reasoning capabilities neither improve nor harm fairness compared to non-reasoning approaches.
3. Query Formulation Impact: The study highlights that query formulation (rewriting keywords into natural language) significantly improves relevance for all models but has a negligible effect on fairness metrics.
4. Attribute-Specific Disparities: The research identifies that geographic attributes (especially subject geography) consistently exhibit the lowest fairness scores (10-15% lower than other attributes) across all models, regardless of architecture.

4. Key Results

Relevance vs. Fairness Trade-off

• Relevance: Reranking significantly improved nDCG@10 across all models, especially when using rewritten natural language queries. Reasoning models generally performed competitively with non-reasoning models in relevance.
• Fairness (AWRF): The AWRF scores remained stable (0.33–0.35) across all models and settings.
  • Equivalence: Statistical tests (TOST) confirmed that all rerankers (both reasoning and non-reasoning) produced fairness scores statistically equivalent to their initial retrieval baselines.
  • No Improvement: Reasoning models did not leverage their "deliberative" capabilities to correct demographic imbalances.
  • No Harm: Conversely, they did not degrade fairness compared to standard rerankers.

The Oracle Experiment

To isolate fairness from relevance limitations, the authors reranked an "Oracle" list where relevance was maximized (nDCG $\approx$ 1.0).

• Even with perfect relevance, fairness scores did not significantly improve.
• Pointwise Models: Reasoning pointwise models (Rank1, Qwen3) showed slightly higher AWRF than non-reasoning counterparts, suggesting they might implicitly consider diversity when relevance is equal, but the difference was weak.
• Listwise Models: The trend reversed; the non-reasoning listwise model (RankZephyr) slightly outperformed the reasoning listwise model (ReasonRank). The authors hypothesize that reasoning listwise models may be over-optimized for strict relevance instructions, ignoring diversity signals.

Demographic Breakdown

• Geography Gap: "Subject geography" consistently showed the lowest fairness scores (e.g., Oracle AWRF $\approx$ 0.76–0.82) compared to attributes like language or gender (AWRF $\approx$ 0.87–0.96).
• Root Cause: Geographic information is often not explicitly present in the text of documents, making it difficult for rerankers (which rely on text content) to condition on these attributes.

5. Significance and Conclusion

Primary Conclusion:
Current implementations of reasoning rerankers preserve the fairness characteristics of their input ranking rather than actively improving them. The "reasoning" process, as currently trained on relevance datasets (like MS MARCO), focuses on query-document matching and does not inherently incorporate demographic awareness.

Implications:

1. Limitations of Reranking: Reranking alone cannot fix fairness issues if the initial candidate pool lacks demographic diversity. If the top 500 documents from the initial retrieval are geographically skewed, the reranker cannot introduce diversity that does not exist in the pool.
2. Need for Upstream Intervention: Improving search fairness requires interventions beyond the reranking stage, such as diversifying document collections, auditing representational gaps, and designing retrieval strategies that actively surface underrepresented perspectives.
3. Query Formulation: Natural language query rewriting is more impactful for overall system effectiveness (relevance) than the choice between reasoning and non-reasoning architectures.

Future Work:
The authors suggest that future research must focus on steering reasoning models toward fairness-aware outcomes through specific training objectives or prompts, rather than relying on the emergent properties of general reasoning. Additionally, fairness metrics must evolve beyond exposure-based measures (AWRF) to include calibration and intersectional fairness.