LLMs Are Not a Silver Bullet: A Case Study on Software Fairness

1. Problem Statement

Software fairness is a critical requirement for high-stakes decision-making systems (e.g., credit scoring, hiring, criminal justice). While traditional Machine Learning (ML) methods have been extensively studied for bias mitigation in tabular data settings, the rapid rise of Large Language Models (LLMs) has led to their adoption for similar tasks. Recent studies suggest LLMs, particularly those using in-context learning (ICL), can effectively mitigate bias by manipulating prompt demonstrations.

However, there is a lack of direct, large-scale empirical comparison between state-of-the-art traditional ML methods and LLM-based approaches. It remains unclear whether LLMs offer genuine advantages over established ML techniques or if reported successes are artifacts of specific evaluation settings. This gap leaves software engineers without clear guidance on whether to adopt LLMs for fairness-critical tabular applications.

2. Methodology

The authors conducted a large-scale empirical study comparing 8 traditional ML-based bias mitigation methods against 8 LLM-based methods across 6 fairness evaluation tasks derived from three real-world datasets: Adult, Compas, and Credit.

• Datasets & Tasks: The study utilized real-world tabular data with sensitive attributes (sex, race, age) to define six tasks (e.g., Adult-Sex, Compas-Race).
• Traditional ML Baselines: Evaluated four base models (Logistic Regression, Random Forest, SVM, Deep Neural Network) combined with eight mitigation strategies spanning:
  • Pre-processing: FairMask, LTDD, CoT.
  • In-processing: ADV (Adversarial Learning).
  • Post-processing: EOP, ROC.
  • Ensemble: MAAT, MirrorFair.
• LLM Approaches: Used GPT-4o-mini as the primary model. Evaluated zero-shot prompting and eight ICL strategies, including random demonstration selection, label flipping, distribution-controlled few-shot (balanced/minority-balanced/minority-unbalanced), and genetic search for fairness-aware demonstrations (FCG).
• Evaluation Metrics:
  • Fairness: Statistical Parity Difference (SPD), Average Odds Difference (AOD), Equal Opportunity Difference (EOD).
  • Performance: Accuracy, Precision, Recall, F1-score.
• Experimental Variations:
  • RQ2: Tested four advanced LLMs (GPT-5, Gemini-2.5, DeepSeek-v3.2, Qwen3-Max) to check if model strength changes the outcome.
  • RQ3: Compared balanced test sets (artificially equalized demographics) vs. random test sets (real-world imbalanced distributions) to analyze evaluation sensitivity.
  • RQ4: Investigated Supervised Fine-Tuning (SFT) on the full training set (with and without pre-processing) to see if full data access closes the performance gap.

3. Key Contributions

1. Large-Scale Comparative Study: The first comprehensive empirical comparison of SOTA ML and LLM bias mitigation methods on real-world tabular data.
2. Identification of Evaluation Artifacts: Demonstrated that prior positive results for LLMs are largely driven by artificially balanced test distributions rather than robust algorithmic superiority.
3. Analysis of Learning Paradigms: Showed that while fine-tuning improves LLM performance over ICL, it still fails to consistently outperform traditional ML, challenging the notion that LLMs are a universal upgrade.
4. Open-Source Replication: Released a full replication package including scripts and data to facilitate future research.

4. Key Results

RQ1: ML vs. LLM Performance

• Traditional ML consistently outperforms LLMs. Across all six tasks, ML-based methods achieved significantly better fairness and predictive performance.
  • Fairness: ML reduced bias metrics (SPD, EOD, AOD) by 48.3% – 59.6% compared to LLMs.
  • Performance: ML achieved 11.1% higher Accuracy and 9.3% higher F1-score on average.
  • Statistical Significance: ML was significantly better in 72.2% of fairness comparisons and 83.3% of performance comparisons.

RQ2: Impact of Model Strength

• Replacing GPT-4o-mini with stronger, reasoning-oriented models (e.g., Qwen3-Max, GPT-5) did not change the conclusion. Traditional ML remained superior in 82.1% of comparisons. Scaling up LLM size or reasoning capability is insufficient to overcome the limitations in tabular bias mitigation.

RQ3: Sensitivity to Evaluation Settings

• Balanced vs. Random Test Sets: Prior studies often use balanced test sets. The authors found that using balanced test sets artificially inflated fairness improvements for LLMs by 36.2% – 138.4%.
• When evaluated on realistic, imbalanced distributions (random test sets), the apparent fairness advantages of LLMs vanished, revealing their poor generalization to real-world data skew.

RQ4: Fine-Tuning vs. In-Context Learning

• Fine-Tuning (SFT) significantly improved LLM performance over ICL (reducing bias by ~40-50% and improving accuracy).
• However, SFT still lagged behind Traditional ML. Fine-tuned LLMs only showed significant advantages in 37.5% of performance comparisons and 0% of fairness comparisons against traditional ML.
• Cost: Fine-tuning introduced significant monetary costs (avg. $8.32/task) and training time, further reducing its practical viability compared to efficient ML methods.

5. Significance and Implications

• LLMs Are Not a Silver Bullet: The paper concludes that LLMs should not be assumed as a default replacement for traditional ML in fairness-critical tabular applications. Established ML techniques remain more effective, efficient, and reliable.
• Importance of Realistic Evaluation: The study highlights a critical flaw in prior research: evaluating fairness on balanced datasets creates misleading conclusions. Future research and practice must prioritize evaluation under realistic, imbalanced data distributions.
• Cross-Paradigm Evaluation: The findings underscore the necessity of comparing new paradigms (LLMs) directly against established baselines (ML) rather than evaluating them in isolation.
• Guidance for Practitioners: Software engineers should rely on evidence-driven method selection. Adopting LLMs for tabular fairness tasks requires rigorous validation, as current LLM-based approaches do not offer a clear advantage over traditional ML.

In summary, this paper serves as a reality check for the software engineering community, demonstrating that while LLMs are powerful, they do not currently solve the specific challenges of tabular bias mitigation better than traditional, well-established machine learning techniques.