🤖 machine learning

Auditing LLMs for Algorithmic Fairness in Casenote-Augmented Tabular Prediction

This technical report audits LLMs for algorithmic fairness in housing placement predictions, finding that fine-tuning models with augmented casenote summaries can simultaneously improve accuracy and reduce fairness disparities, while zero-shot classification introduces no additional textual biases.

Original authors: Xiao Qi Lee, Ezinne Nwankwo, Angela Zhou

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiao Qi Lee, Ezinne Nwankwo, Angela Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a social worker trying to help people experiencing homelessness find a permanent place to live. You have two types of information about each person:

The "Checklist" (Tabular Data): A list of facts like age, gender, how long they've been on the street, and whether they have a job application.
The "Story" (Casenotes): Short, messy notes written by outreach workers after talking to the person. These notes contain the rich, human details that the checklist misses.

For a long time, computers have been good at reading the Checklist to predict who gets housing. But now, we have Large Language Models (LLMs)—super-smart AI that can read the Story notes too. The big question is: If we let the AI read the stories, will it be fair to everyone, or will it accidentally treat some groups worse than others?

This paper is like a "safety inspection" report. The authors tested different AI setups to see if they could predict housing placement accurately without being unfair to specific groups (like women or Black people).

Here is the breakdown of their findings using simple analogies:

1. The "Smart Student" vs. The "Experienced Teacher"

The researchers tested two types of AI "students":

The Zero-Shot Student: This AI is like a brilliant student who has read the whole internet but has never seen this specific housing problem before. They are guessing based on general knowledge.
The Fine-Tuned Teacher: This AI is the same student, but they have been given a crash course (training) on the specific housing data. They know the rules of this specific game.

The Finding:

The Teacher (Fine-tuned) was much better at getting the right answer (accuracy).
BUT, the Teacher also became a bit "biased." Because there were way more men than women in the data (like a classroom with 4 boys for every 1 girl), the Teacher learned to guess "Male" outcomes more often, accidentally ignoring the needs of the women.
The Student (Zero-shot) was worse at guessing the right answer, but they were actually fairer because they hadn't been "trained" on the skewed data yet.

2. The "Highlighter" Effect (Adding Summaries)

The researchers tried feeding the AI a summary of the outreach worker's notes (the "Story").

For the Student (Zero-Shot): The summary was like a highlighter pen. It helped the student see the most important parts of the story, making them much smarter and more accurate.
For the Teacher (Fine-Tuned): The summary was a bit like noise. Since the Teacher already knew the rules, the extra text sometimes confused them, making their accuracy drop slightly.
The Surprise: Even though the Teacher's accuracy dropped a tiny bit with the summary, their fairness improved dramatically. The summary helped the Teacher understand the unique context of the women and minority groups, balancing out the bias they had learned from the raw numbers.

3. The "One-Feature" Trap

The researchers also tried telling the AI: "Hey, the most important thing to look at is 'Where did they live before 2019?'"

For the Student: This was helpful. It gave them a clear direction.
For the Teacher: This was dangerous. The Teacher had already learned that "Where they lived before" is a strong predictor, but it's also a feature that carries historical bias (because of past discrimination in housing). By forcing the Teacher to focus only on that one thing, they accidentally amplified the unfairness, making the AI ignore other important factors.

The Big Takeaway (The "Golden Rule")

Think of the AI like a new employee at a housing agency:

Don't just give them the numbers. If you only give them the checklist, they might miss the human story.
Don't just let them guess. If they haven't been trained on your specific data, they will be slow and inaccurate.
The Sweet Spot: The best approach found in this study was to train the AI (Fine-tuning) but also give it the summarized stories (Casenotes).
- This combination made the AI accurate and fair.
- The stories acted as a "reality check," reminding the AI that every person is unique, preventing it from just following the statistical majority (the men) and ignoring the minorities (the women).

In short: AI can be a powerful tool for social good, but it needs to be "audited" carefully. If you just let it learn from historical data, it might repeat past mistakes. But if you give it the human context (the stories) and train it carefully, it can actually help fix those inequalities.

1. Problem Statement

The paper addresses the growing interest in using Large Language Models (LLMs) for high-stakes social service prediction tasks, specifically housing placement for homeless individuals. While LLMs can process unstructured data (like street outreach casenotes) to augment traditional tabular data, their algorithmic fairness properties in this context are poorly understood.

The core problem is twofold:

Bias Propagation: LLMs are known to harbor societal biases (gender, race), and tabular models often exhibit disparate error rates across protected groups due to data imbalances. It is unclear if combining LLMs with tabular data exacerbates these disparities.
Data Utilization: Nonprofits possess rich, unstructured casenote data that is currently underutilized for predictive modeling due to its unstructured nature. The challenge is to leverage this text to improve prediction accuracy without introducing new forms of algorithmic bias.

2. Methodology

Dataset and Task

Context: A collaboration with a nonprofit street outreach organization.
Data: 471 clients observed consistently from 2019–2021.
Task: Multi-class classification predicting the highest housing placement level achieved by 2021 ( $Y \in \{0, 1, 2, 3\}$ ), where 0 is street homelessness and 3 is permanent supportive housing.
Inputs:
- Tabular Covariates: Demographics (race, gender, age), engagement history, and binary indicators extracted from casenotes (e.g., service refusal, document possession).
- Unstructured Text: LLM-generated summaries of historical casenotes (heavily redacted, short, and narrative).
Models Evaluated:
- LLMs: Llama 3 variants (17B and 70B parameters) tested in Zero-Shot and Fine-Tuned settings.
- Baseline: Random Forest trained on tabular data alone.
Augmentation Strategies:
1. Casenote Summaries: Adding text summaries of client history to the prompt.
2. Feature Importance Cues: Explicitly prompting the model with the most predictive tabular feature (identified via Random Forest as "max housing placement before 2019").

Fairness Evaluation Metrics

Since the outcome is multi-class, the authors used multiclass extensions of standard metrics across race (White, Black, Other) and gender (Male, Female) subgroups:

Statistical Parity (SP): Measures the independence of predicted outcomes from group membership.
Equality of Opportunity (EoO): Measures whether groups achieve equal True Positive Rates (TPR) for each class.
Metrics Reported: Mean and Max deviations ( $SP_{Mean}, SP_{Max}, EoO_{Mean}, EoO_{Max}$ ).

3. Key Contributions

Empirical Audit: The first audit of LLM-based tabular classification for housing placement using real-world social service casenotes, benchmarking 17B/70B Llama models against a Random Forest baseline.
Fairness-Accuracy Trade-off Analysis: Demonstrated that while fine-tuning improves accuracy, it can amplify subgroup disparities, particularly for minority groups (e.g., women).
Impact of Text Augmentation: Showed that adding casenote summaries can reduce fairness gaps even when accuracy gains are marginal, suggesting text can act as a fairness regularizer.
Feature Importance Risks: Revealed that explicitly highlighting the "most important" historical feature can reinforce historical biases, whereas summaries can help mitigate them.

4. Key Results

A. Performance Without Summaries

Accuracy: The Fine-Tuned 70B model achieved the highest accuracy (70.2%), significantly outperforming Zero-Shot models (~23–33%) and the Random Forest baseline (61.2%).
Fairness: The Fine-Tuned 70B exhibited the worst fairness disparities (highest EoO Mean/Max). The fine-tuning process appeared to amplify biases present in the imbalanced dataset (4:1 male-to-female ratio), leading to poorer performance for minority groups.

B. Impact of Casenote Summaries

Zero-Shot Models: Adding summaries improved accuracy significantly (up to +13.9% for the 17B model) by providing context that pre-trained models lacked.
Fine-Tuned Models: Summaries had a neutral-to-negative effect on accuracy for the 70B model (likely due to noise) but significantly improved fairness.
- Key Finding: The Fine-Tuned 70B with summaries reduced $EoO_{Mean}$ from 0.071 (baseline) to 0.021, outperforming the Random Forest baseline in fairness while maintaining high accuracy.
Model Size Interaction: Smaller models (17B) handled noisy zero-shot inputs better than the 70B, but the 70B leveraged summaries more effectively after fine-tuning to improve fairness.

C. Feature Importance Cues

Zero-Shot: Adding the top predictive feature ("max placement before 2019") boosted accuracy but worsened fairness (increased EoO disparities). This suggests the feature correlates with historical systemic biases (race/gender) that the model then amplifies.
Fine-Tuned: Explicitly emphasizing the top feature in a fine-tuned model provided no accuracy gain and disrupted the model's learned internal weighting, leading to severe fairness degradation (EoO Max rose to 0.470 in the 70B+Summary+Feature setting).
Layering Features: Adding more features (V2–V11) did not consistently improve fairness compared to using only the top feature.

D. Subgroup Analysis (TPR)

Gender: Female True Positive Rates (TPR) were consistently lower than males, especially in fine-tuned models, reflecting the skewed 4:1 data distribution. Summaries helped narrow this gap.
Race: Racial disparities were less pronounced than gender disparities, likely due to a more uniform distribution of samples across racial groups.
Class Bias: Zero-shot models tended to over-predict "Class 2" (shelter/transitional housing), a "safe" middle-ground prediction, until fine-tuning specialized them to the task.

5. Significance and Recommendations

Text as a Mitigation Tool: Contrary to the fear that adding text introduces new biases, the study finds that casenote summaries can mitigate algorithmic fairness disparities in tabular prediction. The unstructured text provides nuance that helps the model treat disadvantaged groups more equitably, even if it doesn't always boost raw accuracy.
Caution with Fine-Tuning: Fine-tuning on imbalanced social service data can drastically worsen outcomes for minority groups. Fairness metrics must be monitored alongside accuracy.
Avoid "Top Feature" Prompting in Fine-Tuned Models: While helpful for zero-shot models, explicitly telling a fine-tuned model to focus on a historically predictive feature can override its learned balance and reintroduce historical biases.
Implementation Guidance: For high-stakes social services, the optimal approach identified is Fine-Tuning + Casenote Summaries. This combination leverages the LLM's ability to interpret nuanced text to smooth out disparities while maintaining high predictive performance, offering a low-burden implementation path for nonprofits.

Conclusion: The paper concludes that LLMs do not necessarily introduce new textual biases beyond existing tabular biases in this specific context (due to redacted, short notes). Instead, when used correctly (specifically with summaries and careful fine-tuning), they offer a unique advantage over traditional tabular models by effectively incorporating unstructured context to improve both accuracy and fairness.