💬 NLP

"Not in My Backyard": LLMs Uncover Online and Offline Social Biases Against Homelessness

This paper introduces a novel multi-domain corpus and audit protocol for tracking social biases against homelessness across online and offline discourse, revealing that while large language models can identify such biases, they suffer from significant miscalibration that systematically over-flags "NIMBY" sentiments and under-detects factual claims.

Original authors: Jonathan A. Karr, Benjamin F. Herbst, Matthew L. Sisk, Xueyun Li, Ting Hua, Matthew Hauenstein, Georgina Curto, Nitesh V. Chawla

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jonathan A. Karr, Benjamin F. Herbst, Matthew L. Sisk, Xueyun Li, Ting Hua, Matthew Hauenstein, Georgina Curto, Nitesh V. Chawla

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, chaotic town square where millions of people shout their thoughts, complaints, and stories every second. Now, imagine that some of these shouts are about a very serious, painful problem: people who have nowhere to sleep. This is the world of "homelessness," a situation where people lack a safe place to live. In this digital town square, people argue about where to build shelters, whether homeless people deserve help, or why they are even there. Sometimes, these arguments are helpful; other times, they are filled with prejudice, fear, or mean-spirited stereotypes.

Scientists who study language on computers (a field called Natural Language Processing) have built "smart robots" called Large Language Models (LLMs). Think of these robots as super-fast librarians who can read millions of books, tweets, and forum posts in a blink. Their job is often to act as a "bias detector," scanning the noise to find specific types of mean or unfair speech. But here's the catch: just because a robot is fast and has read a lot doesn't mean it understands the feeling or the context of what it's reading. It might mistake a question for an attack, or think that mentioning a house means someone hates homeless people. This paper asks a crucial question: Can these digital librarians actually tell the difference between a genuine complaint and a hateful bias, or are they just guessing based on the wrong clues?

The "Not in My Backyard" Mix-Up

In this study, a team of researchers from the University of Notre Dame and the United Nations University decided to test these smart robots on a very specific, sticky topic: the bias against people experiencing homelessness (PEH). They wanted to see if the robots could accurately spot when people were saying things like "Not In My Backyard" (NIMBY)—a fancy way of saying, "I don't want a homeless shelter built near my house."

To do this, they built a massive digital library. They gathered over 50,447 pieces of text from ten different U.S. cities, pulling from four very different places: Reddit threads, X (formerly Twitter) posts, news articles, and actual transcripts from city council meetings where real people argue about policy. They covered a whole decade, from 2015 to 2025.

But a library is only as good as its catalog. So, the researchers didn't just let the robots label everything. They brought in human experts—people who work with homeless populations in non-profit organizations—to read a carefully selected sample of 1,698 posts. These humans acted as the "gold standard," the truth-tellers who decided what each post actually meant. They used a detailed checklist with 16 different categories to label the text, ranging from "expressing an opinion" to "harmful generalizations" and, of course, "Not In My Backyard."

The Robot's Big Mistake

The researchers then asked six different smart robots (including big names like GPT-4.1, Gemini, and LLaMA) to read the same posts and guess the labels. They expected the robots to be pretty good, maybe getting about 43% of the tricky labels right (a score called "macro-F1"). And sure enough, the robots looked okay on paper. They seemed to be doing a decent job.

But then, the researchers looked closer, and that's when they found the glitch.

They discovered that the robots were suffering from a massive "calibration error." Imagine a smoke detector that is so sensitive it screams "FIRE!" every time you toast a piece of bread. That's what happened here. Every single robot the team tested was way too eager to label a post as "Not In My Backyard."

On average, the robots over-claimed NIMBY bias by 11.5 percentage points compared to the human experts. In plain English: if a human said, "This person is just stating a fact," the robot often shouted, "This person is a NIMBY!"

For example, if someone posted, "They've got all that affordable housing, but the downside is it's in [City]," the humans labeled this as just "stating a fact" or "expressing an opinion." But four out of the six robots immediately tagged it as "Not In My Backyard." Why? Because the robots got tricked by two simple clues:

Housing words: If the text mentioned "housing" or "affordable housing," the robot assumed someone was against it.
Question marks: If the post was a question, the robot assumed it was a sarcastic, rhetorical attack.

The robots were essentially playing a game of "guess the bias" based on a shallow dictionary of words, rather than actually understanding the meaning. They treated the mention of housing as proof of opposition to housing.

The "Fact" Blind Spot

The robots didn't just over-claim bias; they also missed the truth. The study found that the robots were terrible at spotting when someone was actually providing a fact or a claim. They under-detected factual statements by a whopping 30.5 percentage points.

This is a dangerous mix. If you use these robots to monitor public opinion for a city council, you would end up with a distorted picture of reality. You would think the community is much more hostile and opposed to shelters than they actually are (because of the fake NIMBY alarms), and you would think they are offering much less evidence-based information than they really are (because the robots ignored the facts).

What This Means for the Future

The researchers didn't just point out the problem; they offered a way to fix it. They realized that you can't just trust the robots' raw numbers. Even if a robot says it's "80% accurate," it might still be lying about how many people are biased.

The paper suggests that if cities or organizations want to use these AI tools to track stigma against homeless people, they must run a "prevalence audit." This means constantly checking the robot's work against a small, trusted group of human experts to see if the robot is inflating the numbers. They also found that smaller, local robots (like the ones you could run on your own computer for privacy) were even worse at this than the big, expensive ones.

In the end, this paper is a warning label for the digital age. It shows that while AI is a powerful tool for listening to the world, it is currently a very clumsy listener. It hears the words "housing" and "question mark" and jumps to the conclusion that someone is being mean. Until we teach these robots to understand the nuance between a fact and a fear, we can't let them run the show on their own. We need to keep the human experts in the loop to make sure we aren't just listening to the robot's imagination.

Technical Summary: "Not in My Backyard": LLMs Uncover Online and Offline Social Biases Against Homelessness

1. Problem Statement

Homelessness remains a persistent global challenge, with over 876,000 people experiencing homelessness (PEH) recorded in the U.S. in 2025. A significant barrier to alleviating this crisis is social bias, which dehumanizes PEH, shapes public perception, and influences policymaking, often leading to the rejection of shelters and services. While online textual media (Reddit, X/Twitter, news) and offline city council discourse reflect and influence public opinion, existing resources lack the capacity to reliably monitor these biases across multiple domains.

Previous work, such as OATH (Ranjit et al., 2024), provided expert annotations for X (Twitter) but was limited to a single platform and keyword, failing to audit label-prevalence shifts when scaling with Large Language Models (LLMs). Municipal partners require more than simple overlap scores; they need to know if automated monitors systematically inflate or deflate specific stigma categories (e.g., "Not In My Backyard" or NIMBY) that drive public backlash.

2. Methodology

2.1 Corpus Construction

The authors constructed the first multi-domain PEH bias corpus comprising 50,447 texts collected from 2015 to 2025 across ten U.S. cities. The cities were selected in two kNN-matched clusters (five smaller, five larger) based on population, homelessness rates, and Gini index. Data sources include:

Reddit: Posts and comments.
X (formerly Twitter): Non-repost posts.
News: Articles.
City Council Meetings: Transcripts (verbatim for some cities; audio/video transcribed via Whisper for others; agenda summaries excluded).

Text was retrieved using an 11-term PEH lexicon (e.g., "homeless," "affordable housing," "unhoused") and anonymized using spaCy and pydeidentify to remove PII. For X data, where geolocation is available for <3% of posts, a multi-stage geolocation pipeline inferred user locations, successfully locating approximately 80% of users in the 2025 subsample.

2.2 Taxonomy and Annotation

The study formalizes PEH bias detection as a 16-category multi-label classification task. The taxonomy extends the nine frames from OATH (Ranjit et al., 2024) to include platform-specific nuances (e.g., distinguishing genuine vs. rhetorical questions, fact/claim vs. observation).

A Negative Bias Frame was defined for shelter-siting monitoring, containing five specific categories:

Rhetorical question
Not in my backyard (NIMBY)
Harmful generalization
Deserving/undeserving
Racist

A stratified gold-standard set of 1,698 items was created, annotated by three raters trained in collaboration with non-profit partners. To handle subjectivity, the authors employed soft labeling (Fornaciari et al., 2021), where the reference label for a category is determined by a majority vote (2-of-3) among annotators. The remaining corpus was labeled using GPT-4.1 with few-shot prompts, serving as "teacher" annotations rather than ground truth.

2.3 Evaluation Protocol

The study benchmarks six prompted LLMs (zero-shot and few-shot) against the gold-standard set:

Models: GPT-4.1, Gemini 2.5 Pro, Grok-4, LLaMA 3.2 3B, Qwen 2.5 7B, and Phi-4 Mini.
Metrics: Macro-F1 score and a novel prevalence-gap audit. The prevalence gap ( $\hat{\pi}_{model} - \pi_{ref}$ ) measures the systematic shift in the positive rate of a category compared to the gold standard, revealing whether a model over- or under-tags specific biases.

3. Key Results

3.1 Benchmark Performance

Macro-F1: Closed-source APIs (GPT-4.1, Gemini 2.5 Pro) achieved moderate performance with an average macro-F1 of approximately 43-44% (few-shot). Local models (LLaMA, Phi-4, Qwen) scored lower, averaging around 32%. Few-shot prompting provided marginal improvements but did not close the gap between local and closed models.
Moderate F1 vs. Systematic Error: While F1 scores suggested viability, they masked significant calibration issues.

3.2 Prevalence Gaps and Miscalibration

The prevalence-gap audit revealed systematic miscalibration across all models:

NIMBY Over-tagging: Every model over-tagged the "Not in my backyard" category. The pooled average gap was +11.5 percentage points (pp).
Factual Claim Under-detection: Every model under-detected "provide fact/claim" labels, with a pooled gap of -30.5 pp.
Model Variance: While Qwen trailed GPT-4.1 by ~10 F1 points, its NIMBY over-tagging was 7 times worse (+27.4 pp vs. +4.0 pp).

3.3 Error Analysis

Analysis of consensus false positives (items where $\ge$ 3 of 6 zero-shot models incorrectly tagged NIMBY) identified specific failure modes:

Lexical Proxies: Models treated the mere presence of housing/homeless vocabulary (89.7% in false positives vs. 72.2% in true negatives) and question marks (32.5% vs. 9.7%) as proxies for resident opposition.
Lack of Intent Detection: Models failed to distinguish between texts that discuss housing (e.g., "They've got all that affordable housing, but the downside is it's in [City]") and texts that oppose housing siting.
Consequence: This leads to a distortion where communities appear more opposed to shelters and less evidence-based than they actually are.

4. Contributions

Multi-Domain Corpus: Release of a 50,447-item corpus spanning Reddit, X, news, and city council transcripts from ten U.S. cities, collected via an 11-term PEH lexicon.
Gold-Standard Reference: A 1,698-item stratified set with a 16-category multi-label taxonomy, annotated by domain experts and non-profit partners, utilizing soft-labeling to handle subjectivity.
Prevalence-Gap Audit Protocol: A methodology that complements F1 scores with systematic calibration checks to detect systematic shifts in label prevalence, crucial for monitoring tools.
LLM Benchmarking & Error Analysis: Demonstration that moderate F1 scores coexist with large miscalibrations, specifically identifying that models use housing vocabulary and question forms as NIMBY proxies.

5. Significance and Claims

The paper claims to provide the first multi-domain NLP resource for monitoring PEH stigma, co-designed with non-profit partners. Its primary significance lies in demonstrating that accuracy (F1) is insufficient for bias monitoring; without prevalence-gap audits, LLMs can systematically distort the reality of public opinion, making communities appear more hostile to shelters than they are.

The authors emphasize that the corpus and audit protocol offer municipalities a reproducible foundation for stigma monitoring. They caution that LLM-labeled dashboards should not be treated as ground truth without recurring audits against a gold-standard reference set. The work supports the deployment of "directional monitoring" tools, provided they are paired with prevalence-gap caps and regular human-in-the-loop validation to prevent the amplification of false opposition signals.

Limitations Noted by Authors:

The gold standard is a reference for auditing, not an infallible truth on subjective bias.
Rare categories (e.g., "racist") have too few positives in the gold set for stable prevalence-gap calculations.
The corpus is limited to ten U.S. cities and English text, not representing all global or socio-economic contexts.
Bot activity in social media was not verified.