💬 NLP

OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants

This paper introduces OATH-Frames, a nine-category typology for analyzing online attitudes toward homelessness in the U.S., and demonstrates that leveraging large language models to assist domain experts can scale the annotation of 2.4 million Twitter posts with a 6.5x speedup and minimal performance loss, yielding nuanced insights into public sentiment across different demographics and regions.

Original authors: Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn, Laura Petry, Olga Koumoundouros, Jayne Bottarini, Peichen Liu, Eric Rice, Swabha Swayamdipta

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn, Laura Petry, Olga Koumoundouros, Jayne Bottarini, Peichen Liu, Eric Rice, Swabha Swayamdipta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand how millions of people feel about a complex, emotional issue like homelessness. If you tried to read every single tweet about it, you'd be buried under a mountain of text faster than you could blink. It's like trying to drink from a firehose.

This paper, titled "OATH-Frames," is about building a better pair of glasses to see what's actually in that firehose. The researchers used a mix of human social work experts and powerful AI (Large Language Models) to sort through 2.4 million tweets about homelessness in the U.S.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "One-Size-Fits-All" Glasses

Before this study, if you wanted to analyze tweets about homelessness, you might use standard tools that just ask: "Is this tweet mean?" (Toxicity) or "Is this tweet happy or sad?" (Sentiment).

The researchers argue these tools are like trying to sort a mixed bag of fruit using only a "Red" and "Green" sticker.

The Flaw: A tweet might say, "Homeless people are lazy and dirty." Standard tools might miss this because it's not using "swear words" (low toxicity) and it's not "happy" (negative sentiment). But it is deeply harmful.
The Reality: People's attitudes are messy. You can feel sympathy for a homeless person and be angry at the government for not fixing it, all in the same sentence. Simple "good/bad" labels miss this nuance.

2. The Solution: The "OATH-Frames" Menu

The team created a new "menu" of nine specific categories (frames) to describe exactly how people are talking about homelessness. Think of this like a chef's tasting menu instead of just "food."

They grouped these nine categories into three main flavors:

Critiques (The Complaints):
- Government Critique: Blaming politicians or laws.
- Societal Critique: Blaming society or "hypocritical" people.
- Money Aid: Arguing about where tax dollars should go.
Perceptions (The Stereotypes):
- Harmful Generalizations: Calling all homeless people thieves, addicts, or lazy.
- Deserving vs. Undeserving: Arguing that some groups (like veterans) deserve help more than others (like immigrants).
- Not In My Backyard (NIMBY): "I don't mind homeless people, just not in my neighborhood."
Responses (The Actions):
- Solutions: Suggesting fixes, shelters, or policy changes.
- Personal Interaction: Sharing a story about meeting a homeless person.
- Media Portrayal: Talking about how the news or TV shows homelessness.

3. The Process: The "Human-AI Dance"

Sorting 2.4 million tweets by hand would take a human team years. Sorting them with AI alone is risky because AI can misunderstand subtle social cues (like sarcasm or specific political references).

So, they invented a "Human-AI Dance":

The Experts: Social work experts first defined the rules (the "menu") and labeled a small batch of tweets to teach the AI what to look for.
The AI Assistant (GPT-4): The AI tried to label thousands of tweets. When it got confused, it explained its reasoning (like a student showing their work).
The Check-Up: The human experts looked at the AI's "work" and fixed the mistakes. They found that by letting the AI do the heavy lifting and just checking the results, they got 6.5 times faster at labeling, with only a tiny drop in accuracy.
The Final Stretch: Once the AI learned the ropes from the experts, they trained a smaller, faster AI model to label the remaining 2.4 million tweets on its own.

4. What They Found: The Hidden Patterns

Once they had all 2.4 million tweets sorted into these nine categories, they could see patterns that were previously invisible.

The "Toxicity" Blind Spot: They found that many tweets containing harmful stereotypes (like calling homeless people "dirty") had low "toxicity" scores. The AI's new "OATH-Frames" caught these, while the old tools missed them.
Location Matters:
- In California, Washington, and Oregon, people were more likely to use Harmful Generalizations (stereotypes). The researchers think this is because there are more visible, unsheltered homeless populations there, leading to more direct (and sometimes negative) reactions.
- In New York, people were more likely to argue about Deserving vs. Undeserving groups. This seemed linked to news about migrants and asylum seekers arriving in the city, sparking debates about who "deserves" aid.
The "Ukraine vs. Immigrant" Split:
- When people compared homeless Americans to Ukrainians, the tweets were mostly about Government Critique and Money Aid (e.g., "Why are we sending billions to Ukraine when we have homeless people here?").
- When people compared homeless Americans to Immigrants, the tweets were much more hostile, filled with Harmful Generalizations and NIMBY attitudes (e.g., "They are taking our jobs/housing").
Time Matters: During big news events (like the Russian-Ukraine war funding debates), the tweets spiked in specific categories. When Congress was discussing aid for Ukraine, the "Money Aid" and "Deserving vs. Undeserving" arguments exploded in the tweets.

The Bottom Line

This paper didn't just count tweets; it taught a computer how to understand the nuance of human anger, sympathy, and confusion regarding homelessness.

By combining human social work expertise with AI speed, they created a tool that can spot subtle, harmful attitudes that standard "mean vs. nice" detectors miss. They showed that how we talk about homelessness changes depending on where we live, who we are comparing them to, and what's happening in the news right now.

Note: The paper focuses entirely on analyzing these online attitudes to help researchers and advocacy groups understand public opinion better. It does not claim to solve homelessness directly or offer clinical advice, but rather provides a clearer lens to see the problem.

Problem Statement
Public attitudes toward complex societal issues, such as homelessness in the U.S., are valuable for policy and reform but are difficult to understand at scale. Traditional social science methods, including ethnographic studies and qualitative surveys, are limited in scope and cannot process the volume of online discourse generated on platforms like Twitter. While Large Language Models (LLMs) offer a potential solution for scaling social science research, their reliability in processing nuanced, latent attitudes toward sensitive topics remains an open question. Existing sentiment and toxicity classifiers often fail to capture the subtle, harmful stereotypes and specific framing of homelessness, particularly when such rhetoric is masked by humor, sarcasm, or subtle generalizations.

Methodology
The authors propose a collaborative framework involving social work experts and LLMs to characterize online attitudes. The methodology consists of three primary components:

Frame Discovery: Using grounded theory and framing theory, four domain experts analyzed a sample of 3.1 million Twitter posts (filtered from an initial 5 million) to develop a novel typology called OATH-Frames (Online Attitudes Towards Homelessness). This typology categorizes attitudes into three overarching themes—Critiques, Perceptions, and Responses—which are further divided into nine fine-grained, issue-specific frames:
- Critiques: Government Critique, Societal Critique, Money Aid Resource Allocation.
- Perceptions: Harmful Generalization, Deserving vs. Undeserving, Not in My Backyard (NIMBY).
- Responses: Solutions/Interventions, Personal Interaction, Media Portrayal.
Annotation Strategies: The authors employed three distinct annotation approaches to evaluate scalability and accuracy:
- Expert-Only: Domain experts annotated 4,100 posts, achieving a macro F1 of 0.55 and a Fleiss' $\kappa$ of 0.46.
- LLM-Assisted Expert (Expert+GPT-4): This pipeline involved GPT-4 generating initial predictions with Chain-of-Thought (CoT) reasoning, which were then validated and corrected by experts. Experts used OATH-Guidelines to refine prompts and filter out false positives. This approach resulted in a 6.5× speedup in annotation time compared to experts working alone, with only a 3-point reduction in macro F1 (0.52 vs. 0.55).
- Model Prediction: A Flan-T5-Large model was fine-tuned on the combined expert and expert+LLM datasets (approx. 10.4k examples) to predict frames on a large-scale dataset of 2.4 million posts. This model achieved a macro F1 of 0.50, performing comparably to the expert+LLM setting.
Analysis: The authors utilized the 2.4M predicted annotations to analyze attitudes across states, time periods, and comparisons with other vulnerable populations. They also compared OATH-Frames against standard toxicity and sentiment classifiers.

Key Results

Annotation Efficiency: The Expert+GPT-4 pipeline significantly reduced the time required for annotation (from ~187 seconds to ~29 seconds per post) while maintaining high accuracy, demonstrating a viable trade-off for scaling socially sensitive tasks.
Limitations of Existing Classifiers: The study found that standard toxicity and sentiment classifiers fail to identify harmful rhetoric against people experiencing homelessness (PEH). A majority of posts labeled with the Harmful Generalization frame had toxicity scores below 0.5, and sentiment distributions for harmful and non-harmful posts were statistically similar.
Sociopolitical Trends:
- Geographic Variation: Harmful Generalization was most prominent in posts about California, Washington, and Oregon, correlating with higher populations of unsheltered PEH. In contrast, Deserving vs. Undeserving and Money Aid were more prominent in New York, correlating with discussions on immigration and asylum seekers.
- Temporal Correlation: Attitudes spiked in December 2022, coinciding with U.S. Congressional debates on aid for Ukraine. This period saw increased Government Critique and Deserving vs. Undeserving frames, where funding for Ukraine was contrasted with the domestic homelessness crisis.
- Population Comparisons: When PEH were compared to Ukrainians, public discourse focused on Government Critique and resource allocation. Conversely, comparisons between PEH and immigrants elicited NIMBY attitudes and Harmful Generalizations.

Significance and Claims
The paper claims to provide a generalizable framework for understanding nuanced public attitudes at scale on socially sensitive issues. By introducing OATH-Frames, the authors demonstrate that modeling specific framing typologies offers distinct advantages over coarse-grained sentiment and toxicity analysis, particularly in detecting subtle stereotyping and harmful rhetoric against marginalized populations. The work serves as a blueprint for integrating social science expertise with LLM capabilities to accelerate the analysis of public opinion, enabling advocacy groups and policymakers to gain deeper insights into complex social issues like homelessness. The authors explicitly state that their data and code will be made publicly available to support further research in this domain.

1. The Problem: The "One-Size-Fits-All" Glasses

2. The Solution: The "OATH-Frames" Menu

3. The Process: The "Human-AI Dance"

4. What They Found: The Hidden Patterns

The Bottom Line

More like this