Large language models can disambiguate opioid slang on social media

This paper demonstrates that large language models significantly outperform traditional lexicon-based strategies in accurately disambiguating ambiguous opioid slang and identifying relevant social media posts across lexicon-based, lexicon-free, and emergent slang scenarios, thereby enhancing the monitoring of the opioid overdose crisis.

The Gist

Imagine you are trying to find a few specific, rare needles hidden inside a massive, chaotic haystack the size of a city. This is the challenge researchers face when trying to monitor the opioid crisis on social media.

Every day, millions of people post on Twitter, Reddit, and other platforms. Most of these posts are about cats, weather, or dinner. Only a tiny fraction mention drugs. To find the "needles" (posts about opioids), researchers used to use a keyword list (a lexicon). They would say, "If a post contains the word 'heroin' or 'fentanyl,' keep it."

But here's the problem: Drug users often use slang to hide from police and algorithms. They might call heroin "smack" or fentanyl "fenty." The trouble is, these words have innocent meanings too. "Smack" can mean hitting a drum; "fenty" can mean a makeup brand; "lean" can mean being skinny.

If you just search for those words, you get thousands of false alarms (haystack noise) and miss the real needles because the slang changes faster than your list can be updated.

The New Solution: The "Super-Reader" AI

This paper introduces a new tool: Large Language Models (LLMs). Think of these AI models not as simple search engines, but as super-readers who have read almost everything on the internet. They don't just look for a specific word; they understand the context and the vibe of a sentence.

The researchers tested four of the smartest AI models available (GPT-4, GPT-5, Gemini, and Claude) to see if they could act as these super-readers. They set up three different "games" to test them:

Game 1: The "Ambiguous Word" Challenge

The Setup: The AI was given a list of posts containing tricky words like "fenty" or "smack."
The Task: Decide: Is this person talking about drugs, or are they just talking about makeup or hitting a drum?
The Result: The old keyword lists were terrible at this. They either missed almost everything or flagged thousands of innocent posts. The AI, however, acted like a detective. It looked at the whole sentence and said, "Ah, this 'smack' is about hitting a drum, but this other 'smack' is definitely about heroin." The AI was vastly more accurate than the old lists.

Game 2: The "No Clues" Challenge

The Setup: The AI was given a huge pile of random posts from New York and California. It wasn't told to look for any specific words.
The Task: Find the opioid posts from scratch.
The Result: This is where the AI really shined. The old keyword lists missed about 60-90% of the drug posts because they didn't have the right slang words on their list. The AI, using its "common sense" and understanding of how people talk, found almost all of them. It cast a wider net without getting confused by the noise.

Game 3: The "Fake Slang" Challenge

The Setup: The researchers took real drug posts and replaced the slang words with names of Pokémon (like "Charizard" or "Pikachu").
The Task: Could the AI figure out that even though the word was fake, the context still meant it was about drugs?
The Result: Yes! The AI realized that even if someone said, "I'm feeling the Charizard," in a context that usually describes being high, it was likely about drugs. This proves the AI isn't just memorizing a dictionary; it understands the situation.

Why This Matters

Think of the old method (keyword lists) as a bouncer with a strict guest list. If your name isn't on the list, you don't get in. But drug users just change their names (slang) to sneak in, or the bouncer accidentally lets in thousands of people who look like they're on the list but aren't.

The new method (AI) is like a very observant security guard. They don't need a list. They watch how people act, what they say, and who they are with. They can spot the suspicious behavior even if the person is wearing a disguise.

The Bottom Line

The study shows that these AI models are incredibly good at finding the "needles" in the haystack. They make fewer mistakes than the old keyword lists and catch many more drug-related posts that were previously invisible.

Important Note: The authors are very careful to say this tool is for public health monitoring (like a weather radar for a storm), not for spying on individuals. They want to use this data to help communities prevent overdoses and offer treatment, not to arrest people or censor their posts.

In short: AI is becoming a powerful new microscope for understanding the opioid crisis, helping us see the problem clearly so we can fix it.

Technical Summary

Here is a detailed technical summary of the paper "Large language models can disambiguate opioid slang on social media."

1. Problem Statement

Monitoring the opioid overdose crisis via social media offers real-time insights but faces a "needle in a haystack" challenge: the vast majority of social media text is unrelated to opioids.

• Limitations of Lexicon-Based Approaches: Current methods rely on fixed lists (lexicons) of opioid terms. However, slang terms for opioids (e.g., "smack," "blues," "lean") often have common non-drug meanings.
  • False Positives: Including ambiguous terms captures massive noise (e.g., searching "h" returns non-heroin posts).
  • False Negatives: Excluding ambiguous terms misses relevant content.
  • Rigidity: Lexicons cannot adapt to emergent slang, misspellings, or "algospeak" (terms designed to evade censorship) without constant manual updates.
• Limitations of Topic Modeling: Open-vocabulary approaches like Latent Dirichlet Allocation (LDA) are difficult to scale for specific labeling and often fail to clearly map topics to specific substances.
• The Gap: There is a need for a scalable, context-aware method to disambiguate slang terms and identify low-prevalence opioid-related content without relying on exhaustive pre-defined keyword lists.

2. Methodology

The authors evaluated four state-of-the-art Large Language Models (LLMs): GPT-4, GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro. They designed a pipeline and evaluated performance across three distinct tasks, comparing LLMs against six established benchmark lexicons (DEA, RedMed, Sarker, Graves, Yang, Chary).

A. Prompt Engineering

• Strategy: The team used an iterative prompting scheme. The model was first asked to reason through the context step-by-step before generating a final single-label classification ("opioid-related," "not opioid-related," or "unsure").
• Optimization: Prompts were optimized on a curated set of 100 tweets using GPT-4, setting the temperature to 0 for deterministic output. The final prompt instructed the model to act as an expert in online slang and social media culture.

B. Three Evaluation Tasks

1. Lexicon-Based Task (Disambiguation):

   • Dataset: September 2022 Twitter "Spritzer" dataset (1% random sample, ~100M tweets).
   • Focus: Tweets containing three ambiguous terms: "fenty" (fentanyl vs. makeup), "smack" (heroin vs. hitting), and "lean" (codeine drink vs. posture).
   • Goal: Determine if the LLM could correctly distinguish the drug-related usage from non-drug usage within the context of a specific post.
   • Labeling: Manual annotation of all "fenty" and "smack" tweets; stratified sampling of the massive "lean" dataset.

2. Lexicon-Free Task (Open-Vocabulary Classification):

   • Dataset: ~3 million geolocated tweets from New York and California (2007–2022).
   • Goal: Identify opioid-related posts without any pre-filtering or specific keyword prompts. The LLM must detect relevance based solely on context.
   • Labeling: Stratified sampling of ~4,062 tweets for manual ground truth.

3. Emergent Slang Task (Generalization):

   • Dataset: 80 opioid-related and 80 non-opioid tweets containing ambiguous slang.
   • Modification: Real slang terms were replaced with Pokemon names (e.g., replacing "fenty" with "Pikachu") to simulate novel emergent slang not present in training data.
   • Goal: Test if LLMs could infer opioid relevance based on context even when the specific keyword was unknown/fake.

3. Key Results

A. Lexicon-Based Task (Disambiguation)

• Performance: All four LLMs vastly outperformed lexicon-based strategies.
  • "Fenty" Subtask: LLM F1 scores ranged from 0.824 to 0.972, compared to the best lexicon strategy's F1 of 0.126.
  • "Smack" Subtask: LLM F1 scores ranged from 0.540 to 0.862, compared to the best lexicon strategy's F1 of 0.009.
• Precision vs. Recall: LLMs maintained high precision while significantly improving recall over the "exclude all" strategy.
  • GPT-4 and GPT-5 achieved near-perfect precision on "fenty" (1.00) with zero false positives.
  • Performance varied slightly by model and term, but all LLMs demonstrated superior ability to handle ambiguity compared to static lists.

B. Lexicon-Free Task

• Recall Improvement: LLMs achieved significantly higher recall (0.693–0.965) compared to lexicons (0.053–0.386).
• F1 Scores: LLMs achieved F1 scores of 0.544–0.769, surpassing lexicons (0.080–0.540).
• Case Studies:
  • LLMs correctly identified posts using brand names like "Norcos" (hydrocodone) which were missing from most lexicons.
  • LLMs successfully identified context-heavy references (e.g., "Actavis" referring to a cough syrup company) that lexicons missed.
  • Note: In one instance (Tweet 4), LLMs correctly identified a reference to "Actavis" as opioid-related, while the manual human labeler initially missed it, suggesting LLMs may sometimes be more accurate than human annotators regarding obscure context.

C. Emergent Slang Task

• Robustness: When slang terms were replaced with Pokemon names, LLMs maintained high precision (0.924–1.000) and recall (>0.500).
• Comparison: Lexicons failed completely on modified tweets (recall dropped to 0.100 or 0.000) because they rely on exact string matching.
• Implication: LLMs can generalize context to identify potential new slang terms even without prior exposure to the specific word.

4. Key Contributions

1. Demonstration of Contextual Disambiguation: Proved that LLMs can effectively resolve the ambiguity of opioid slang (e.g., distinguishing "lean" as a drink vs. a posture) at scale, a task where traditional NLP and lexicons fail.
2. Superiority Over Lexicons: Provided empirical evidence that LLMs outperform the six most common opioid lexicons in F1 score, recall, and adaptability across three different evaluation settings.
3. Generalizability: Showed that the performance is consistent across four different commercial LLMs (GPT-4, GPT-5, Claude, Gemini), suggesting this is a general capability of modern LLMs rather than a quirk of a single model.
4. Emergent Slang Detection: Demonstrated that LLMs can infer the relevance of novel terms based on context, offering a mechanism to detect emerging drug slang before it is added to official lexicons.

5. Significance and Implications

• Public Health Monitoring: This approach enables more accurate, real-time monitoring of the opioid crisis by capturing relevant data that current methods miss (high recall) while filtering out noise (high precision).
• Downstream Analysis: By providing cleaner, more comprehensive datasets, this method enhances the accuracy of predictive models for overdose trends and harm reduction strategies.
• Scalability: While LLM inference has a cost, the ability to pre-filter massive datasets automatically reduces the need for expensive manual review of millions of posts.
• Ethical Considerations: The authors emphasize that this tool should be used for population-level monitoring to inform public health policy. They explicitly oppose using LLMs for individual surveillance, profiling, or criminalization, noting that social media posts are not definitive proof of individual drug use.
• Broader Application: The framework is applicable to other low-prevalence, highly stigmatized topics (e.g., other illicit drugs, mental health crises) where slang and ambiguity are prevalent.

6. Limitations

• Cost: Commercial LLM API usage can be prohibitively expensive for massive datasets, though open-source alternatives exist (requiring significant GPU resources).
• Prompt Engineering: The study optimized prompts using GPT-4; other models might benefit from specific prompt tuning.
• Ground Truth: Manual labeling is imperfect; the study found instances where LLMs were likely more accurate than human annotators regarding obscure cultural references.
• Content Filters: Some tweets triggered content restriction errors, though these were predominantly non-opioid related (violence/sexual content).