🤖 AI

LLM-guided Semi-Supervised Approaches for Social Media Crisis Data Classification

This paper presents the first empirical evaluation of LLM-guided semi-supervised learning for classifying crisis-related tweets, demonstrating that methods like LG-CoTrain significantly outperform classical approaches in low-resource settings and can transfer LLM knowledge into compact, deployable models that rival zero-shot large language models.

Original authors: Jacob Ativo, Bharaneeshwar Balasubramaniyam, Anh Tran, Khushboo Gupta, Hongmin Li, Doina Caragea, Cornelia Caragea

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacob Ativo, Bharaneeshwar Balasubramaniyam, Anh Tran, Khushboo Gupta, Hongmin Li, Doina Caragea, Cornelia Caragea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, chaotic emergency situation, like a hurricane or a wildfire. During these events, millions of people post updates on social media. Some posts say, "My house is flooded!" (a rescue request), while others say, "The roads are clear" (a status update).

The problem? There are too many posts for humans to read in real-time. Computers need to sort them automatically. But here's the catch: to teach a computer to sort these posts, you usually need a huge stack of examples where humans have already labeled them (e.g., "This is a rescue request," "This is not"). In a real disaster, you don't have time to gather thousands of labeled examples before you need the system to work.

This paper is about a clever trick to teach computers how to sort these crisis posts using very few human examples, by borrowing the "brainpower" of a super-smart AI (a Large Language Model, or LLM).

Here is the breakdown of their experiment, explained simply:

The Cast of Characters

The "Student" (Small Models): These are standard, efficient computer programs (like BERTweet) that can be deployed quickly on a phone or server. They are smart, but they need training.
The "Teacher" (The LLM): This is a massive, super-intelligent AI (GPT-4o in this study). It knows a lot about language and can guess what a post means without ever being trained on the specific disaster. It's like a genius who has read every book in the library but has never seen a specific flood.
The "Classroom" (The Data): The researchers used data from 10 past disasters (like Hurricane Harvey or the Kerala Floods). They pretended to be in a "low-resource" classroom by giving the students only a tiny handful of labeled examples (5, 10, or 25 examples per category) and a huge pile of unlabeled posts.

The Experiment: Two New Teaching Methods

The researchers tested two new ways to use the "Genius Teacher" to help the "Student" learn from the unlabeled pile.

Method 1: VerifyMatch (The "Double-Check" System)
Imagine the Genius Teacher writes down an answer for a student's homework. But instead of just accepting it, a "Verifier" (another smart model) checks the answer. If the Verifier agrees with the Teacher, the student learns from it. If they disagree, the student ignores it. This prevents the student from learning the wrong answer just because the Teacher was confident but wrong.

Method 2: LG-CoTrain (The "Study Buddy" System)
This is the star of the show. Imagine two students (two small models) studying together.

The Genius Teacher gives them a hint (a "pseudo-label") on a hard problem.
Student A tries to solve it. Student B tries to solve it.
They compare notes. If they both agree with the Teacher's hint, they both learn from it.
They swap roles and help each other get better.
The Result: This method was incredibly good when the students had almost no human help (only 5 or 10 examples). It learned faster and better than any other method.

What They Found (The Results)

The "Low-Resource" Win: When the students had almost no human examples (5 or 10), the LG-CoTrain method was the clear winner. It learned so well that it actually performed better than the Genius Teacher (GPT-4o) working alone!
- Analogy: It's like a small, fast car that, with a little bit of coaching from a race car driver, ends up driving faster than the driver could have driven the car alone.
The "More Data" Shift: As the researchers gave the students more human examples (up to 50), the fancy new methods didn't need the Genius Teacher as much. A simpler, older method called Self-Training (where the student just guesses on its own and learns from its own best guesses) became the champion.
The "Calibration" Check: In a crisis, you don't just want the computer to be right; you want it to know how sure it is. If it says "99% sure this is a rescue request," it better be right.
- VerifyMatch was great at this. It was very honest about its confidence.
- LG-CoTrain was the most accurate but needed a bit more data to become very confident in its answers.

The Big Takeaway

The paper proves that you can take the "knowledge" of a massive, expensive AI and transfer it into a small, cheap, fast computer program.

If you have almost no data: Use the LG-CoTrain method. It uses the big AI to teach the small AI how to learn quickly.
If you have some data: You might not need the big AI at all; a simpler method works just fine.
If you need to know how sure the computer is: Use VerifyMatch or a method called AUM-ST-MixUp, which are very careful about their confidence levels.

Why This Matters for Disasters

In a real disaster, you might have 5 minutes to set up a system with only 10 labeled tweets from a volunteer. This research shows that you can use a big AI to "teach" a small, fast AI in those 5 minutes, creating a tool that can instantly sort thousands of incoming tweets to help rescue teams find people in need. It's a way to make high-tech AI practical and affordable for real-world emergencies.

Technical Summary: LLM-guided Semi-Supervised for Crisis Data Classification

Problem Statement

In disaster management, social media platforms provide a critical, bidirectional stream of information where authorities disseminate warnings and the public provides situational updates. However, effectively integrating these streams into real-time operations is hindered by information overload characterized by high volume, velocity, and varying veracity. While Machine Learning (ML) and Natural Language Processing (NLP) have been applied to automatically classify crisis-related tweets into actionable categories (e.g., infrastructure damage, rescue requests), supervised models typically require substantial amounts of high-quality human-labeled data. This data is often scarce in the time-sensitive context of disaster response.

Existing solutions include domain adaptation and traditional semi-supervised learning (SSL), which combine small labeled sets with large unlabeled sets via pseudo-labeling. However, the quality of pseudo-labels generated by standard "teacher" models in early training stages can be noisy, leading to error propagation. With the advent of Large Language Models (LLMs), there is potential to leverage their zero-shot or few-shot capabilities to generate higher-quality pseudo-labels for smaller, more deployable models. Yet, prior to this work, no empirical evaluation existed regarding LLM-guided SSL specifically within the crisis domain.

Methodology

The authors present the first empirical evaluation of LLM-guided semi-supervised learning for classifying crisis-related tweets. The study utilizes the HumAID dataset, a benchmark comprising 77,196 human-annotated tweets from 19 disaster events, categorized into 11 humanitarian classes. The experiments focus on 10 specific disaster events, simulating low-resource settings with labeled data budgets of 5, 10, 25, and 50 examples per class.

Baselines and Models

The study compares two LLM-guided SSL approaches against established classical SSL baselines and supervised limits:

Supervised Baselines:
- Lower Bound: BERTweet trained solely on the limited labeled set ( $D_L$ ).
- Upper Bound: BERTweet trained on the full labeled training split.
- Zero-Shot LLM: GPT-4o evaluated in a zero-shot setting (no task-specific training).
Classical SSL Baselines:
- Self-Training (ST), Uncertainty-Aware Self-Training (UST), MixMatch, AUM-based Self-Training (AUM-ST), AUM-ST-MixUp, and Confidence-based MixUp in Self-Training (Conf-ST-MixUp).
LLM-Guided SSL Approaches:
- VerifyMatch: Integrates LLM-generated pseudo-labels with a verifier model to cross-validate predictions before training, combined with confidence-aware MixUp to mitigate confirmation bias and overconfident errors.
- LLM-guided Co-Training (LG-CoTrain): A dual-model co-training framework where two classifiers iteratively learn from each other while incorporating pseudo-labels generated by GPT-4o. Unlike MixUp-based methods, LG-CoTrain retains all pseudo-labeled data without modification, relying on cross-model agreement for stability.

Experimental Setup

Encoder: All neural models utilize BERTweet as the backbone.
Pseudo-Labeling: GPT-4o is used to generate zero-shot pseudo-labels for the unlabeled training set ( $D_U$ ).
Metrics: Performance is evaluated using Macro-F1 (for balanced class performance) and Expected Calibration Error (ECE) (to measure the alignment between predicted probabilities and empirical correctness).
Hyperparameters: Tuned via Weights & Biases and Optuna, though the authors note potential instability in automated tuning for low-resource transformer fine-tuning.

Key Results

Performance in Low-Resource Settings

LG-CoTrain Superiority: In extremely low-resource settings (5 and 10 labeled examples per class), LG-CoTrain significantly outperforms all classical SSL baselines and the zero-shot GPT-4o baseline. It achieves the highest averaged Macro-F1 scores (0.608 at 5 lb/cl and 0.619 at 10 lb/cl).
VerifyMatch: Demonstrates competitive performance with classical SSL methods but exhibits stronger calibration properties (lower ECE), particularly at moderate label budgets (25–50 lb/cl).
Zero-Shot vs. SSL: While zero-shot GPT-4o performs competitively (Macro-F1 ~0.641), it is outperformed by LG-CoTrain in several specific disaster events (e.g., Hurricane Harvey, Irma, Maria) and in the most constrained settings. Notably, compact semi-supervised models can outperform very large LLMs in zero-shot settings in specific scenarios.

Impact of Label Quantity

Narrowing Gap: As the number of labeled examples increases (e.g., 50 lb/cl), the performance gap between LG-CoTrain and other methods narrows.
Self-Training Emergence: At 50 lb/cl, Self-Training (ST) emerges as a strong baseline, achieving the highest Macro-F1 (0.655), surpassing LG-CoTrain. This suggests that when sufficient labeled data is available, the marginal benefit of LLM-guided pseudo-labeling diminishes.

Calibration and Ablation

Calibration: AUM-ST-MixUp achieves the lowest ECE across most label budgets, indicating superior calibration. VerifyMatch also maintains low ECE. LG-CoTrain prioritizes accuracy in low-resource settings but shows improved calibration as more labeled data becomes available.
Ablation Study: An ablation study comparing LG-CoTrain (with GPT-4o labels) against Self-guided Co-Train (SG-CoTrain) (using BERTweet teacher labels) reveals a substantial performance gap. LG-CoTrain outperforms SG-CoTrain by an average of 0.212 Macro-F1 at 5 lb/cl. This confirms that the quality of pseudo-labels from a zero-shot LLM is significantly superior to those from a small model trained on extremely limited data.

Event-Specific Variations

Performance varies significantly across events. Models struggle with highly imbalanced events (e.g., Kerala Floods 2018, where one class dominates) and events with rare classes (e.g., Cyclone Idai). GPT-4o's zero-shot performance correlates with the quality of LG-CoTrain results; events where GPT-4o generates high-quality pseudo-labels yield better co-training outcomes.

Significance and Contributions

The paper claims the following contributions and implications:

First Empirical Evaluation: This work provides the first systematic evaluation of LLM-guided SSL for crisis-related tweet classification, benchmarking VerifyMatch and LG-CoTrain against established SSL methods on the HumAID dataset.
Practical Pathway for Deployment: The findings highlight a practical pathway for real-world disaster response. Specifically, LG-CoTrain offers a mechanism to transfer knowledge from powerful LLMs into smaller, more deployable models (like BERTweet) that can operate efficiently at scale with low latency.
Resource-Dependent Strategy: The study establishes that the optimal method depends on the available annotation budget:
- Extreme Scarcity (5–10 lb/cl): LLM-guided SSL (specifically LG-CoTrain) is essential for achieving high performance.
- Moderate Resources (50+ lb/cl): Simpler, cost-effective methods like Self-Training become competitive or superior, reducing the need for expensive LLM API calls.
- Calibration Needs: When reliable confidence estimates are critical for decision-making, methods like AUM-ST-MixUp or VerifyMatch are preferred.
Cost-Benefit Analysis: The authors estimate the deployment cost (including LLM pseudo-labeling and training) at approximately $36 per event, suggesting that while LLM-guided SSL is costlier than pure SSL, it provides a necessary performance boost when labeled data is critically scarce.

In conclusion, the paper demonstrates that while zero-shot LLMs are strong baselines, LLM-guided semi-supervised learning effectively distills this capability into compact models, offering a scalable solution for crisis data classification where labeled data is limited.