Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

Technical Summary: Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

Problem Statement
As Large Language Models (LLMs) become embedded in search engines and conversational interfaces, an implicit assumption has emerged: that model scale and general capability are sufficient for the nuanced, fine-grained classification of misinformation discourse. Specifically, there is a question of whether zero-shot LLMs can reliably classify user responses to misinformation claims as "belief" (propagating the claim), "fact-check" (correcting it), or "other" (neutral/unrelated). This task is non-trivial due to linguistic asymmetry; fact-checks often contain explicit corrective markers, whereas belief is frequently implicit, affective, or sarcastic. The paper tests the assumption that zero-shot inference by frontier models can match or exceed the performance of small, task-specific fine-tuned transformers in this verification context.

Methodology
The study evaluates nine models across three paradigms on a curated dataset of 900 Reddit comments spanning three PolitiFact-verified misinformation claims (Environment, Health, Immigration).

• Dataset: The dataset consists of 300 comments per topic (100 per class: belief, fact-check, other), balanced to prevent macro-F1 distortion. Annotations were performed by human annotators with substantial inter-annotator agreement ($\kappa = 0.752$). A pre-filtering step using GPT-4o was employed to identify candidate comments, with the authors verifying that this introduced negligible bias.
• Models:
  • Zero-Shot Generative LLMs: Three open-source Llama variants (3B, 8B, 70B) and three commercial frontier models (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6).
  • Zero-Shot NLI Baseline: BART-MNLI, which reframes classification as textual entailment.
  • Supervised Fine-Tuning: DistilBERT and RoBERTa, fine-tuned on the 900-comment dataset.
• Experimental Design:
  • Label Schemas: Models were tested under two conditions: a Universal Label (UL) schema (generic labels applied across all topics) and Topic-Specific schemas (fine-grained labels tailored to the discourse of each claim).
  • Evaluation: Stratified 5-fold cross-validation was used. Performance was measured using macro-F1 and per-class F1 scores. Statistical significance was assessed via paired Wilcoxon signed-rank tests and permutation tests with Holm–Bonferroni correction.
  • Constraints: Zero-shot inference was deterministic (temperature=0). Commercial models were queried via APIs, while open-source models were run locally.

Key Results
The central finding is that the assumption of scale sufficiency does not hold; task-specific fine-tuning significantly outperforms zero-shot configurations, particularly on the critical "belief" class.

1. Fine-Tuning Superiority: Fine-tuned RoBERTa achieved a macro-F1 of 0.62, significantly outperforming the best zero-shot model, Claude Haiku 4.5 (0.50). This advantage was consistent across topics and label schemas.
2. The "Belief" Gap: The supervised advantage is concentrated on the "belief" class, which zero-shot models consistently under-detect.
   • Under universal labels, zero-shot models detected belief with F1 scores ranging from 0.17 (Claude Sonnet 4.6) to 0.47 (Llama-3.2-3B, which collapsed on the "other" class).
   • Fine-tuned RoBERTa achieved a belief F1 of 0.52, a gain of $\Delta = +0.18$ over the best balanced zero-shot model.
   • The "belief" class remains the most difficult to detect even under supervision, suggesting a linguistic asymmetry where implicit/affective cues are harder to capture than explicit corrective markers.
3. Scaling Does Not Consistently Help:
   • Within the Llama family, scaling from 8B to 70B did not yield consistent improvements; the 70B model performed worse on the "belief" class (0.24) compared to the 8B model.
   • Among commercial models, the larger/more capable Claude Sonnet 4.6 underperformed the smaller Claude Haiku 4.5 (0.42 vs. 0.50 macro-F1). Sonnet's performance collapsed on the "belief" class (0.17) and triggered safety-aligned refusals on sensitive comments, indicating that safety alignment, rather than capacity limits, hindered performance.
4. Label Schema and Topic Interaction: Zero-shot performance is highly sensitive to label wording and topic. Topic-specific labels improved performance for some models (e.g., BART-MNLI in Environment) but not others, and the same model could vary by >0.13 macro-F1 across topics.
5. BART-MNLI as a Baseline: Despite being a zero-shot NLI model, BART-MNLI provided a surprisingly strong, balanced, and cost-efficient baseline, outperforming several generative LLMs in specific settings without incurring API costs.

Significance and Claims
The paper argues that in the context of misinformation verification, where missing a "belief" response is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.

• Challenging the Scale Assumption: The study provides empirical evidence that scale and general capability are insufficient for fine-grained verification tasks. The "belief" class, which is implicit and affective, is systematically under-detected by zero-shot models, including proprietary frontier models.
• Safety Alignment Artifacts: The underperformance of models like Claude Sonnet 4.6 is attributed to safety alignment mechanisms that make models reluctant to assert that a comment "spreads misinformation" without strong contextual evidence, rather than a lack of reasoning capacity.
• Cost-Effectiveness: Fine-tuned RoBERTa achieves superior performance at a fraction of the per-query cost of commercial LLMs. While zero-shot models (particularly BART-MNLI) offer a practical baseline when labeled data is unavailable, they are not a substitute for supervised models when high-stakes detection of misinformation propagation is required.
• Linguistic Asymmetry: The difficulty in detecting "belief" is identified as a structural limitation of the task (implicit vs. explicit cues) rather than a solvable problem solely through model scaling.

The authors conclude that while large generative models are powerful, they do not yet replace the need for supervised, task-specific fine-tuning for reliable misinformation response classification, particularly when the goal is to identify the propagation of false claims rather than just their correction.