Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

Technical Summary: Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

Problem Statement
The media's framing of climate change—specifically whether coverage emphasizes threats (risks, dangers) or solutions (actions, remedies)—significantly influences public engagement and policy support. While manual content analysis is the standard for studying these framing patterns, it is labor-intensive and difficult to scale. Automated detection using Natural Language Processing (NLP) offers a solution but requires validation against human judgments. A critical gap exists in the literature regarding the direct comparison of two dominant automated approaches for sentence-level framing detection in a specific domain (climate) and language (German): fine-tuned encoder models versus few-shot prompting of Large Language Models (LLMs). Furthermore, most existing work relies on generic frame taxonomies rather than the threat-versus-solution distinction central to climate communication research, and German-language climate media analysis remains underrepresented.

Methodology
The study evaluates two distinct approaches on a corpus of 440 manually coded articles from two major Austrian newspapers (Kronen Zeitung and Der Standard), published between January and June 2024. The dataset was segmented into 10,981 sentences, with binary labels for "threat orientation" and "solution orientation" treated as independent dimensions.

1. Fine-Tuned BERT Approach:

   • Model: A monolingual German BERT model (deepset/gbert-large) was fine-tuned for sentence-pair classification.
   • Input Format: To incorporate context, the input was formatted as a sentence pair: [CLS] previous sentence [SEP] target sentence [SEP]. This allows the model to utilize the immediately preceding sentence as context.
   • Training: Two separate binary classifiers (one for threat, one for solution) were trained using the AdamW optimizer for 6 epochs.
   • Ablation: A single-sentence variant (without the preceding context) was also trained to measure the impact of context.

2. Few-Shot LLM Prompting Approach:

   • Model: The open-weights Llama 4 Maverick model was accessed via API.
   • Prompt Architecture: Prompts were written entirely in German and structured with XML tags containing eight sections: role definition, speech-act taxonomy, framing definitions, classification rules, positive/negative examples, the full article text, and the target sentence.
   • Reasoning: The model employed Chain-of-Thought (CoT) reasoning, requiring a justification field before the final classification.
   • Output: Structured JSON output included a boolean classification, a reasoning field, and a confidence score.
   • Filtering: A threshold sweep (0.00 to 1.00) was performed on confidence scores to optimize metrics, though this proved ineffective.

Key Contributions

• Systematic Comparison: The paper provides a direct, controlled comparison between fine-tuned BERT and few-shot LLM prompting for sentence-level climate framing detection in German, a domain where such comparisons are lacking.
• Contextual Analysis: It demonstrates that providing the preceding sentence as context in a sentence-pair format significantly improves BERT performance over single-sentence inputs.
• Prompt Engineering: The authors detail a comprehensive prompt architecture that integrates speech-act taxonomies, domain-specific coding rules, and CoT reasoning, mirroring the information available to human coders.

Results

• Performance: The fine-tuned BERT classifiers outperformed the LLM-based classifiers on both tasks. BERT achieved an F1 score of 0.83 for both threat and solution detection, while the LLM achieved an F1 of 0.78.
• Context Impact: The ablation study confirmed the value of context. The BERT single-sentence model (without the preceding sentence) dropped to F1 scores in the low 0.70s, a decrease of approximately 10 points compared to the sentence-pair model.
• Disagreement Analysis: On the subset of sentences where the two methods disagreed (approx. 20-24% of the test set), BERT maintained an F1 of 0.65 (threat) and 0.60 (solution), whereas the LLM dropped to near-random levels (F1 ~0.34–0.35). BERT's predictions on these ambiguous cases aligned with ground truth roughly twice as often as the LLM's.
• Error Patterns:
  • BERT: Struggled with indirect framing where context lay beyond the immediate preceding sentence and with sentences contrasting threats and solutions.
  • LLM: Suffered from "context bleeding," where solution-oriented material elsewhere in the full article caused the model to misclassify locally neutral sentences as solution-oriented. The LLM also exhibited internal inconsistencies where its reasoning contradicted its classification.
• Confidence Scores: The LLM's self-reported confidence scores were poorly calibrated. Despite mean confidence exceeding 0.93, error rates were over 20%. Adjusting thresholds based on confidence did not improve F1 scores.

Significance and Claims
The paper claims that for sentence-level framing classification tasks where labeled training data is available, fine-tuned encoder models remain the superior choice, offering higher accuracy and better handling of ambiguous cases compared to few-shot LLM prompting. The study validates the sentence-pair approach as an effective, lightweight method for incorporating context in encoder-based classifiers.

However, the authors note that the LLM approach achieves competitive performance without requiring model training, making it a viable alternative for settings where labeled data is scarce or unavailable. The results contribute to the broader discourse in computational social science regarding the trade-offs between supervised fine-tuning and in-context learning. The paper modestly concludes that while LLMs are powerful, they currently lack the reliability of fine-tuned models for this specific task, particularly regarding confidence calibration and the handling of complex contextual dependencies. Future work is suggested to explore article-level train-test splits and the effect of larger context windows.