Retrieving Floods without Floodlights: Topic Models as Binary Classifiers for Extreme Climate Events in German News

Technical Summary: Retrieving Floods without Floodlights

Problem Statement
The paper addresses a critical challenge in climate discourse research: the retrieval of relevant news articles regarding extreme climate events from large databases. While dictionary-based keyword retrieval is a standard initial step, it suffers from high rates of false positives due to the polysemic nature of hazard-related terms (e.g., "flood" referring to emotions or "floodlights," or "drought" referring to low water levels in rivers unrelated to a specific drought event). Existing deep learning-based classifiers often require large amounts of annotated data for training, which is frequently unavailable in socio-environmental research. Furthermore, treating all climate hazards as a single category in Natural Language Processing (NLP) tasks may obscure significant variations in how different events are reported and linguistically framed.

Methodology
The authors propose a novel approach to using unsupervised Topic Models (TMs), specifically Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF), as binary classifiers without modifying their training procedures. The method operates as follows:

1. Data Collection: A corpus of over 13 million German news articles (2000–2024) was retrieved using hazard-specific keywords. After rule-based filtering to remove "intruder" keywords (e.g., "Flutlicht" for flood) and local news, a sample of ~2.4 million documents remained.
2. Topic Modeling: TMs were trained on the unique documents for each of seven hazard types (cold waves, droughts, floods, heat waves, landslides, storms, wildfires).
3. Binary Classification Logic: Instead of human interpretation of topics, the authors utilize the posterior distributions estimated by the TMs:
   • Topic-Document Relation: A document is considered to discuss a topic if the topic's probability in that document exceeds a threshold ($\theta$).
   • Topic Relevance: A topic is classified as "relevant" if it satisfies one of two criteria based on predefined keywords:
     • Keyword Proximity: The topic assigns a probability higher than a threshold ($\gamma$) to at least one keyword.
     • Top Terms: A keyword appears among the top $k$ most probable terms of the topic.
   • Document Classification: A document is classified as relevant if it is associated with at least one "relevant" topic.
4. Comparative Baselines: The TM approach was compared against:
   • A fine-tuned text embedding classifier (SetFit with BAAI/bge-m3).
   • An open-weight Large Language Model (LLM) (Mistral Ministral-3-14b) used in a zero-shot prompting setup.
   • An ensemble strategy using majority voting across the three models.

Key Contributions

1. Data Analysis: The study provides an annotated analysis of 3,150 German news articles covering seven distinct extreme climate events, highlighting the varying difficulty of retrieval across different hazards.
2. Unsupervised Classification: The authors demonstrate a method for using standard TMs as binary classifiers without requiring retraining or complex human interpretation of topics, relying instead on keyword probabilities within the topic distributions.
3. Hazard-Dependence: The paper provides evidence that classification performance is highly dependent on the specific hazard type, arguing against the practice of aggregating all climate events into a single NLP category.

Results
The study evaluated performance using precision, recall, and F1 scores on a test set of 100 documents per hazard type:

• Overall Performance: The fine-tuned text embedding classifier generally achieved the highest F1 scores, followed closely by the LLM and the TM ensemble. The LLM achieved the highest recall (often near 1.0) but the lowest precision.
• TM Performance: While TMs generally had lower F1 scores than deep learning alternatives in aggregated results, the gap was often small (e.g., only 0.001 difference for wildfires). Crucially, specific TM configurations (e.g., tm-p) achieved significantly higher precision than other models, albeit at the cost of recall. For certain hazards (cold waves, storms, wildfires), TM approaches performed on par with or better than fine-tuned embeddings.
• Hazard Variability: Performance varied drastically by hazard. Landslides and wildfires were the easiest to identify (F1 > 0.75 for most models), while cold and heat waves proved most challenging (F1 < 0.60 even for the best models).
• Interpretability: Unlike deep learning models, the TM approach offers full transparency; a document's classification can be traced back to specific topics and keywords that triggered the decision.

Significance and Claims
The paper claims that while deep learning models often yield superior aggregate metrics, Topic Models offer a compelling alternative for climate research due to their interpretability and higher precision in specific contexts. The authors argue that the "black box" nature of deep learning and LLMs is a disadvantage when researchers need to understand why a document was classified as relevant.

Furthermore, the study emphasizes that hazard-dependence is a critical factor. The authors assert that treating extreme climate events as a monolithic category risks biasing research conclusions toward hazards that are easier to detect linguistically. They conclude that TMs, despite being unsupervised and simpler, can effectively refine retrieval results, particularly when high precision is prioritized over recall or when interpretability is required. The paper also notes that LLMs, even when open-weight and locally run, are not immune to precision issues and require careful evaluation, suggesting TMs could be used as a pre-filter to reduce the computational and financial costs of using LLMs.