💻 computer science

Geolocating News about Extreme Climate Events: A Comparative Analysis of Off-the-Shelf Tools for Toponym Identification in German

This study conducts a comparative analysis of three off-the-shelf Named Entity Recognition tools (Flair, Spacy, and Stanza) for identifying toponyms in German news articles about extreme climate events, demonstrating how their differing outputs propagate through downstream geolocation tasks to significantly impact conclusions regarding countries' prominence in media coverage.

Original authors: Brielen Madureira, Mariana Madruga de Brito, Andreas Niekler

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Brielen Madureira, Mariana Madruga de Brito, Andreas Niekler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: "Where did this extreme weather event actually happen?"

You have a pile of German news articles. Some talk about floods in Brazil, others about heatwaves in Germany. The problem is, the articles are messy. One article might say, "The floods in Porto Alegre, Brazil, remind us of the Hunsrück region in Germany." A human reader instantly knows the disaster is in Brazil. But a computer? It might get confused by the word "Germany" and think the flood happened there.

This paper is about testing three different "computer detectives" (software tools) to see which one is best at finding the right location in these news stories.

The Three Detectives

The researchers tested three popular, off-the-shelf software tools (think of them as three different brands of GPS):

Flair
Spacy
Stanza

Their job is Named Entity Recognition (NER). In plain English, this means scanning a sentence and highlighting words that look like places (like "Paris," "Amazon," or "Hunsrück").

The Experiment: A "Blind" Test

Usually, to test a detective, you give them a list of answers you already know is correct (a "gold standard"). But in the real world, researchers often don't have that list. So, the authors set up a tricky test:

They fed all three tools a collection of 983 German news articles about disasters (floods, fires, heatwaves).
They asked each tool to list the places it found.
They then tried to guess the country of the disaster based only on those lists.

What They Found: The Detectives Disagree

Here is the big surprise: The three tools didn't agree with each other very often.

Different Lists: If you gave the same article to all three, they would highlight different sets of words as "places." One might see 14 places, another 15, and they might not even agree on which 14.
The "Noise" Problem:
- Spacy was like a detective who sees too much. It flagged generic words like "city," "downtown," or "airport" as specific places. This created a lot of "noise" (false alarms).
- Stanza sometimes got confused by words that sound like places but aren't, like adjectives (e.g., "Italian" triggering a search for places in Italy, even if the article was about an Italian embassy in France).
- Flair was the most careful. It found fewer places, but the ones it found were more likely to be real, valid locations.

The Ripple Effect: Why It Matters

The paper shows that choosing the wrong detective changes the final verdict.

Wrong Country Guesses: Because the tools found different lists of places, they guessed the country of the disaster differently. In some cases, the choice of tool changed the predicted country by more than 10%.
The "Popularity" Ranking: The researchers tried to rank countries by how often they appeared in the news.
- If you used Flair, the ranking looked very close to the human-annotated truth.
- If you used Spacy or Stanza, the ranking was slightly off. For example, one tool might think Switzerland is the most talked-about country, while the humans know it's actually Brazil.

The Takeaway

The authors aren't saying one tool is perfect or that the others are useless. They are saying: Don't just pick a tool because it's popular.

If you are a researcher trying to understand climate change coverage, the tool you choose acts like a filter. A slightly different filter can change your conclusions about which countries are most affected or most discussed.

In short: Even if you are using "ready-made" software, you have to check its work. If you don't, you might accidentally write a report saying the floods happened in Germany when they were actually in Brazil, simply because your computer got confused by the word "Germany" appearing in the text.

Technical Summary: Geolocating News about Extreme Climate Events

Problem Statement
Determining the geolocation of extreme climate events and disasters within text is a critical challenge in climate impact and adaptation research. While Named Entity Recognition (NER) tools are standard for identifying toponyms (location names) as candidate event locations, these tools are not error-free. A specific difficulty arises when texts contain references to multiple locations, including those unrelated to the actual event (e.g., a German news article discussing floods in Brazil that mentions German dialects or regions). Computational tools often struggle to distinguish between the event's actual location and other referenced places, leading to potential errors in document geolocation. This issue is compounded by the lack of robust, systematic evaluations of off-the-shelf NER tools specifically for German texts in the context of climate discourse.

Methodology
The authors conducted a comparative analysis of three popular off-the-shelf NER tools for German: Flair (model de-ner-large), Spacy (model de_core_news_lg), and Stanza (model de). The study utilized a corpus of 983 German news articles (published 2000–2024) concerning seven types of climate hazards (heat waves, wildfires, floods, storms, droughts, cold waves, and landslides).

The evaluation followed a four-step pipeline designed to assess how NER outputs propagate to downstream tasks without relying on a gold standard for toponym annotation:

Identification: Each document was processed by the three NER tools to extract tokens labeled as locations (LOC).
Querying: Unique toponyms were queried against two geographical databases, Geonames and Nominatim, to retrieve coordinates and country metadata.
Resolution: Ambiguities were resolved using a "spatial minimality" heuristic, mapping toponyms to coordinates closest to the polygon formed by unambiguous toponyms in the document.
Prediction: Three distinct methods were applied to determine the document's geographical focus (country) based on the resolved toponyms:
- Majority Voting: Selecting the most frequent country among identified toponyms.
- Closest to Centroid: Calculating the centroid of the toponym polygon and selecting the nearest toponym's country.
- Keyword Proximity: Restricting toponyms to those co-occurring with hazard keywords in the same sentence, or selecting the closest toponym if no co-occurrence exists.

Performance was evaluated extrinsically against a human-annotated gold standard of document-level geographical focus. Metrics included Intersection over Union (IoU) for toponym overlap, exact and partial match rates for country prediction, and Kendall $\tau$ and Spearman rank correlations for country frequency rankings.

Key Results

Toponym Identification Variance: The tools produced significantly different sets of toponyms. Spacy identified the highest number of unique toponym types (6,700) but had the lowest coverage in geographical databases (42.75% in Geonames, 63.58% in Nominatim). Flair identified fewer types (4,639) but achieved higher database match rates (63.16% in Geonames, 81.20% in Nominatim).
Downstream Disagreement: Variations in toponym identification led to substantial divergence in country predictions. Depending on the method and database, the choice of NER tool resulted in at least 10% (and up to >20%) of documents receiving different country predictions.
Prediction Accuracy: Flair consistently achieved the highest accuracy in predicting the correct country, followed by Stanza and Spacy. The "Keyword Proximity" method generally yielded the best performance. The choice of NER tool alone could cause a difference of up to 5 percentage points in prediction accuracy.
Ranking Correlation: When ranking countries by media prominence, Flair showed the highest correlation with the gold standard (Spearman $r=0.833$ ), outperforming Stanza ($0.758$) and Spacy ($0.753$). While all tools correctly identified the top two countries, the ordering of subsequent countries varied significantly; for instance, Spacy ranked France three positions higher than Stanza.
Error Sources: Spacy tended to identify generic terms (e.g., Stadt, Innenstadt) as toponyms, while Stanza frequently identified adjectival and demonymic forms (e.g., italienischen), introducing noise. The lack of distinct labels for geopolitical entities versus nationalities in German NER training data was identified as a contributing factor to these errors.

Significance and Claims
The paper argues that while off-the-shelf NER tools are often accepted at face value, their selection can discernibly influence downstream conclusions in climate discourse research. The authors demonstrate that without careful comparative evaluation, researchers risk:

Introducing unwarranted negative impacts on prediction accuracy (up to 5 p.p. lower).
Generating country rankings that are less correlated with reality.
Drawing potentially flawed conclusions regarding the prominence of specific countries in media coverage.

The study does not propose a new, superior NER model but rather provides empirical evidence that setup decisions, which may appear minor, require rigorous evaluation. It highlights the necessity of error analysis and tool comparison, particularly when gold standards for toponym annotation are unavailable, to ensure the reliability of data-driven analyses in climate adaptation research.