Comparative Evaluation of GPT-4o-mini and Llama-3.1-8B for Extracting Climate Hazard Impacts from Nigerian News Corpus
This study benchmarks GPT-4o-mini against Llama-3.1-8B on a Nigerian news corpus for climate hazard impact extraction, revealing that while both models perform similarly in classification, GPT-4o-mini offers superior accuracy and significantly lower hallucination rates, highlighting a critical reliability trade-off for high-stakes documentation in Global South contexts.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world, vast amounts of information about disasters are trapped inside unstructured news articles. These stories describe floods, storms, and heatwaves, but they are written in paragraphs of text rather than organized tables of data. For researchers and emergency planners, turning these stories into useful statistics is a monumental task. Traditionally, humans had to read every single article to pull out details like the date of an event, the specific location, and the number of people affected. This manual work is slow and impossible to scale when thousands of stories are published. Recently, a new type of computer program called a large language model has emerged. These are sophisticated tools trained on massive amounts of text that can read, understand, and summarize information. They offer a way to automate the extraction of facts from news, potentially turning chaotic stories into clear, actionable data for disaster management. However, not all of these tools are created equal, and their reliability in high-stakes situations, especially in regions with unique languages and reporting styles, remains a critical question.
A team of researchers set out to test how well two different computer models could perform this difficult job using news reports from Nigeria. The study focused on a specific challenge: extracting details about climate hazards, such as floods and storms, from a corpus of 1,226 unique Nigerian news articles. The researchers compared a well-known commercial model, GPT-4o-mini, against a free, open-source model called Llama-3.1-8B. The goal was to see which tool could more accurately identify the type of disaster, pinpoint the exact location down to the local government area, and extract specific numbers regarding human and economic impacts. To ensure a fair and rigorous test, the researchers built a system where both models read the same articles and tried to fill out a structured form with the facts, which was then checked against a gold standard of 100 articles manually annotated by human experts.
The results showed a clear distinction between the two tools, particularly when it came to the most critical information. Both models were quite good at identifying the general type of hazard, such as distinguishing a flood from a storm, with the commercial model performing slightly better. They also both managed to identify the country correctly almost every time. However, as the task became more specific, the gap widened. When asked to name the specific state or city where a disaster occurred, the commercial model was significantly more accurate than the open-source version. The open-source model struggled more with these finer details, often confusing similar-sounding place names or missing the specific administrative regions that are vital for local emergency response.
The most significant finding, however, concerned the numbers. In disaster reporting, knowing exactly how many people were displaced or how much money was lost is essential for planning aid. The researchers found that the open-source model generated a much larger volume of numerical records, but a vast majority of these numbers were not actually supported by the text of the news articles. In fact, only about 36 percent of the numbers produced by the open-source model could be traced back to a specific sentence in the source text. The other two-thirds appeared to be guesses or fabrications, a phenomenon known as hallucination. In contrast, the commercial model was much more careful. While it produced fewer numbers overall, about 66 percent of them were grounded in the actual text. This means that when the commercial model reported a number, it was far more likely to be a fact taken directly from the story rather than a made-up figure.
This difference in reliability has profound implications for how these tools are used in the real world. The study suggests that while open-source models can be useful for getting a general overview of where disasters are happening, they are not yet trustworthy enough to be used alone for extracting specific impact figures. If a planner relied on the open-source model's numbers without checking them, they might base critical decisions on data that simply does not exist in the source material. The commercial model proved to be more dependable for these high-stakes details, though it still made errors. The researchers concluded that for the best results, these computer tools should be viewed as assistants that help humans process large volumes of text quickly, but the final verification of important numbers must still be done by people. This approach allows for the speed of automation while ensuring the accuracy needed to protect lives and resources in regions like Nigeria, where climate hazards are frequent and severe.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.