The Limits of LLM Forecasting: Parametric Knowledge Gaps Across Conflict Zones

The Big Idea: AI is "Blind" to Some Wars

Imagine you are trying to predict when a storm is going to get worse. You have two tools:

1. A Super-Intelligent Weather Forecaster (The LLM): It has read every newspaper, book, and website ever written. It knows everything about history and geography.
2. A Simple Rain Gauge (The Logistic Regression): It doesn't know where it is or who is involved. It just looks at the water level in the gauge right now and the last few days.

The paper asks: Who is better at predicting the storm?

The shocking answer is: The simple rain gauge wins. The super-intelligent forecaster fails spectacularly, but not because it's "dumb." It fails because it has been trained on a very biased diet of news.

The Problem: The "News Diet" Bias

The researchers looked at 22 different places where fighting was happening between 2020 and 2026. They measured how much English-language news covered each place compared to how much fighting was actually happening.

• The "VIP" Zones: Places like Israel and Ukraine get massive attention. For every single fight, there are dozens of news articles.
• The "Forgotten" Zones: Places like Myanmar or the Democratic Republic of Congo get almost no attention. For every fight, there might be less than one article.

The Gap: There is a 224x difference in coverage. The most covered place gets 224 times more news per fight than the least covered place.

Because Large Language Models (LLMs) are trained on this English news, they have a "mental map" of the world that is completely distorted. They know a lot about the VIP zones and almost nothing about the Forgotten zones.

The Experiment: Guessing the Next Fight

The researchers asked two famous AI models (Llama-3.3 and GPT-4o) to predict if violence would get worse in the next month for these 22 countries. They gave the AI no data—just the country name and the date. This is called a "zero-shot" test, meaning the AI had to rely entirely on what it memorized during training.

They compared the AI to:

1. The "Always Yes" Baseline: A person who just guesses "Yes, it will get worse" for every single country.
2. The "Simple Math" Baseline: A computer program that only looks at the numbers of fights in the last few weeks, ignoring the country name entirely.

The Results: The AI Got Stuck in a Rut

The results were not a smooth gradient where the AI was "okay" at some places and "great" at others. Instead, the AI broke into two distinct, weird behaviors:

1. The "Panic Button" AI (Llama)

• Where it failed: In the Forgotten Zones (under-covered).
• What it did: It predicted "Yes, violence will get worse" for every single case.
• The Analogy: Imagine a security guard who has never seen a specific neighborhood. Because he's heard rumors that "all bad things happen there," he assumes everything is a crime. He screams "ALARM!" for every single person walking down the street.
• The Result: He caught every real emergency (100% recall), but he also screamed at innocent people. His performance was identical to the "Always Yes" person. He wasn't forecasting; he was just categorizing the country as "chaotic."

2. The "Denial" AI (GPT-4o)

• Where it failed: In the VIP Zones (over-covered).
• What it did: It predicted "No, violence will not get worse" for every single case, even when fighting was actually spiking.
• The Analogy: Imagine a weather forecaster who has watched the same storm for 10 years. He thinks, "We've seen this before, it's just background noise." He ignores the fact that the wind is picking up and says, "It's fine."
• The Result: He missed every single real escalation (0% recall). He was so confident in his "world knowledge" that he ignored the actual danger.

The Twist: Giving the AI More Data Makes It Worse

The researchers thought, "Maybe the AI just needs more facts." So, they gave the AI a report card with the actual number of fights from the last three months (structured evidence).

• What happened? The AI got worse.
• Why? The AI tried to read the numbers, but its "brain" was still stuck in its old habits.
  • In the Forgotten Zones, seeing the numbers confused the "Panic Button" AI, making it less accurate.
  • In the VIP Zones, the "Denial" AI still refused to believe the numbers.
• The Winner: The Simple Math baseline (the rain gauge) still won. It didn't care about the country name or the news coverage. It just looked at the numbers: "If the water level went up, the storm is getting worse." It was right 2.4 times more often than the AI with the evidence.

The Conclusion: It's Not About Data, It's About Bias

The paper concludes that the problem isn't that the AI lacks data. The problem is how the AI interprets the data based on where the country is.

• For the Forgotten: The AI assumes chaos is the default state.
• For the VIPs: The AI assumes stability is the default state.

The AI isn't "forecasting" (predicting the future based on signals); it is categorizing (labeling countries based on its training data).

The Takeaway for Real Life

If you are a humanitarian worker trying to use AI to save lives in a forgotten conflict zone:

1. Don't trust the AI's gut feeling. It will likely just guess "Yes, it's bad" for everything, which is useless.
2. Don't assume giving it more news articles will help. It might just confuse it.
3. Simple tools work better. A basic system that tracks the numbers of recent events is actually more reliable than a super-intelligent AI that has read the whole internet.

The paper calls for a new way of testing AI: We shouldn't just ask "Is the AI smart?" We need to ask "Is the AI smart for this specific country?" because right now, it is smart for some and blind for others.