Technical Summary: Are LLMs Ready for Conflict Monitoring? Empirical Evidence from West Africa

Problem Statement

The integration of Large Language Models (LLMs) into humanitarian and security decision-making pipelines raises critical concerns regarding their normative neutrality. While LLMs offer potential for automated conflict event extraction, their application in conflict monitoring risks introducing systematic distortions that could undermine human rights documentation and humanitarian response. Specifically, there is a lack of empirical evidence regarding whether model outputs faithfully reflect the realities of political violence or introduce biases that favor state actors over non-state actors, or misclassify legitimate battles as civilian-targeted violence. This study addresses the gap in understanding how general-purpose and domain-adapted models perform on conflict data from West Africa (Nigeria and Cameroon), focusing on systematic errors, actor-based selection bias, and sensitivity to lexical framing.

Methodology

The study utilizes the Armed Conflict Location & Event Data (ACLED) project as a gold-standard ground truth, covering conflict events in Nigeria and Cameroon from 2020–2024. The prediction task involves multi-class classification into six canonical categories: Violence against civilians (V), Battles (B), Explosions (E), Protests (P), Riots (R), and Strategic developments (S).

Models Evaluated:

• Vanilla Open-Weight Models: Gemma 3 4B, Llama 3.2 3B, Mistral 7B, and OLMo 2 7B.
• Domain-Adapted Models: AfroConfliBERT (fine-tuned ConfliBERT) and AfroConfliLLAMA (Llama-based decoder fine-tuned via LoRA) on a corpus of 81,104 ACLED African conflict events.

Experimental Design:

• Data Processing: A standardized pipeline was applied for actor normalization (mapping raw mentions to State vs. Non-State actors) and stratified sampling to ensure balanced representation.
• Evaluation Metrics: The study employed Statistical Parity Difference (SPD) and Equalized Odds (True Positive Rate and False Positive Rate disparities) to assess group-level fairness.
• Counterfactual Analysis: Controlled perturbations were applied to event descriptions, including lexical framing (delegitimization/legitimization phrases), action substitutions, intensity modifiers, and actor substitutions. Flip rates and confidence shifts were measured against neutral controls.
• Error Trace Profiling: Rationale-Flip Concordance (RFC) and Layer Integrated Gradients (LIG) were used to analyze whether model reasoning aligns with input changes or constitutes confabulation.
• Calibration: Post-hoc calibration using isotonic regression and temperature scaling was applied to align predicted confidence with empirical accuracy.

Key Contributions

1. Systematic Evaluation of LLM Bias on ACLED: This work provides the first comprehensive evaluation of LLM bias on conflict event data, bridging computational linguistics and empirical conflict studies through a targeted analysis of Nigeria and Cameroon.
2. Quantification of Normative Directionality: The study rigorously quantifies the directionality of model errors, distinguishing between "False Illegitimation" (misclassifying battles as violence against civilians) and "False Legitimation" (misclassifying violence against civilians as battles).
3. Analysis of Lexical Framing Sensitivity: The research evaluates how few-shot in-context learning (ICL) and specific lexical cues shift normative directionality, demonstrating that vanilla models are highly fragile to semantic framing.
4. Error Trace and Confabulation Analysis: Using RFC and LIG, the study demonstrates that vanilla LLMs often mask normative bias through unfaithful rationale confabulations, whereas domain-adapted models show different attribution patterns.
5. Introduction of Domain-Adapted Models: The paper introduces AfroConfliBERT and AfroConfliLLAMA, investigating the extent to which domain adaptation mitigates observed biases and identifying remaining limitations.

Key Results

1. Bifurcated Divergence in Normative Directionality

• Vanilla Models: Exhibit a statistically significant False Illegitimation bias. For instance, Gemma misclassified 18.29% of legitimate battles (B) as civilian-targeted violence (V) in Cameroon while making zero False Legitimation errors. Llama showed similar patterns (10.32% in Cameroon).
• Domain-Adapted Models: AfroConfliBERT and AfroConfliLLAMA achieved near-directional neutrality, with Legitimization Bias differences statistically indistinguishable from zero.

2. Persistent Actor-Based Selection Bias

Despite achieving directional neutrality, domain adaptation did not eliminate actor-based selection bias.

• In Nigeria, state actors were legitimized 36.5% more often than non-state actors in identical tactical contexts across models.
• Vanilla models like Llama showed a Statistical Parity Difference (SPD) of -0.364 in Nigeria, indicating they are over 36% less likely to assign a violence label to state actors compared to non-state actors.
• Even fine-tuned models (AfroConfliBERT/LLAMA) exhibited severe selection bias (SPD ~ -0.31), suggesting a deeply embedded posterior associating "state" with non-violent status.

3. Fragility to Lexical Framing

• Vanilla Models: Highly vulnerable to delegitimizing phrases. In Cameroon, inserting "violating human rights" caused a flip rate of 66.7%. In Nigeria, phrases like "unprovoked" and "using excessive force" drove flip rates of 21.1% and 19.3%, respectively.
• Regional Divergence: Lexical sensitivity is context-dependent; perturbations significant in Nigeria were often insignificant in Cameroon, and vice versa.
• Domain-Adapted Models: Demonstrated near-zero flip rates (0.0% to ~11%) across all perturbation categories, showing robustness to semantic shifts.

4. Error Trace and Reasoning

• Confabulation: Vanilla models frequently altered their stated rationale (up to 97.7% of flips for Llama) without anchoring the reasoning to the injected perturbation (concordance rates as low as 8.3% for Mistral), indicating unfaithful explanations.
• Attribution: AfroConfliBERT's predictions were primarily driven by core kinetic verbs (60%), confirming semantic anchoring to factual actions, though it showed some reliance on punctuation (40%), suggesting a partial "Clever Hans" effect.

5. Calibration and Selective Prediction

• Vanilla LLMs exhibited significant overconfidence (high Brier scores) without calibration. Isotonic regression reduced Brier scores by ~27% in Cameroon.
• Selective prediction analysis suggests that to maintain >80% accuracy in Cameroon, vanilla models require human-in-the-loop oversight for approximately 47% of cases.

Significance and Claims

The paper concludes that current LLMs are not ready for unsupervised deployment in conflict monitoring. The findings highlight two distinct failure modes:

1. Directional Bias: Vanilla models systematically delegitimize state actors by misclassifying battles as violence against civilians.
2. Lexical Fragility: General-purpose models rely on superficial framing cues rather than factual event content, making them susceptible to manipulation via strategic lexical insertion.

While domain adaptation (AfroConfliBERT/LLAMA) substantially mitigates directional bias and lexical fragility, it fails to resolve actor-based selection bias. The authors argue that neither architecture currently reconciles directional neutrality, actor-based fairness, and lexical robustness simultaneously.

The paper calls for three evidence-grounded interventions:

1. Fairness-aware fine-tuning: Explicitly optimizing for actor-based parity alongside domain adaptation.
2. Mandatory adversarial robustness evaluation: Pre-deployment testing against lexical manipulation to detect flip rates.
3. Context-specific human-in-the-loop oversight: Calibrated to regional difficulty, acknowledging that even robust models require human review for a significant portion of cases in complex conflict zones like Cameroon.

The study emphasizes that relying on these models without bias correction risks producing data that downplays state-led violence and inflates the violent profile of non-state actors, with direct implications for resource allocation and protection decisions in conflict-affected populations.