🤖 AI

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

The paper introduces "Age of LLM," a novel turn-based 1v1 benchmark featuring fog of war, unrestricted diplomacy, and strict action reliability constraints to evaluate how large language models reason, deceive, and track beliefs under adversarial uncertainty, revealing that nuclear strategies dominate while diplomatic agreements rarely succeed.

Original authors: Arnaud Ricci

Published 2026-06-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Arnaud Ricci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes chess match, but instead of a wooden board and pieces, two giant AI brains are playing a digital war game on a 13-by-7 grid. They can't see the whole board; half of it is covered in a thick, shifting fog. They have to build armies, manage secret resources, and talk to each other, all while trying to blow up the other player's base.

This is Age of LLM, a new experiment designed to see how well AI models think when they are blind, under pressure, and forced to follow strict rules without any help from a teacher.

Here is the breakdown of what happened, using simple analogies:

1. The Game: A Foggy War Room

Think of the game like a spy thriller.

The Fog of War: You can only see what your own units are standing next to. The enemy's resources and troops are hidden. If you try to move a tank into a spot you can't see, the game says, "Nope, that's illegal," and you lose that turn.
The Secret: The most powerful weapon is a nuclear bomb. To build it, you need a secret resource called "uranium." The enemy doesn't know how much uranium you have, and you don't know theirs. This allows for bluffing.
The Rules: The AI is given a rulebook but no strategy guide. It's like giving a chef a list of ingredients and kitchen tools but telling them, "Make a meal, but don't tell you what to cook." The AI has to figure out the plan itself.

2. The Big Surprise: Everyone Chose the "Nuclear Rush"

The researchers expected the AI to try many different strategies, like building a massive tank army or negotiating peace. Instead, almost everyone (about 78% to 85% of the time) chose the same path: The Nuclear Rush.

The Strategy: Build a mine to get uranium, build a silo to hold the bomb, scout the enemy, and then launch.
The Result: It was a race. The first one to launch won. The second one to launch lost.
The Twist: Because the game rules say launches happen secretly and at the same time, nobody ever saw the other person launch before they had to decide to launch themselves. So, nobody ever tried to "counter-launch" to stop the other guy. It wasn't that the AI was too dumb to think of it; the game mechanics made it impossible to react in time. It was like two people flipping a switch at the exact same second; neither could see the other's hand move first.

3. The "Tank" Strategy: Fast but Rare

Only a few AIs tried to win by driving tanks into the enemy base.

The Analogy: This is like a sprinter vs. a marathon runner. The tank strategy was much faster (winning in about 12 turns vs. 19 for nukes), but it was very hard to pull off. It required perfect coordination and luck. Most AIs were too scared to try it, so they stuck to the "safer" nuclear race.

4. The Talk: Lots of Bluffing, Very Little Trust

The AI models were allowed to send text messages to each other. They sent thousands of messages!

The Bluff: Even when an AI was losing badly, it would often send messages saying, "I'm about to crush you!" or "Surrender now!" This is called bluffing. The paper found that losers bluffed just as often as winners. It's like a poker player with a terrible hand pretending to have a Royal Flush.
The Silence: When the game was actually over, the losers rarely said "Good game" or "You won." They kept talking until the very last second.
The Secret: The most interesting finding was about deception. When an AI was building its secret nuclear silo, it rarely mentioned it in its messages. It would say things like, "We are just doing some peaceful farming," while secretly building a bomb. But once the bomb was ready to fire, they often announced it loudly. It seems the AI learned to hide the preparation but not the execution.

5. The Real Test: Reliability vs. Intelligence

The paper asked a crucial question: Does thinking harder make you win?

The Finding: Not necessarily. The AI that spent the most time "thinking" (processing tokens) didn't always win.
The Real Winner: The AI that won most often was the one that made the fewest mistakes. In this game, if you try to move a unit into a foggy area you can't see, the game ignores your command. You lose a turn.
The Lesson: The best player wasn't the smartest philosopher; it was the most reliable soldier. The AI that kept its head in the game, remembered where the enemy was, and didn't try to do the impossible, won. The paper suggests that in complex, real-world tasks, not messing up might be more important than being a genius.

6. Why This Matters (According to the Paper)

Most tests for AI are like pop quizzes: "Here is a math problem, solve it." The AI can see the whole problem and just give the answer.
Age of LLM is different. It's like a live, multi-day survival game where:

You can't see everything.
You have to remember what happened yesterday.
You have to follow strict rules or get penalized.
You have to talk to an opponent who might be lying.

The paper concludes that this is a better way to see how AI actually "thinks" when it's not just reciting facts from a textbook. It shows us that for AI to be useful in the real world, it needs to be reliable (not making up facts or forgetting rules) just as much as it needs to be smart.

In short: The AI models played a foggy war game. Most chose the nuclear route. They lied a lot in chat. And the winner wasn't the one who thought the hardest, but the one who made the fewest silly mistakes.

Technical Summary: Age of LLM

Problem Statement

Current Large Language Model (LLM) benchmarks predominantly evaluate single-turn competence on tasks with full observability and unambiguous answers (e.g., MATH, HumanEval, MMLU). These benchmarks fail to assess competencies required for real-world adversarial decision-making: planning under uncertainty, reasoning about hidden opponents, temporal commitment, and the reliability of structured output over sequential turns. Furthermore, public benchmarks suffer from data contamination, where frontier models trained on web-scraped corpora may have memorized benchmark items, inflating scores without reflecting genuine reasoning ability.

Age of LLM addresses these gaps by introducing a turn-based 1v1 benchmark where two LLMs compete on a 13×7 grid to destroy an enemy base. The environment is designed to stress-test models under three specific conditions:

Fog of War: Partial observability where enemy units, resources, and locations are hidden until scouted.
Full Diplomacy: Models can exchange free-text messages, propose ceasefires, issue ultimatums, or negotiate peace, with no binding constraints other than game rules.
Reliability Constraints: Every turn requires strict adherence to a JSON schema. Illegal actions (e.g., acting on fog-hidden cells or referencing destroyed units) are silently discarded, wasting a portion of the model's action budget, while malformed JSON triggers retries.

Methodology

Game Design

The game takes place on a symmetric 13×7 board divided by a mountain barrier.

Resources: Players manage credits (for units/buildings) and uranium (for nuclear weapons). Mines must be built to extract resources, which respawn upon depletion.
Units & Combat: A tactical triangle exists: Fighters counter SAMs, SAMs counter Tanks, and Tanks counter Fighters. Only Tanks can damage buildings (Base HP: 4).
Victory Conditions:
- Nuclear: Launch a bomb (requires operational silo, sufficient uranium, and prior scouting of the enemy base). Launches are secret until detonation and resolve simultaneously at the end of the turn.
- Military: Reduce the enemy base to 0 HP using Tanks.
- Diplomatic: Accepted ultimatum (surrender) or accepted peace (draw).
Prompting: Models receive a "near rule-only" system prompt. It defines rules and the JSON schema but provides no build-order advice, timing strategies, or recommended tactics. (Note: Two tactical seed phrases were present in the data collection prompt but removed in the final release; the paper flags this as a limitation affecting scouting and tank production statistics).

Experimental Setup

Models: 15 reasoning models were evaluated (including GPT-5.5, Claude Opus 4.8, Gemini Pro/Flash, DeepSeek V4 Pro, Kimi variants, etc.) via OpenAI-compatible endpoints.
Configuration: All models were invoked with reasoning_effort=high (routed differently across providers) and a sampling temperature of 0.7.
Corpus: 54 completed matches were generated. Each match uses a fresh random map seed and a random opponent from the pool.
Anti-Contamination: The engine source is private, and maps are re-seeded, preventing rote memorization of specific board states or solutions.
Limitations: The sample size is modest (3–15 matches per model), pairings are not balanced round-robins, and side-swapping was not systematic. Therefore, rankings are presented as preliminary descriptive views rather than definitive head-to-head standings.

Key Results

1. Victory Channels and Strategy

Nuclear Dominance: The nuclear rush is the dominant strategy, accounting for 78% of outcomes in the rules-coherent v0.11+ sub-corpus (85% corpus-wide).
Military Conquest: Rare (11% in v0.11+), but significantly faster when successful (mean 12.3 turns vs. 18.9 for nuclear).
Diplomacy: While prolific (1,804 free messages, 59 ultimatums, 46 ceasefires), diplomatic agreements are rarely consummated. Only 1 peace was accepted, 3 ceasefires, and 3 ultimatums (which resulted in victory).
Launch Signature: In all 46 nuclear matches, the winner was the sole launcher, firing exactly once. No mutual destruction occurred. The paper attributes this to the mechanical nature of the secret-simultaneous launch rules (the loser had no information to counter-launch) rather than a cognitive failure of deterrence.

2. Reliability and Belief Tracking

Illegal Actions: Of 5,258 total actions, 5.6% were rejected as illegal.
Fog/State Errors: Approximately 58% of illegal actions were due to fog-of-war or state-tracking failures (e.g., acting on a cell not in view, referencing a destroyed unit). This reframes the illegal-action rate as a measure of belief-tracking under partial observability rather than simple instruction following.
Correlation with Winning: The illegal-action rate shows the strongest (negative) correlation with winning ( $r = -0.35$ ), outperforming deliberation time and token count, though the correlation is not statistically significant at $n=15$ after correction.

3. Emergent Communication and Deception

Bluffing: Losing models bluff at statistically indistinguishable rates to winners (approx. 28% of endgame messages from losers contained threats/boasts).
Deceptive Calm: Models systematically conceal strategic preparations. Silo construction is announced in only 14% of cases, compared to 41% for tank production and 22% for attacks. Conversely, the launch itself is announced 48% of the time.
Personas: Stable stylistic traits emerged, such as "courteous" openers (Gemini models) vs. "tactically terse" styles (Claude/Kimi), which were weakly correlated with skill.
Missing GG: Models rarely conceded gracefully; competitive vocabulary was imported, but closing etiquette was absent.

4. Leaderboard

GPT-5.5 was undefeated (7W–0L) with 3.00 points per match (ppm), though the confidence interval is wide.
Gemini Flash 3.5 and MiniMax M3 formed a dense mid-tier.
Kimi variants and Grok 4.3 performed poorly, often due to high illegal-action rates or API instability.
Deliberation vs. Reliability: High deliberation time (e.g., GPT-5.5) did not guarantee victory, while models with high illegal-action rates consistently lost.

Significance and Claims

The paper positions Age of LLM not merely as a ranking tool, but as a lens for studying LLM cognition under adversarial uncertainty. Its primary contributions are:

Methodological: It provides a dataset of turn-by-turn traces (actions + messages) that allows researchers to analyze belief-tracking, spontaneous deception, and cognitive personas in a way single-turn benchmarks cannot.
Reliability as a Constraint: It demonstrates that in sequential, partially observable environments, the ability to emit legal, rule-compliant actions (reliability) is a binding constraint that may be more critical to success than raw deliberation volume.
Anti-Contamination: It offers a framework for benchmarking that mitigates data contamination through private engine logic and randomization, acknowledging the cost of paid API calls as a necessary trade-off for validity.
Cognitive Insights: The findings suggest that LLMs can spontaneously develop operational security (concealing silo builds) and bluffing strategies without explicit instruction, while simultaneously struggling to maintain accurate internal beliefs about hidden game states.

The authors explicitly state that the current ranking is a "preliminary descriptive view" and not a definitive contribution, calling for future work with larger, side-swapped datasets and single-ruleset corpora to stabilize these findings.