💻 computer science

The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models

This paper investigates whether language models can sustain philosophical conceptual analysis through iterated counterexample and repair cycles, finding that while they can engage in the process, the loop quickly yields diminishing returns characterized by increasing verbosity without improved accuracy and a tendency for the models to over-accept invalid counterexamples compared to human judges.

Original authors: Daniel Drucker, Kyle Mahowald

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Drucker, Kyle Mahowald

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of philosophers playing a high-stakes game of "Gotcha."

One person proposes a simple definition for a word, like "A game is something you do for fun." The next person tries to break that definition by finding a real-life situation where the definition fails (a counterexample). For instance, "What about a professional poker player who is stressed and hates the game but keeps playing to win money? That's still a game, but it's not 'for fun'."

The first person then has to fix their definition to include the poker player but still exclude things that aren't games. They try again, and the cycle repeats. This is called Conceptual Analysis, and it's how philosophers have refined ideas for thousands of years.

The researchers in this paper asked: Can AI language models play this game?

They set up a digital version of this game where one AI acts as the "Proposer" (making definitions) and another acts as the "Critic" (finding holes in the definitions). They let them play this loop over and over again—sometimes 50 times in a row—to see if the AI gets smarter at defining things like "friend," "lie," or "sandwich."

Here is what they found, translated into everyday terms:

1. The AI Judge is a bit too generous

In the game, the "Critic" needs to spot bad definitions. The researchers had human experts and an AI judge rate the AI's counterexamples.

The Result: The AI judge said "Yes, that's a valid counterexample!" about twice as often as the human experts did.
The Metaphor: Imagine a strict teacher (the human) and a very eager student (the AI) grading a test. The student thinks every wrong answer is a brilliant trick question, while the teacher knows some are just mistakes. However, they did agree on the "big hits"—the really obvious flaws. So, while the AI is a bit too easy on itself, it isn't completely hallucinating.

2. The "Longer is Better" Trap

The researchers expected that as the AI played more rounds, the definitions would get sharper and more accurate, just like a human philosopher refining an idea over years.

The Result: Instead of getting better, the definitions just got longer and more wordy.
The Metaphor: Imagine trying to describe a "dog" to an alien.
- Round 1: "A dog is a furry animal." (Too broad: includes cats).
- Round 2: "A dog is a furry animal that barks." (Too broad: includes some seals).
- Round 50: "A dog is a furry animal that barks, has four legs, is not a seal, is not a cat, usually has a tail, likes bones, and is not a robot, unless it's a specific type of robot dog..."
  The AI kept adding "exceptions" and "rules" to patch every hole, but it never actually figured out the core truth of what a dog is. The definition became a massive, clunky laundry list of rules rather than a clear, simple insight.

3. Some Concepts are Just Harder to Pin Down

The researchers tested 20 different concepts. They found that some words were easy for the AI to "break" with counterexamples, while others were nearly impossible.

Easy to break: Words like "neighbor" or "friend." The AI could easily find weird edge cases (e.g., "Is a neighbor someone you've never met but lives next door?").
Hard to break: Words like "mistake" or "expert." These concepts seemed to resist the game. The AI struggled to find valid counterexamples, suggesting these ideas are inherently fuzzier or harder to define with strict rules.

4. The "Memory" Didn't Help

The researchers tried two versions of the game:

Memoryless: The AI only sees the current definition.
With History: The AI sees the entire history of every previous definition and counterexample.

The Result: Having the history didn't make the AI any better. It didn't learn from its past mistakes. It just kept making the same kind of mistakes, but with more words.

The Bottom Line

The paper concludes that while AI can play the "Counterexample Game" and understand the rules, it hits a wall very quickly. It doesn't get smarter with practice; it just gets more verbose.

The researchers suggest this is a great way to test AI. If an AI can't sustain high-level, iterative philosophical reasoning (getting better over time), it tells us something important about how these models think: they are good at patching holes in the short term, but they struggle to build a stable, deep understanding of complex ideas over the long term.

In short: The AI can play the game, but it's like a player who keeps adding more and more rules to the rulebook without ever actually understanding the sport.

Technical Summary: The Counterexample Game

Problem Statement

Conceptual analysis—the process of proposing definitions and refining them through counterexamples—is a foundational method in philosophy. While recent work has utilized "actor-critic" paradigms (where one model generates a proposal and another critiques it) to improve language model (LM) performance in various domains, it remains unclear whether LMs can sustain high-level, iterated philosophical reasoning. Specifically, it is unknown if LMs can engage in the "Counterexample Game"—a process where a definition is challenged by a counterexample, repaired, and then challenged again—to produce increasingly accurate conceptual analyses, or if this process leads to diminishing returns.

Methodology

The authors implemented an iterated conceptual analysis framework involving 20 distinct concepts (10 nouns, 10 verbs) varying in abstractness and social complexity.

The Loop: The process begins with a human-authored seed analysis ( $A_0$ ). In each iteration $i$ , a language model generates a counterexample ( $CE_i$ )—a concrete scenario the current analysis misclassifies. A second model instance then repairs the definition ( $A_i$ ) to accommodate the counterexample without trivializing the analysis or switching concepts.
Experimental Conditions:
- Mixed-Model Chains: Three models (Claude 3.5 Sonnet, GPT-5, Gemini 2.0 Flash) were randomly alternated to stress-test chain dynamics under provider heterogeneity. These chains ran for 50 iterations per concept.
- Self-Play Chains: A single, high-capability model (Claude Opus 4.5) generated both counterexamples and repairs. These chains ran for 10 iterations per concept.
- Memory vs. History: Chains were run in "memoryless" conditions (seeing only the current analysis) or "with-history" conditions (seeing the full prior chain of counterexamples and repairs).
Evaluation:
- Human Judgment: Five raters (two authors and three philosophy graduate students) evaluated a subset of counterexamples for validity (true/false positives/negatives) and importance (1–5 scale).
- Automated Judgment: Claude Opus 4.5 served as an automated judge for the majority of counterexamples and for rating the accuracy and conciseness of the evolving definitions.
- Sub-concept Tagging: A tagging system was developed to identify atomic, testable sub-concepts within definitions to track how conceptual content evolved and oscillated over iterations.

Key Results

1. Validity of Counterexamples

Human-LM Agreement: There is moderate but substantive agreement between human raters and the LM judge regarding counterexample validity ( $\kappa_{val} = 0.42$ for human consensus vs. Opus). However, Opus accepts roughly twice as many counterexamples as humans (60% vs. 32% in the sampled subset), suggesting the LM judge may be overly permissive.
Degradation Over Iteration: Counterexample validity degrades significantly as the chain progresses. In mixed-model chains, validity drops from 48% at iteration 0 to 9% by iteration 49. Even in self-play chains, validity falls from 69% (first counterexample) to 17–30% by iterations 5–9.
Concept Difficulty: Validity rates vary significantly by concept. "Easy" concepts to attack (high validity) include neighbor and to forgive, while "hard" concepts include mistake and to cheat. These differences are consistent across independent chains, suggesting they reflect genuine properties of the concepts rather than noise.

2. Quality of Definitions

Verbosity vs. Accuracy: Extended iteration leads to definitions that become increasingly verbose (growing from ~11 words to ~95 words) without improving accuracy. Accuracy plateaus after the first two iterations, while conciseness steadily degrades.
Looping and Instability: Definitions often become list-like, accumulating exceptions rather than forming stable, stable conceptual insights. Some concepts exhibit high oscillation in their sub-concepts (e.g., expert), where criteria appear and disappear across iterations, while others remain stable.
History Does Not Help: Providing models with the full history of prior iterations does not improve counterexample validity or definition quality compared to memoryless conditions.

3. Systemic Differences

Reasoning Styles: The LM tends to be verbose and formal, often citing external authorities or specific clauses to justify repairs. Humans tend to provide brief, intuitive verdicts and hedge more frequently.
Failure Modes: When models do produce valid counterexamples, the repair process often fails to achieve genuine conceptual insight, instead responding with a "laundry list" of exceptions.

Significance and Claims

The paper positions this work as a "proof of concept" for using philosophical methodology to study language models. The authors claim:

Iterative Repair Limits: While LMs can engage in philosophical reasoning, the counterexample-repair loop hits diminishing returns quickly. Unlike human philosophers who may refine analyses over long periods, LMs in this setting produce longer, less natural definitions without gaining accuracy.
Evaluation Benchmark: The Counterexample Game serves as a fruitful test case for evaluating whether LMs can sustain high-level iterated reasoning. The findings suggest that current actor-critic paradigms may fail in domains requiring deep conceptual stability, as the "critic" often fails to identify genuine errors, or the "generator" responds with verbosity rather than insight.
Concept Heterogeneity: The study highlights that not all concepts are equally amenable to definition. Some concepts (e.g., game) appear to resist stable definitions in general, while others (e.g., to lie) benefit from the model's pre-training on philosophical literature.
Verbosity Bias: The authors suggest that LMs may equate the length of a formulation with quality, a "verbosity bias" that could explain the degradation in conciseness without corresponding gains in accuracy.

The authors conclude modestly that while there is sufficient signal in the task to proceed with automated analysis, the results indicate that LMs currently struggle to replicate the long-term, self-improving nature of human philosophical conceptual analysis.