🤖 AI

Math Takes Two: A test for emergent mathematical reasoning in communication

The paper introduces "Math Takes Two," a new benchmark designed to evaluate whether language models can demonstrate true emergent mathematical reasoning by tasking two agents with co-developing a shared symbolic protocol to solve visually grounded problems from first principles.

Original authors: Michael Cooper, Samuel Cooper

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Cooper, Samuel Cooper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Can Robots "Invent" Math?

Imagine you and a friend are stranded on a deserted island. You can’t speak the same language, and you don’t have any numbers written down anywhere. However, you need to trade coconuts. You see a pile of 12 coconuts, and your friend needs to know exactly how many are there so they can trade you some fish.

Since you don't have the word "twelve," you might start pointing, or making specific hand gestures, or even tapping stones in a pattern. Eventually, you might realize, "Hey, every time I tap this specific rock three times, it means a group of four."

You have just invented math through the need to communicate.

This paper, called "Math Takes Two," asks a profound question: Can Artificial Intelligence do the same thing? Can a computer "invent" math from scratch just because it needs to talk to another computer to solve a problem?

The Problem: Pattern Matchers vs. Real Thinkers

Right now, AI (like ChatGPT) is incredibly good at math, but it might be "cheating."

Think of it like a student who has memorized every answer in the back of the textbook. If you ask them a question from the book, they look like a genius. But if you change the numbers or use a weird new symbol they’ve never seen, they often crumble. They aren't reasoning; they are just recognizing patterns they saw during training.

The researchers want to know if AI can move from being a "Pattern Matcher" (someone who memorizes) to a "Reasoner" (someone who understands the underlying rules).

The Experiment: The "Blind Trading" Game

To test this, the researchers created a game. Imagine two AI agents: The Speaker and The Listener.

The Speaker looks at a picture (for example, a grid of 3x4 apples).
The Speaker cannot show the picture. They can only send a short "text message" using a tiny, random alphabet (like A, B, C, 0, 1, 2, +, *).
The Listener receives the message and has to pick the correct picture from a lineup of options.

The Twist: The researchers don't tell the AI what the symbols mean. They don't say "1 means one object." The AI has to figure out, through trial and error, that "Symbol A" might represent a certain shape, or that "Symbol +" might mean "add these groups together."

To make it even harder, they throw a "Curveball" (Out-of-Distribution testing). Once the AI thinks it has the hang of it, the researchers introduce brand-new shapes and much larger numbers that the AI has never seen before. If the AI truly "understands" math, it should be able to use its new "invented language" to solve these new problems.

What Did They Find?

The researchers compared the AI to Humans.

Humans are Math Wizards: Even when given a weird, made-up alphabet, humans were incredibly good at adapting. We quickly realized, "Oh, I see! This symbol represents a row, and that symbol represents a column," and we could solve problems with huge numbers we'd never seen before.
AI is Still a Toddler: The current AI models struggled. They could handle simple tasks, but as soon as the "Curveball" arrived (new shapes or bigger numbers), they got confused. They were good at mimicking what they had seen, but they weren't "inventing" the rules of math the way humans do.

Why Does This Matter?

If we want to build AI that can solve brand-new scientific mysteries or navigate the unpredictable real world, we can't rely on them just memorizing the internet.

We need AI that can build its own tools. This paper provides a "training ground" to see if we can teach machines to do the most human thing of all: take a messy, visual world and turn it into elegant, logical math.

Technical Summary: Math Takes Two: A Test for Emergent Mathematical Reasoning in Communication

1. Problem Statement

Current Large Language Models (LLMs) demonstrate high proficiency in mathematical benchmarks, but it is debated whether this represents true mathematical reasoning or merely statistical pattern matching of learned human formalisms. Most existing evaluations use predefined mathematical languages (e.g., arithmetic notation), which prevents researchers from observing whether a model can construct abstract mathematical concepts from first principles. The authors identify a gap: we do not know if neural architectures can "discover" mathematical reasoning (like counting or multiplication) through the functional pressure of communication, similar to how human mathematical cognition may have co-evolved with the need for precise symbolic exchange.

2. Methodology

The authors propose Math Takes Two, a new benchmark designed to test the emergence of mathematical reasoning in a communicative, multi-agent setting.

The Task: A "Speaker" agent receives a 2D visual image of object arrays (e.g., $m \times n$ grids) and must communicate the quantity to a "Listener" agent using a highly constrained symbolic string. The Listener must then select the correct target image from a candidate set.
Constraints: To ensure the reasoning is emergent rather than pre-trained, the agents are restricted to a fixed 8-token vocabulary: [A, B, C, 0, 1, 2, +, *]. Crucially, the benchmark prohibits the use of pre-trained LLMs or models trained on human language/math corpora.
Experimental Phases:
1. Preconditioning: Unsupervised interaction to discover mappings between visual quantities and symbols.
2. Practice: Sequential trials with novel objects and larger numbers, allowing for bidirectional feedback and adaptation.
3. Test: Evaluation on entirely unseen objects and quantities (Out-of-Distribution/OOD) without feedback, requiring strict extrapolation.
Baselines: The authors evaluate Symbolic Autoencoders (AE)—models that use a Gumbel-Softmax bottleneck to compress visual features into discrete symbols—under both "Frozen" (pre-trained on reconstruction) and "Unfrozen" (end-to-end training) regimes.

3. Key Contributions

Novel Benchmark: A framework that moves beyond image-to-text mapping to test compositional generalization and symbolic extrapolation in a zero-knowledge setting.
Grounded Reasoning: Unlike symbolic AI, this benchmark requires agents to ground abstract symbols in visual, perceptual stimuli.
Human-in-the-loop Baseline: A comparative study involving human participants to establish a "gold standard" for how biological agents develop notation and strategy under similar constraints.
OOD Evaluation Framework: A systematic method for testing whether a learned protocol can handle "novelty" (new symbols and numbers), which is the hallmark of true reasoning.

4. Results

Human Performance: Human participants demonstrated high accuracy (approx. 87% in the test phase) and were highly adept at adapting their symbolic languages in real-time to handle OOD tasks.
Model Performance:
- All machine baselines (Symbolic AEs and Vision-Language Transformers) lagged behind humans, particularly on OOD tasks.
- The "Generalization Gap": Models struggled most when faced with novel visual primitives (OOD symbols).
- Overfitting Risk: Interestingly, training the symbolic compression layer end-to-end with the task (Unfrozen) improved performance on practice questions but decreased accuracy on the test set, suggesting that models often learn to "cheat" via in-domain heuristics rather than developing generalizable mathematical rules.

5. Significance

This work shifts the paradigm of mathematical evaluation from verification of known syntax to the observation of emergent abstraction. By stripping away human linguistic scaffolding, Math Takes Two provides a rigorous testbed for determining if artificial agents can achieve "bottom-up" mathematical reasoning. The findings suggest that current symbolic compression methods are insufficient for robust generalization, highlighting the need for architectures that can model uncertainty, utilize external memory, and proactively anticipate novelty—traits observed in successful human communicators.