🤖 AI

MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning

The paper introduces MIRA-Math, a deterministic benchmark comprising 2,310 mathematical problems where solvers must identify and request a single missing atomic fact under a strict budget before computing the final answer, thereby enabling the separate evaluation of information-seeking and reasoning capabilities in language models.

Original authors: Charbel Al Bateh, Samer Saab Jr

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Charbel Al Bateh, Samer Saab Jr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like a Sudoku or a math problem. Usually, when you take a test or use a smart computer program, you are given all the pieces you need to finish the picture. You just have to put them together correctly.

But in the real world, information is often missing. You might know the rules of a game, but you don't know the score. Or you know a recipe, but you don't know how much salt to add.

MIRA-Math is a new "test" designed to see if AI models (like the smart chatbots we use today) can handle this specific situation: knowing what they don't know, and asking for exactly the right piece of information to finish the job.

Here is how the paper explains it, using simple analogies:

1. The "Missing Ingredient" Game

Most math tests for AI are like giving a chef a full recipe and all the ingredients, then asking them to bake a cake. The test is: "Can you bake the cake?"

MIRA-Math is different. It gives the AI a recipe that is deliberately incomplete.

The Setup: The AI sees a math problem, but one crucial number or fact is hidden (like a missing ingredient).
The Goal: The AI must realize something is missing, ask for it in plain English, and then use that new fact to solve the problem.
The Catch: The AI has a strict "budget." It can only ask a few questions. If it asks the wrong thing, or asks too vaguely, it fails.

2. The "Strict Librarian"

To make this test fair and precise, the paper introduces a special character: a Fixed Information Holder (think of them as a very strict, literal-minded librarian).

The Librarian's Job: This librarian holds the one missing fact. They do not help you solve the problem. They do not give you hints. They do not explain things.
The Rules:
- If you ask, "What is the missing number?" (too vague), the librarian says, "I don't have that."
- If you ask, "What is the value of variable X?" (precise), and the librarian has that specific fact, they hand it over.
- If you ask for something they don't have, they say, "No."

This setup tests if the AI can be precise in its questions, not just smart at math.

3. Two Types of Challenges

The paper created 2,310 different puzzles, split into two categories:

Type A (The "Fixed Slot"): The AI knows where the missing piece is.
- Analogy: Imagine a form with a blank line that says "Date of Birth: ______". The AI knows it needs to ask for the date. The test is: Can it ask for the date correctly, and then use it to solve the math?
Type B (The "Hidden Slot"): The AI has to find where the missing piece is.
- Analogy: Imagine a form where one of the ten lines is blank, but it doesn't say which one. The AI has to look at the whole form, figure out, "Oh, the 'Phone Number' line is empty," and then ask for that specific thing. This is harder because the AI has to diagnose the problem first.

4. What the Tests Revealed

The researchers tested many different AI models (some very smart, some smaller) and found some surprising things:

Asking and Solving are Different Skills: Just because an AI is good at asking the right question doesn't mean it's good at doing the math afterward.
- Example: One AI asked for the perfect missing fact (like a great detective), but then failed to do the calculation (like a bad accountant).
- Example: Another AI did the math perfectly but asked for the wrong fact, so it never got the answer.
Practice Doesn't Always Help: The researchers tried showing the AI examples of how to ask questions (like a "cheat sheet" or "4-shot prompting"). Sometimes this helped, but often it made the AI worse.
- Why? The AI started copying the examples too rigidly. Instead of looking at the specific problem at hand, it just repeated the pattern it saw in the examples, even when the problem needed a different question.

5. Why This Matters

The authors say this benchmark is like a diagnostic tool for doctors.

Current tests tell you if a student can solve a problem if they have all the notes.
MIRA-Math tells you if a student can identify a gap in their knowledge and communicate effectively to fill it.

The paper concludes that for AI to be truly useful in the real world, where information is often incomplete, it needs to master this specific skill: knowing exactly what to ask for, and then using that answer correctly.

In short: MIRA-Math isn't just testing if AI can do math; it's testing if AI can admit what it doesn't know and ask the right question to fix it, without getting confused or guessing.

Technical Summary: MIRA-Math

Problem Statement

Current mathematical reasoning benchmarks (e.g., GSM8K, MATH) typically provide all necessary facts to solve a problem, assessing only the model's ability to compute a final answer from a complete state. Conversely, broad interactive benchmarks often conflate mathematical reasoning with tool use, retrieval, web navigation, and long-horizon dialogue management. This creates a diagnostic gap: it is difficult to isolate whether a model fails because it cannot identify missing information, cannot request that information precisely, or cannot integrate the acquired information into a correct solution.

MIRA-Math addresses this by introducing a benchmark for minimal information requesting. The core problem is defined as solving mathematical problems where the full latent state has a unique answer, but the solver's view is deliberately incomplete, missing exactly one necessary atomic fact. The solver must recognize the missing fact, request it in natural language under a strict budget, and integrate the returned fact to produce an exact final answer.

Methodology

Task Formalization

Each instance is generated from a latent complete state $z$ determining a unique target answer $y = f(z)$ . The solver receives a private view $v_A = \phi_A(z)$ that omits one necessary atomic fact. The benchmark enforces two conditions:

Global Well-Posedness: The full state (view + missing fact) yields exactly one consistent answer.
Local Underdetermination: The solver's private view alone yields multiple consistent answers.

Interaction Protocol

The evaluation involves two agents:

Solver (Agent A): Receives the private view and a request budget $B$ . It issues free-form natural language requests or provides a final answer.
Information Holder (Agent B): A fixed, constrained LLM (implemented as gpt-4o-mini in experiments) that holds only the missing atomic fact. It operates under a structured-output protocol:
- Offer: If the request semantically matches the held fact, it returns the fact with has_exact_match=true.
- Decline: If the request does not match, it returns a fixed declination message.
- The holder does not solve, explain, negotiate, or volunteer extra hints.

Dataset Construction

MIRA-Math contains 2,310 instances derived from 22 typed mathematical families (covering algebra, probability, linear systems, signal processing, Markov chains, etc.). The dataset is divided into two types:

Type A (Fixed Hint Slot): The missing information is structurally determined (e.g., a Bayes rule instance always withholds the prior).
Type B (Variable Hint Slot): The missing position varies per instance (e.g., a missing coefficient in a linear system or a missing resistor in a circuit), requiring the solver to locate the absent slot before requesting.

The release includes generator code, exact verifiers, canonical hint specifications, and raw interaction logs to ensure reproducibility.

Metrics

The benchmark separates information acquisition from mathematical integration using the following metrics:

Request Hit Rate: Ratio of accepted requests to total messages.
First-Request Success: Fraction of instances where the first request is accepted.
Trace-Level Decomposition: Instances are classified into three disjoint categories:
1. No Canonical Hint Acquired: The solver failed to request the correct fact.
2. Hint Acquired, Integration Failed: The solver got the fact but computed the wrong answer.
3. Success: The solver acquired the fact and computed the correct answer.

Key Results

Experiments across frontier and small models (including gpt-5.1, Gemma-4, gemini-2.5-flash, Grok-4.20, and Llama-3.1-8B) reveal that requesting and solving are separable skills:

Divergence of Skills: High request performance does not guarantee high final-answer accuracy. For example, gemini-2.5-flash achieved the highest request hit rate in zero-shot settings but had lower final accuracy than gpt-5.1 and Gemma-4. Conversely, Gemma-4 achieved the highest zero-shot final accuracy without achieving the highest request metrics.
Failure Modes:
- Acquisition Failures: Models often fail to identify the specific missing slot or ask too vaguely, leading to repeated declines.
- Integration Failures: Stronger models often successfully acquire the canonical hint but fail at the downstream exact arithmetic, symbolic reconstruction, or answer normalization.
Prompting Effects: Four-shot prompting (providing examples of minimal requests) did not uniformly improve performance. In Type B families, where the missing slot varies, few-shot examples sometimes induced "exemplar anchoring," causing models to imitate demonstrated request forms rather than adapting to the specific missing slot, potentially degrading performance.
Type A vs. Type B: Type B families (variable slots) were not uniformly harder in terms of final accuracy; strong models often performed better on Type B than Type A. However, Type A families contained tasks where the request was easy but the downstream computation was brittle (e.g., exact Chinese remainder reconstruction).

Significance and Claims

The paper positions MIRA-Math not as a replacement for full-information mathematical benchmarks, but as a diagnostic tool to expose hidden structure and evaluation gaps. Its primary significance lies in:

Isolating a Specific Bottleneck: It explicitly measures the capability to recognize underdetermined problems, identify the missing atomic fact, and request it precisely, separating this from the ability to compute the final answer.
Data-Centric Design: Following recent trends in responsible dataset design, the release prioritizes reproducibility by providing generators, verifiers, and raw logs, allowing the community to audit the "request-to-solve" pipeline rather than relying solely on aggregate accuracy.
Refining Evaluation: The results demonstrate that a model's performance on standard math benchmarks does not necessarily imply competence in information acquisition. A model may be strong at math but fail MIRA-Math if it cannot identify what is missing, or it may be good at asking but fail the math.

The authors conclude that MIRA-Math serves as a baseline suite for future models and an analysis tool for studying when language models ask for missing mathematical information rather than guessing, without claiming to measure open-ended collaboration, human communication, or real-world decision-making.