AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios

This paper introduces AgentCoMa, a new benchmark demonstrating that while large language models can handle isolated commonsense and mathematical reasoning steps, their performance significantly degrades when combining both types of reasoning in real-world scenarios, revealing a substantial brittleness that is not observed in human annotators or prior benchmarks.

The Gist

Imagine you are hiring a very smart, well-read robot assistant to help you plan your life. You ask it to do two things at once: figure out what makes sense (like "I shouldn't mop a carpet") and do the math (like "If I mop 10 square meters, how much time does that take?").

You'd expect this robot to be great at both. And it is! If you ask it just the common sense question, it gets it right. If you ask it just the math question, it gets that right too.

But here is the twist: When you ask it to do both together, the robot suddenly starts acting like it's lost its mind. It forgets the common sense rule, or it gets the math wrong, even though it knew both answers perfectly when asked separately.

This is exactly what a new research paper called AgentCoMa discovered.

The "Brain Glitch" Experiment

The researchers created a special test (a benchmark) called AgentCoMa. Think of it as a gym workout for AI brains, but instead of lifting weights, the AI has to combine two different types of thinking:

1. Common Sense (The "Fast Brain"): Like knowing that you can't iron a wool sweater because it will shrink, or that you shouldn't vacuum a leather chair. This is intuitive, everyday knowledge.
2. Math (The "Slow Brain"): Like calculating the total cost of groceries or figuring out how many days a project will take. This requires logic and numbers.

They tested 61 different AI models (from small ones to massive super-brains) on these mixed tasks.

The Big Surprise: The "Compositionality Gap"

Here is what happened:

• The Isolated Test: When the AI was asked only the common sense part, it scored 85%. When asked only the math part, it also scored 85%.
• The Combined Test: When asked to do both in one go, the score plummeted to 42%.

That is a 43% drop in performance!

To put this in perspective, the researchers also asked regular humans (non-experts) to take the test. The humans didn't have this problem. They could do the common sense part, the math part, and the combined part with almost the same high accuracy. The AI, however, was falling apart.

Why is the AI failing? (The "Neuron" Mystery)

The researchers didn't just stop at the score; they looked inside the AI's "brain" (its neural network) to see what was going wrong. They found three main reasons:

1. The "Unfamiliar Recipe" Problem:
   Imagine a chef who is a master at making pizza and a master at making pasta. But if you ask them to make a "Pasta-Pizza" (a weird hybrid dish), they freeze. They have never seen a recipe that mixes these two specific things together in their training data. The AI is used to seeing math questions or common sense questions, but rarely both mixed up. It doesn't know how to "switch gears" quickly enough.

2. The "One-Track Mind" Problem:
   When the AI tries to solve the mixed problem, it gets confused and decides to ignore one part of the brain.

   • The Metaphor: Imagine you are driving a car that has two engines: one for steering (common sense) and one for speed (math). When you try to drive a tricky road, the car decides to turn off the steering engine and just floor the gas pedal. It zooms forward with perfect math but crashes because it forgot to steer around the common-sense obstacles.
   • The study found that when solving these mixed tasks, the AI mostly activated the "math neurons" and completely ignored the "common sense neurons."

3. The "Hallucination" Effect:
   Because the AI is ignoring the common sense part, it starts making things up. It might confidently calculate the cost of mopping a carpet, even though carpets can't be mopped. It sounds confident and logical, but it's completely wrong because it lost the context.

What Does This Mean for the Future?

This paper is a wake-up call. It shows that even the smartest AI models today are brittle. They are like a student who can ace a math test and a history test separately, but if you give them a word problem that requires history knowledge to solve the math, they panic.

The Takeaway:
To build truly helpful AI agents (robots that can plan your trip, manage your budget, and cook your dinner), we can't just make them smarter at math or better at facts. We need to teach them how to blend these skills together. We need to train them to keep both their "fast brain" and "slow brain" active at the same time.

Until then, if you ask an AI to do something complex that mixes logic and real-world rules, you might want to double-check its work!

Technical Summary

Here is a detailed technical summary of the paper "AgentCoMa: A Compositional Benchmark Mixing Commonsense and Mathematical Reasoning in Real-World Scenarios."

1. Problem Statement

Large Language Models (LLMs) have demonstrated high proficiency in isolated reasoning tasks, such as pure commonsense reasoning (e.g., understanding social norms) or pure mathematical reasoning (e.g., arithmetic word problems). However, real-world agentic tasks often require the composition of multiple reasoning types. For instance, an agent planning a budget must first use commonsense to identify valid items (e.g., knowing carpet cannot be mopped) and then use math to calculate costs.

Current benchmarks fail to adequately evaluate this mixed-type compositional reasoning. Existing compositional benchmarks (like Bamboogle or MultiArith) typically chain steps of the same reasoning type. Consequently, there is a lack of systematic evaluation for LLMs attempting to combine distinct reasoning modalities (System 1/Intuitive vs. System 2/Deliberative) within a single task.

2. Methodology

A. The AgentCoMa Benchmark

The authors introduce AgentCoMa, a dataset designed to test the ability of LLMs to combine commonsense and mathematical reasoning.

• Structure: Each question consists of two sequential steps:
  1. Commonsense Step: A "which" question requiring the selection of items based on real-world knowledge (e.g., identifying which objects are electrical or safe for room temperature).
  2. Math Step: A single arithmetic operation (addition, subtraction, multiplication, or division) performed on the result of the first step.
• Domains: Questions are grounded in five realistic agentic scenarios: House Working, Web Shopping, Science Experiments, Smart Assistant, and Travel Agent.
• Data Creation: 260 samples were manually created by expert annotators. Crucially, the authors verified that even advanced LLMs (like GPT-4o) fail to generate high-quality samples of this specific mixed-type structure, necessitating human creation.
• Validation: A rigorous multi-step validation process ensures that samples are solvable by humans, require genuine commonsense choices, and involve only one arithmetic operation.

B. Experimental Setup

• Models Evaluated: The benchmark was tested on 61 contemporary LLMs, ranging from 1.5B to 141B parameters. The selection includes:
  • Instruction-tuned models.
  • Reasoning models optimized via Supervised Fine-Tuning (SFT).
  • Reasoning models optimized via Reinforcement Learning (RL).
• Evaluation Protocol:
  • Models were tested on the compositional question (full task).
  • Models were also tested on the individual sub-questions (commonsense only, math only) to establish a baseline.
  • Metrics: Accuracy was measured for each step individually, the intersection of both steps being correct ("Both Correct"), and the final compositional accuracy.
  • Human Baseline: Non-expert human annotators (high school education level) were also tested to establish a human performance ceiling.

C. Interpretability Analysis

To understand why models fail, the authors conducted several diagnostic studies:

1. Context Length Analysis: Testing if the performance drop is due to longer context windows.
2. Training Data Similarity (MIA): Using Min-K%++ to measure how "surprised" models are by the data, indicating if mixed-type reasoning patterns exist in their training corpora.
3. Attention Maps: Analyzing "lookback attention" to see if models attend to the full context or hallucinate.
4. Neuron Pattern Analysis: Using Query-Relevant Neuron Cluster Attribution (QRNCA) to identify which neurons are activated during commonsense, math, and compositional tasks.

3. Key Contributions

1. AgentCoMa Dataset: The first systematic benchmark for evaluating mixed-type (commonsense + math) compositional reasoning in agentic settings.
2. Discovery of the "Compositionality Gap": The identification of a massive performance drop (~30%) when LLMs attempt to combine reasoning types, a gap significantly larger than that observed in same-type compositional benchmarks.
3. Mechanistic Explanation: Evidence showing that LLMs fail to activate the correct neural circuits for mixed reasoning. Instead, they revert to circuits associated with only one reasoning type (typically math), ignoring the commonsense constraints.
4. Human vs. Model Comparison: Demonstration that non-expert humans solve these tasks with high accuracy, proving the difficulty lies in model architecture/training, not task complexity.

4. Key Results

A. Performance Gap

• Isolated Steps: LLMs perform well on individual steps. The median accuracy for the first (commonsense) step is >85%, and the second (math) step is >85%. They can solve both steps in isolation correctly in nearly 75% of cases.
• Compositional Task: When the steps are combined, the median accuracy drops to 42%.
• The Gap: This results in an average compositionality gap of ~29%.
• Comparison: In contrast, on benchmarks like Bamboogle (knowledge-based) and MultiArith (math-based), the compositionality gap is negligible (<1% to 5%). In those cases, failure is usually due to getting a sub-step wrong. In AgentCoMa, models often get the sub-steps right but fail the composition.

B. Human Performance

Non-expert humans achieved 82.8% accuracy on the compositional task, compared to the models' ~42%. Humans also solved the individual steps with high accuracy, showing that the task is not inherently too difficult for humans.

C. Model Trends

• Model Size & Type: The gap persists across all model sizes (3B to 141B) and training strategies (Instruction-tuned, SFT, RL). Even sophisticated RL-tuned reasoning models (e.g., DeepSeek-R1, SimpleRL) struggle as much as standard instruction-tuned models.
• Failure Mode: In approximately 74% of compositional failures, the model successfully solved both individual steps in isolation but failed to combine them correctly.

D. Interpretability Findings

• Training Data Scarcity: Min-K%++ scores were significantly lower for compositional questions than for individual steps, suggesting mixed-type reasoning patterns are rare in training data.
• Neuron Activation: QRNCA analysis revealed that when solving the compositional task, models activate neurons heavily overlapping with the math step (39–54% overlap) but show negligible overlap with the commonsense step (3–10% overlap).
• Hallucination: Models tend to ignore the commonsense context during the math step, leading to "contextual hallucination" where they calculate based on incorrect premises (e.g., mopping a carpet).

5. Significance and Conclusion

The paper highlights a fundamental brittleness in current LLMs: they are excellent at specialized reasoning but struggle to integrate disparate reasoning types required for real-world agency.

• Implication: Simply scaling models or using Reinforcement Learning on single-type tasks does not solve the problem of mixed-type composition.
• Future Work: The findings suggest that future training strategies must explicitly expose models to mixed-reasoning patterns to prevent the "neuron collapse" where the model defaults to a single reasoning circuit.
• Benchmark Utility: AgentCoMa provides a rigorous testbed for evaluating progress in agentic reasoning, moving beyond isolated skill checks to holistic task execution.

In summary, AgentCoMa reveals that while LLMs have mastered the components of reasoning, they have not yet learned the "glue" required to synthesize different reasoning modalities into a coherent solution for real-world problems.