💬 NLP

Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance

This paper identifies a critical gap between the usage of reasoning strategies in successful solutions and their executability when applied as guidance, leading to the proposal of Selective Strategy Retrieval (SSR), a framework that leverages human-model differences to selectively combine strategies and significantly improve mathematical reasoning accuracy across diverse benchmarks.

Original authors: Weida Liang, Yiyou Sun, Shuyuan Nan, Chuang Li, Dawn Song, Kenji Kawaguchi

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Weida Liang, Yiyou Sun, Shuyuan Nan, Chuang Li, Dawn Song, Kenji Kawaguchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a very tricky math puzzle. You have a brilliant human expert and a super-fast computer robot. Both of them can solve the puzzle correctly, but they do it in completely different ways.

The human expert might say, "Look at the shape of this problem; it's like a triangle, so we should use a special triangle rule."
The robot might say, "Let's just plug all the numbers into a giant formula and crunch the math until we get an answer."

The Problem:
Usually, when we try to help a computer solve a problem, we give it examples of how a human solved similar problems. We assume, "If a human did it this way, the computer should be able to copy it."

But this paper discovered a big surprise: Just because a strategy works for a human doesn't mean the computer can actually do it.

Think of it like this:

The Human Strategy is like giving a chef a recipe that says, "Add a pinch of 'soul' and 'intuition'." A human chef knows exactly what that means. But if you give that same instruction to a robot chef, it gets confused. It doesn't know how to measure "soul." The strategy exists, but the robot can't execute it.
The Robot Strategy is like a recipe that says, "Add exactly 3.14 grams of salt." The robot can do this perfectly. But if you give this to a human chef, they might think, "Why so specific? I need to taste it first."

The authors call this gap "Strategy Executability." It's the difference between a strategy being present in a solution and a strategy being doable by the specific model you are using.

The Solution: The "Smart Guide" (Selective Strategy Retrieval)
The researchers built a new system called SSR (Selective Strategy Retrieval). Instead of blindly copying human examples or robot examples, SSR acts like a smart matchmaker.

Here is how SSR works, using a metaphor:

Imagine you are a Travel Agent (the AI model) trying to get a traveler (the math problem) to a destination (the correct answer). You have two guidebooks:

The Human Guidebook: Full of beautiful, high-level maps and scenic routes.
The Robot Guidebook: Full of turn-by-turn GPS directions and traffic data.

In the past, travel agents would just pick one guidebook and hope for the best. Sometimes the scenic route was too vague for the GPS, and sometimes the GPS was too boring for the traveler.

SSR is the Super Travel Agent.
Before giving directions, SSR checks:

"Is this a geometry problem? The Human Guidebook is great for spotting shapes, but the Robot Guidebook is better for calculating angles."
"Is this an algebra problem? The Robot Guidebook's step-by-step math is perfect here."

SSR doesn't just pick one book. It mixes and matches. It might say, "Use the Human's big-picture idea to start, but switch to the Robot's step-by-step math to finish." It only picks the instructions that the Travel Agent (the AI) is actually capable of following right now.

Why This Matters
The paper tested this on hard math competitions (like the AIME).

Old Way: Giving the AI a human example improved the score a little bit, but sometimes it made things worse because the AI got confused trying to copy the human's "intuition."
SSR Way: By picking the executable strategies (the ones the AI can actually do), the AI's accuracy jumped significantly. On some hard tests, it improved by 13 points, which is a huge deal in math competitions.

The Big Takeaway
This paper teaches us that copying isn't always learning. Just because a human does something a certain way doesn't mean an AI should try to copy it exactly.

To make AI smarter, we need to understand how the AI thinks, not just how humans think. We need to give the AI tools it can actually use, rather than tools that look good on paper but are too complex for its "brain" to handle.

In a nutshell:

The Gap: Humans and AI solve problems differently.
The Mistake: Assuming AI can copy human methods perfectly.
The Fix: A system (SSR) that picks the right mix of human and robot strategies based on what the AI can actually do.
The Result: Smarter, more reliable AI math solving.

1. Problem Statement

While example-based guidance (e.g., in-context learning, worked examples) is widely used to improve Large Language Models' (LLM) mathematical reasoning, its effectiveness is highly unstable. Even when guidance is correct and semantically relevant, it often fails to improve performance or can even degrade it.

The authors identify a critical, underexplored gap between Strategy Usage and Strategy Executability:

Strategy Usage: Whether a reasoning strategy appears in a successful solution (e.g., a human or model solving a problem correctly).
Strategy Executability: Whether that same strategy, when extracted and provided as explicit guidance to a target model, actually enables the model to produce a correct solution.

The core problem is that strategies prevalent in successful solutions (especially those from humans) are not necessarily executable by the target model due to structural mismatches in how humans and models reason.

2. Methodology

A. Diagnostic Analysis: Human-Model Dissociation

The authors constructed HM-ReasoningBench, a dataset of 4,850 competition-level math problems, each paired with a human-written solution and a model-generated solution.

Strategy Abstraction: Solutions were decomposed into high-level reasoning strategies (e.g., "coordinate setup," "auxiliary construction," "prime factorization") mapped to a library of 30 canonical templates.
Key Findings:
- Domain-Dependent Divergence: Humans favor structural, conceptual, and geometric strategies (e.g., symmetry, auxiliary lines), while models favor procedural, algebraic, and coordinate-based strategies.
- Usage vs. Executability Gap: A strategy frequently used by humans (e.g., geometric intuition) often fails when given as guidance to a model. Conversely, procedural strategies from models are often more executable by models but less effective for structural problems.
- Complementarity: Neither source is universally superior; the optimal strategy depends on the problem domain and the target model's operational strengths.

B. Proposed Solution: Selective Strategy Retrieval (SSR)

To address this, the authors propose SSR, a test-time framework that explicitly models and optimizes for strategy executability. SSR does not require model retraining.

Core Components of SSR:

Strategy Knowledge Graph ( $G$ ): A heterogeneous graph connecting Problems, Strategies, and Categories. It encodes observed problem-strategy associations and category memberships.
Graph Representation Learning: A Heterogeneous Graph Neural Network (GNN) with transformer-style message passing learns node embeddings. It is trained via contrastive learning to separate "executable" (problem-strategy pairs that lead to success) from "non-executable" pairs.
Multi-Route Retrieval: For a new problem $x$ $x$ , SSR retrieves candidate strategies via three complementary routes:
- Route A (Category-Conditioned): Retrieves strategies proven effective for the problem's category.
- Route B (Problem-Transfer): Retrieves strategies that worked on semantically similar problems in the training set.
- Route C (Semantic Fallback): Retrieves strategies based on surface semantic similarity if other signals are sparse.
Executability Prediction: A supervised predictor estimates the utility $U(s|x)$ of a strategy $s$ for problem $x$ . It uses features from the graph embeddings and retrieval route indicators to predict the probability of success under a fixed prompting protocol.
Selection: SSR ranks candidates by their predicted executability score and selects the top strategies (up to 5) to guide the model.

3. Key Contributions

Conceptual Insight: Identification of the systematic dissociation between strategy usage (prevalence in correct solutions) and strategy executability (effectiveness as guidance), explaining the instability of current guidance methods.
Dataset: Creation of HM-ReasoningBench, a paired dataset of human and model solutions for controlled analysis of reasoning strategies.
Framework: Development of Selective Strategy Retrieval (SSR), a lightweight, test-time framework that operationalizes executability through empirical signals and source-aware selection.
Empirical Validation: Demonstration that modeling executability yields robust improvements over direct solving, in-context learning, and single-source guidance across diverse benchmarks and model sizes.

4. Experimental Results

The authors evaluated SSR on three benchmarks: HM-ReasoningBench, AIME 2025, and MathArena Apex, using models like Qwen3-8B/14B and DeepSeek-R1-Distill-Qwen-7B.

Performance Gains:
- SSR consistently outperformed Direct Solving (DS), In-Context Learning (ICL), and single-source guidance (Human-only or Model-only).
- AIME 2025: Improved accuracy by +13 points (e.g., Qwen3-8B: 67.33% $\to$ 74.00%).
- Apex: Improved accuracy by +5 points (e.g., Qwen3-8B: 8.16% $\to$ 13.06%).
- Closed-Source Models: Similar gains were observed on GPT-4.1 and o3-mini.
Ablation Studies: Removing any of the three retrieval routes degraded performance, confirming the necessity of combining coarse-grained (category), fine-grained (problem-transfer), and semantic signals.
Failure Mode Analysis:
- Human guidance primarily reduced structural reasoning failures (missing global structure).
- Model guidance primarily reduced algebraic manipulation errors.
- SSR successfully mitigated both types of errors by selecting the appropriate strategy source based on context.
Efficiency: SSR reduced total output tokens compared to Direct Solving, indicating it steers models away from unproductive exploration rather than just generating longer traces.
Comparison to Strong Baselines: SSR outperformed or matched computationally expensive baselines like Self-Consistency (SC) and Tree-of-Thoughts (ToT) while using only a single generation pass.

5. Significance and Impact

Redefining Guidance: The paper shifts the paradigm from selecting guidance based on semantic similarity or correctness to selecting based on operational executability. It proves that "correct" strategies are not always "usable" strategies for a specific model.
Human-Model Alignment: It highlights that human and model reasoning are structurally different. Effective AI assistance requires bridging this gap by adapting guidance to the model's specific operational strengths rather than forcing human-style abstraction.
Training Implications: The findings suggest that current training data may over-represent procedural/algebraic patterns while under-representing structural/conceptual strategies. Future training curricula could be designed to improve the executability of abstract strategies.
Practical Utility: SSR offers a low-cost, training-free method to significantly boost the reasoning capabilities of compact models, making high-level mathematical reasoning more accessible and reliable.