🤖 machine learning

Error-Driven Prompt Optimization for Arithmetic Reasoning

This paper introduces an error-driven prompt optimization framework that significantly enhances the arithmetic reasoning capabilities of on-premises small language models, enabling them to outperform larger models like GPT-3.5 Turbo in privacy-compliant industrial settings without requiring costly fine-tuning.

Original authors: Árpád Pándy, Róbert Lakatos, András Hajdu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Árpád Pándy, Róbert Lakatos, András Hajdu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Local Accountant vs. The Cloud Giant

Imagine you have a very smart, but slightly forgetful, local accountant (a Small Language Model or SLM) who works in your secure office. You also have a super-genius consultant (a Large Language Model like GPT-3.5) who lives in a giant cloud office.

The problem? Your company deals with sensitive financial data. You cannot send this data to the cloud consultant because of privacy rules. You must keep everything in your local office.

However, your local accountant is terrible at math. If you ask, "What is the percentage change in revenue?" they might guess the number or get the units wrong (saying "million" instead of "percent"). They are fast and private, but not accurate enough for serious work.

This paper introduces a clever training method to turn that forgetful local accountant into a math wizard, without sending their data to the cloud or paying for expensive retraining.

The Strategy: "Code, Don't Guess"

The researchers realized that asking the accountant to just "think" about the math was the problem. Instead, they taught the accountant to act like a programmer.

The Old Way: "Here is a table of numbers. Tell me the answer." (The accountant guesses).
The New Way: "Here is a table. Write a short computer script (Python code) that does the math for you, then run it."

This is like giving the accountant a calculator instead of asking them to do long division in their head. The computer code is perfect at math; the accountant just has to write the instructions correctly.

The Secret Sauce: Learning from Mistakes (The "Error-Driven" Loop)

Even with the calculator, the accountant still made mistakes. Sometimes they wrote the wrong formula, or they got the "scale" wrong (saying the answer is in "thousands" when it should be "percent").

The paper's main innovation is a systematic "Mistake Detective" process:

Run the Test: The accountant tries to solve 497 financial questions.
Catch the Errors: The system looks at the questions the accountant got wrong.
Group the Clues: Instead of looking at every mistake individually, the system groups similar mistakes together.
- Analogy: Imagine a teacher grading a test. Instead of saying "You got question 5 wrong," they notice, "Oh, every student who got question 5 wrong also got question 12 wrong because they all forgot to divide by 100."
Write a Rule: For each group of mistakes, the researchers write a specific rule to fix it.
- Example Rule: "If the question asks for 'percentage change,' the answer must be in 'percent,' not 'dollars'."
Try Again: They add this rule to the accountant's instructions and run the test again.
Repeat: They keep finding the biggest group of remaining mistakes, writing a new rule, and testing until the mistakes stop improving.

The Results: Beating the Big Guy

By following this "find the mistake, write a rule, repeat" cycle, the researchers achieved something surprising:

The Starting Point: The local accountant (Qwen3 4B) started with an accuracy of about 30% (barely better than guessing).
The Improvement: After adding just three specific rules derived from their mistake analysis, the accuracy jumped to 70.8%.
The Victory: This local, private accountant outperformed the giant cloud consultant (GPT-3.5 Turbo), which only scored 66.2%.

The "Too Many Rules" Trap

The paper also discovered an important limit. Imagine you give your accountant a rulebook.

Too few rules: They make simple mistakes.
Just the right number of rules: They are perfect.
Too many rules: The accountant gets confused. The rulebook becomes so thick and complex that they start ignoring the rules or mixing them up.

The researchers found an "optimal number" of rules. Adding more rules after this point actually made the performance worse. It's like trying to memorize a 50-page instruction manual for a simple task; you just get overwhelmed.

Summary of Claims

Privacy First: You can build a highly accurate AI for sensitive data (finance, health) that stays entirely on your own computer (on-premises).
No Expensive Training: You don't need to spend millions retraining the model. You just need to analyze its errors and write simple text rules to fix them.
Small Can Beat Big: A small, efficient model, when guided by the right "error-driven" rules, can do a better job at arithmetic reasoning than a massive, expensive model.
The Method: The key is Code Generation (making the AI write code to do the math) combined with Iterative Prompt Optimization (systematically fixing the AI's specific types of mistakes).

In short, the paper shows that you don't need a bigger brain to solve math problems; you just need a better set of instructions that help the brain avoid its specific blind spots.

Technical Summary: Error-Driven Prompt Optimization for Arithmetic Reasoning

Problem Statement

While Large Language Models (LLMs) excel in natural language understanding, their ability to perform accurate arithmetic reasoning on structured tabular data remains fragile, often producing inconsistent or incorrect numerical outputs. This limitation is particularly critical for regulated sectors like finance and healthcare, where data privacy mandates that sensitive information remain within secure, on-premises environments, precluding the use of API-based cloud models. Although Small Language Models (SLMs) offer a privacy-compliant alternative, they frequently exhibit fundamental limitations in arithmetic tasks, often underperforming even larger models when deployed locally without specialized optimization.

Methodology

The paper proposes a systematic, error-driven optimization framework designed to enhance a Code Generation Agent (CGA) running on on-premises SLMs. The methodology integrates three core components:

Task Decomposition and Code Generation: Instead of direct answer generation, the system reframes the task as code generation. The SLM generates executable Python code based on a restructured table (converted to an annotated value list) and the user query. A deterministic executor then runs this code to produce the final result, separating semantic interpretation from arithmetic computation.
Error-Driven Rule Induction: The core innovation is an iterative loop that identifies, clusters, and remediates prediction errors without costly model fine-tuning.
- Feature Extraction: Failed predictions are analyzed to extract features including calculation patterns, code calculation patterns, scale mismatches, value matches, and specific error types (e.g., selection, calculation, scale, sign, syntax, runtime).
- Clustering: The HDBSCAN algorithm clusters these erroneous predictions based on Hamming distance to identify groups sharing a common root cause.
- Rule Formulation: For the largest error clusters, domain experts formulate specific Domain-Specific Prompt Rules (DSPR) to address the identified root cause (e.g., clarifying that "percentage change" implies a "percent" scale).
- Validation: New prompt rules are validated using the McNemar test on the specific error cluster. A rule is accepted only if it yields a statistically significant improvement ( $\Delta EM \geq 50\%$ and $p \leq 0.0625$ ) and improves global performance.
Optimal Rule Set ( $K_{opt}$ ): The framework seeks an optimal number of rules. The authors posit that while initial rules fix common errors, adding too many rules ( $K > K_{opt}$ ) leads to cognitive overload and rule conflicts, degrading performance.

Key Contributions

Error-Driven Optimization Framework: A novel algorithm (Algorithm 1) that automates the identification of error root causes and the iterative refinement of prompt rules, enabling SLMs to "learn from mistakes" without parameter updates.
Performance Surpassing Larger Models: The study demonstrates that a 4B-parameter SLM (Qwen3 4B), when equipped with this error-driven prompt optimization and the CGA framework, achieves higher accuracy than the significantly larger GPT-3.5 Turbo in a privacy-preserving, on-premises setting.
Theoretical Insight into Rule Saturation: The paper formalizes the concept of an optimal rule set ( $K_{opt}$ ), arguing that SLM optimization requires a balance between informativeness and cognitive load, distinct from simple rule accumulation strategies used for larger models.

Results

The framework was evaluated on the TAT-QA dataset, specifically filtering for arithmetic calculation questions.

Baseline Performance: The base Qwen3 4B model achieved an Exact Match (EM) score of 59.96% with a simplified prompt.
Iterative Improvement: Through three accepted iterations of rule addition (addressing "percentage change" scales, "change in percentage" as subtraction, and "year average" calculations), the model's performance improved incrementally.
Final Performance: The optimized Qwen3 4B agent achieved a final Exact Match (EM) accuracy of 70.82%.
Comparison: This result surpasses the GPT-3.5 Turbo baseline (66.27%) under the same Code Generation Agent and table restructuring conditions. However, the paper notes that while the SLM matched the larger model in overall EM, its raw numerical value accuracy (excluding scale) remained slightly lower (77.06% vs. 78.92%), suggesting that rule tuning effectively guides format but may not fully close the gap in core arithmetic robustness.

Significance and Claims

The paper claims that the perceived arithmetic limitations of locally deployed SLMs are not fundamental but are artifacts of inference strategy. By combining task decomposition (code generation) with systematic, error-driven prompt optimization, it is possible to develop reliable, interpretable, and industrially deployable AI assistants that operate entirely on-premises.

The authors emphasize that this approach offers a viable alternative to costly fine-tuning or reliance on external APIs for sensitive data workflows. The study concludes that developing trustworthy tabular reasoning AI is achievable through a "blueprint" of systematic error analysis and concise rule induction, enabling small models to outperform larger counterparts in specific, privacy-constrained domains. The work highlights that SLM optimization is a distinct discipline requiring careful management of rule complexity to avoid cognitive overload.