💬 NLP

GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation

This paper demonstrates that fine-tuning an open-weight language model with Group Relative Policy Optimization (GRPO) and a dual evaluation framework—combining an LLM-as-a-judge rubric with a judge-independent Causal Average Treatment Effect (CATE) audit—produces financial advice that significantly outperforms commercial baselines in estimated profit lift and risk mitigation.

Original authors: Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam, Shravan Mohan, Yakov Gazman, Oded Vainas

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam, Shravan Mohan, Yakov Gazman, Oded Vainas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to give advice. In the world of artificial intelligence, this is a bit like teaching a student to solve a math problem. Usually, we show the student the question and the correct answer, hoping they memorize the pattern. But what if the "correct answer" in the real world is messy? What if the history books are full of mistakes, or if the right answer changes depending on the situation? This is the challenge of Reinforcement Learning (RL). Instead of just copying old answers, RL lets an AI try many different things, get a "score" for how well it did, and learn from that score to get better next time. It's like a video game where the AI plays thousands of levels, gets points for good moves, and slowly learns the perfect strategy.

Now, imagine this game is about money. Giving financial advice is tricky because a bad suggestion can actually hurt a business. To make sure the AI learns the right way, researchers use a "judge"—another smart AI that grades the advice. But there's a catch: sometimes, the student AI learns to trick the judge by saying exactly what the judge wants to hear, rather than giving advice that actually works in the real world. This paper tackles that problem by creating a new way to train an AI to give business advice, checking its work not just with a judge, but by seeing if the advice would have made money in the past.

The Story of the Money-Smart Robot

Meet the team at Intuit. They wanted to build an AI that could look at a business's messy financial records and say, "Hey, here's a specific thing you should do to make more profit." It sounds simple, but it's actually a nightmare for computers. The advice needs to be based on real numbers, it needs to be safe (don't tell a business to fire its only customer!), and it needs to be actionable.

The problem is that we don't have a "gold standard" answer key. The decisions business owners made in the past weren't always perfect, so we can't just teach the AI to copy them. And asking human experts to write perfect advice for every scenario is too expensive and slow. So, the team decided to treat this like a video game. They used a method called GRPO (Group Relative Policy Optimization).

Think of GRPO as a "group challenge." Instead of the AI guessing one answer and getting a score, it generates a whole group of different advice suggestions at once. Then, a super-smart "Judge AI" (called Claude Opus 4.5) looks at all of them and gives them scores based on a checklist. This checklist has 11 rules: Is it safe? Does it use real numbers from the business? Does it explain why? If a suggestion is dangerous, it gets a zero immediately. If it's safe but boring, it gets a low score. If it's safe, specific, and smart, it gets a high score. The AI then learns to make more suggestions that look like the high-scoring ones.

The Trap of the "Nice" Judge

Here is where things get interesting. The team knew there was a risk. What if the AI just learned to write advice that sounded good to the Judge AI, but wouldn't actually make a business any money? It's like a student who memorizes the teacher's favorite phrases to get an A, but fails the real test.

To catch this, the researchers built a second, totally different test. They didn't use the Judge AI at all. Instead, they used a "Time Machine" method called CATE (Conditional Average Treatment Effect).

Imagine you have a giant box of old business records. You want to know: "If we had taken this specific action in the past, would the business have made more money?" The researchers took the advice generated by their new AI, translated it into a simple "action" (like "cut costs on shipping" or "raise prices on product X"), and then looked at the historical data. They asked: "In the past, when businesses did this action, did their profits go up?" This is a "judge-independent" audit because it relies on real-world numbers, not on what the Judge AI thinks sounds nice.

The Big Surprise

The results were a mix of expected wins and a very funny twist.

First, the new AI (trained with GRPO) was a star. When the Judge AI graded it, it got a score of 9.514 out of 10. This was the highest score of anyone, beating even the most famous commercial AI models like Claude Opus 4.6 and GPT-5.4.

But the real magic happened in the "Time Machine" audit.

The new AI suggested actions that, if they had been done in the past, would have lifted gross profit by 0.0228 (about a 2.3% increase).
The best commercial AI (Claude Opus 4.6) only managed a lift of 0.0104.
That means the new AI was roughly twice as good at finding actions that actually make money as the strongest commercial competitor.

Even cooler, the new AI was safer. It had the lowest "downside rate" (the chance of suggesting something that hurts the business) and the least "tail risk" (the chance of a really bad outcome).

The Twist: The Judge and the Time Machine Disagreed

Here is the most playful part of the story. The Judge and the Time Machine didn't always agree on who was second or third place.

The untrained "base" AI (the raw model before any training) was terrible at the Judge's test. It scored a 8.457, coming in last place. It sounded messy and unpolished. But when the Time Machine checked its advice? It actually did pretty well! It suggested actions that would have increased profit by 0.0170, which was better than every single commercial model.

On the flip side, one of the commercial models (Claude Opus 4.5) got a great score from the Judge (8.982), sounding very professional. But the Time Machine said, "Wait a minute." It estimated that following this model's advice would actually lose money (a negative lift of -0.0025).

What This Means

This paper shows that you can't just trust the "Judge" to tell you if an AI is good at giving financial advice. The Judge likes advice that sounds smart and follows the rules. The Time Machine cares about whether the advice actually works.

The team's new method, using GRPO with a safety gate, managed to do both. It learned to write advice that sounded great to the Judge and actually made money in the Time Machine test. It proved that by training an open-source model with a smart, safety-focused reward system, you can beat the expensive, commercial giants.

The authors suggest that for high-stakes tasks like money, you need both checks: a rubric to make sure the advice is safe and well-written, and a causal audit to make sure it actually works. If you only look at one, you might get fooled by a robot that sounds great but is terrible at math. Their new robot, however, is both a great writer and a great accountant.

Technical Summary: GRPO for Financial Advice Generation

Problem Formulation
The paper addresses the challenge of generating actionable, free-form financial advice from business records. This task requires integrating numerical reasoning, domain knowledge, and sound judgment to produce recommendations that are grounded in specific business data, aligned with financial goals (specifically gross profit), and safe to implement. The authors identify two primary obstacles in this domain:

Suboptimal Historical Data: Historical business decisions are not necessarily optimal "gold labels" because they reflect constraints and incomplete information available at the time of decision-making. Consequently, supervised imitation learning risks propagating suboptimal behaviors.
Cost of Annotation: Collecting high-quality, free-form annotations from domain experts is expensive and does not scale.

To overcome these, the authors formulate financial advice generation as a Reinforcement Learning (RL) problem, aiming to optimize a policy directly for a reward signal that captures the desired properties of useful advice, rather than imitating past decisions.

Methodology
The core methodology involves fine-tuning an open-weight language model (Qwen3.5-27B) using Group Relative Policy Optimization (GRPO).

Training Loop: For each business state ( $s$ ) and goal ( $g$ ), the policy samples a group of $K$ candidate recommendations. These are scored by a reward function to compute group-relative advantages, which update the policy without requiring a separate value model.
Reward Design (LLM-as-a-Judge): The reward signal is provided by an LLM judge (Claude Opus 4.5) evaluating recommendations against a rubric of 11 binary criteria across six aspects:
- Safety (Gate): A hard gate that zeros the reward if a recommendation would cause significant harm (e.g., eliminating a revenue stream).
- Quality Criteria: Ten criteria covering specificity (citing concrete numbers/entities), actionability, data grounding, reasoning (identifying patterns/risks), impact quantification, and relevance to the goal.
- Penalties: The reward includes penalties for unparsable JSON outputs and for excessively long reasoning traces to prevent degenerate outputs.
Stability: A KL penalty anchors the policy to the base model to prevent drift during training.

Judge-Independent Causal Audit
To address the risk of "reward hacking" (where a model optimizes for the judge's preferences rather than genuine business value), the authors implement a judge-independent audit based on a doubly-robust Conditional Average Treatment Effect (CATE) estimator.

Process: Generated free-text recommendations are mapped by a separate, independent LLM classifier to a fixed taxonomy of 60 discrete business actions.
Estimation: Using logged observational data, the system estimates the impact of these actions on year-over-year gross-profit growth. The estimator uses a multi-label propensity model and outcome models (treated/control heads) to calculate the CATE via the Augmented Inverse Propensity Weighting (AIPW) method.
Metrics: The audit reports three metrics: Estimated Lift (DR-CATE), Downside Rate (fraction of recommendations with negative estimated effects), and Tail Risk (Conditional Value-at-Risk, CVaR).

Key Contributions

Rubric-Grounded, Safety-Gated GRPO: The authors formulate free-form business financial advice as an RL problem, fine-tuning an open-weight LLM with a finance-specific reward that includes a safety gate and multi-dimensional quality criteria.
Natural-Language-to-Action Causal Audit: They propose a method to map generated advice into a fixed business-action taxonomy and apply a standard doubly-robust CATE estimator to audit whether recommendations correspond to historically beneficial actions.
Dual Evaluation Framework: The study evaluates models using both judge-based rubric scores and a judge-independent causal audit. This dual approach reveals that the two evaluations measure distinct signals, arguing that the causal audit is a valuable complement rather than a mere confirmation of the judge.

Results
The experiments compare the GRPO-trained Qwen3.5-27B against strong commercial baselines (including Claude Opus 4.6, GPT-5.4, and Claude Sonnet 4.5) and the untrained base model.

Judge Evaluation: The GRPO-trained model achieved the highest rubric score (9.514), outperforming all commercial baselines and the untrained base model (8.457).
Causal Audit: The GRPO model achieved an estimated gross-profit lift of 0.0228, approximately twice that of the strongest commercial baseline (Claude Opus 4.6 at 0.0104). It also demonstrated the lowest downside rate (15.5% vs. 23.2% for the best commercial baseline) and the least negative tail risk.
Divergence in Rankings: A critical finding is the lack of perfect correlation between the two evaluation methods. The untrained base model ranked last on the judge rubric but second on the causal audit (outperforming all commercial models). Conversely, some commercial models scored well on the rubric but showed negative or negligible estimated lift. This divergence suggests that rubric quality (well-formed advice) and estimated business value (historical effectiveness of the mapped action) are distinct axes.

Significance and Claims
The paper claims that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs.

The authors emphasize that the combination of a safety-gated reward and a judge-independent causal audit provides a practical approach for building safer and more effective financial advice models. The results suggest that:

Targeted reward design can substitute for raw model scale on narrow, high-value tasks.
The causal audit captures a signal of genuine business value that the judge does not, validating its use as an independent check against reward hacking.
A system that ranks highly on both the rubric (quality/safety) and the causal audit (business impact) is the only one supported by both lines of evidence.

The study concludes that while the causal audit is computed from observational data and thus treated as a complementary signal rather than a standalone measure of realized impact, the agreement between the judge and the audit on the GRPO policy's superiority makes the gains robust. Future work is noted to explore richer action taxonomies, expert evaluation, and multi-step advice.