💬 NLP

FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking

FinRAG-12B is a production-validated, data-efficient 12B-parameter framework that combines a specialized training pipeline with calibrated refusal mechanisms to deliver highly accurate, grounded, and cost-effective question answering for over 40 financial institutions, outperforming GPT-4.1 in citation quality and query resolution while significantly reducing latency and operational costs.

Original authors: Denys Katerenchuk, Pablo Duboue, Keelan Evanini, David Gondek, Nithin Govindugari, Olivier Allauzen, Joshua Baptiste, David J More, Joshua Schechter

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Denys Katerenchuk, Pablo Duboue, Keelan Evanini, David Gondek, Nithin Govindugari, Olivier Allauzen, Joshua Baptiste, David J More, Joshua Schechter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant for a bank. You need someone who is incredibly smart, knows the bank's rules inside out, and—most importantly—never makes things up. If they don't know the answer, they must say, "I don't know," rather than guessing and giving you bad financial advice that could get the bank in trouble.

This paper introduces FinRAG-12B, a specialized AI assistant designed exactly for this job. Here is how the authors built it and why it works, explained simply.

The Problem: The "Over-Agreeable" Assistant

Standard AI models (like the ones you might chat with online) are great at writing stories or coding, but they have a bad habit in banking: they are too eager to please. If you ask them a question they can't answer from their memory, they often just make up a plausible-sounding answer. In a bank, this is dangerous. If a customer asks about mortgage penalties and the AI guesses wrong, it could lead to lawsuits or lost money.

The Solution: A "Recipe" for a Smart, Honest Assistant

The team created a 12-billion-parameter AI (think of this as a very large, but not gigantic, brain) and trained it using a specific "recipe" to fix these issues. They didn't just feed it a massive pile of data; they curated a high-quality, smaller dataset (only 143 million "tokens," or words) to teach it three specific skills:

1. The "Citation" Skill (Showing Your Work)

Imagine a student taking a test. A normal AI might just write the answer. FinRAG-12B is trained to act like a student who must show their work.

How they did it: They created a training pipeline where the AI had to answer questions using only specific documents provided to it (like a bank's policy manual).
The Result: When the AI answers, it points to the exact sentence in the document that supports its answer. It learned to say, "According to page 4 of the policy, the fee is $50," rather than just guessing. In tests, it was better at this than even the most advanced commercial AI models.

2. The "Honesty" Skill (Knowing When to Stop)

This is the most critical part. The team realized that if an AI is trained only on questions it can answer, it will never learn to say "I don't know."

The Analogy: Imagine a teacher who only asks questions the student knows the answer to. The student will never learn to admit ignorance.
The Fix: The team deliberately fed the AI 22% of questions where the answer wasn't in the provided documents. They taught it that in these cases, the correct answer is "I don't know."
The Result: The AI learned to be "calibrated." It refuses to answer when it's unsure (12% of the time), which is much safer than the base model (which guessed 95% of the time) and less annoying than other models that refuse too often (20% of the time).

3. The "Curriculum" Learning (Schooling the AI)

They didn't dump all the data on the AI at once. They used a "curriculum" approach, like a school system:

Stage 1 (General School): They taught the AI general financial language using open-source data. This gave it a solid foundation.
Stage 2 (Specialized Training): Then, they introduced it to real, private banking conversations and specific rules.
Why it matters: If they mixed these two types of data randomly, the AI got confused and performed poorly. By teaching it in stages, it learned the rules first, then how to apply them to real life.

Real-World Results: The "Live Test"

The authors didn't just test this in a lab; they deployed it in 40+ financial institutions. Here is what happened:

More Questions Solved: The new AI successfully answered 7.1% more customer questions than the old system without needing a human to step in.
Speed and Cost: It is 3 to 5 times faster than using a top-tier commercial AI API. More importantly, it costs 20 to 50 times less to run.
- Analogy: Using the old commercial AI is like taking a private jet for a short trip to the grocery store. FinRAG-12B is like driving a reliable, efficient electric car. It gets you there faster and for a fraction of the price.
Customer Happiness: While the quality of individual answers didn't change drastically, customers were happier overall because more of their questions got answered successfully.

Summary

FinRAG-12B is a specialized banking AI that was trained to be honest, grounded, and efficient. By teaching it to cite its sources and admit when it doesn't know the answer, the authors created a system that is safer for banks, cheaper to run, and actually solves more customer problems than the "off-the-shelf" alternatives.

Note on Limitations: The authors are careful to say this was tested specifically on banking questions (like loans and accounts). They don't claim it works for stock trading or insurance yet, and they couldn't share their private training data due to privacy laws, though they did share their public training methods.

Technical Summary: FinRAG-12B

Problem Statement

The adoption of Large Language Models (LLMs) in the banking sector is hindered by strict requirements for factual accuracy, regulatory compliance, and verifiable grounding. Off-the-shelf models often suffer from hallucinations, over-agreeable behavior (sycophancy), and a lack of alignment with dynamic, domain-specific data such as interest rates and institutional policies. In Retrieval-Augmented Generation (RAG) systems, models may ignore retrieved context, rely on irrelevant passages, or fabricate answers when information is unavailable. This poses significant regulatory and financial risks. Furthermore, existing solutions often fail to balance the trade-off between answering queries and refusing to answer when evidence is insufficient, leading to either unsafe hallucinations or excessive refusal (over-caution).

Methodology

The authors propose FinRAG-12B, a 12B-parameter LLM optimized for banking RAG, built upon the Gemma 3 12B-IT base model. The approach centers on a unified, data-efficient training framework that jointly optimizes answer quality, citation grounding, and calibrated refusal.

1. Data Curation Pipeline

The training corpus consists of 98,648 samples (143M tokens), prioritizing quality over scale. The pipeline involves:

RAG-v1 (Open Source): 43,581 samples filtered via JudgeLM to ensure high-quality supervision.
Synthetic SEC QA: 16,773 samples generated from SEC filings (10-K, 10-Q) using a five-stage pipeline. This includes segmenting documents, generating questions at four difficulty levels, rephrasing to match real user query distributions (style, length, formality), enforcing citation-style outputs, and injecting distractor passages to simulate retrieval noise.
CommonCrawl Financial Subset: 20,499 samples derived from real user queries (non-PII) and relevant passages, filtered via a random forest classifier to ensure privacy compliance.
Refusal Calibration Data: 17,795 proprietary examples (22% of the mixture) where the retrieved context is topically relevant but insufficient. The target output is an explicit "I don't know," training the model to abstain rather than hallucinate.

2. Training Strategy

The model employs a two-stage curriculum learning approach:

Stage 1 (Domain Adaptation): Trained on 60,354 samples (RAG-v1 + Synthetic SEC) with a conservative learning rate ( $1 \times 10^{-6}$ ). This establishes foundational capabilities for grounded generation and citation alignment using reproducible data.
Stage 2 (Task Specialization): Continued training on 38,294 samples (CommonCrawl + Proprietary Banking) with a higher learning rate ( $5 \times 10^{-6}$ ). This stage introduces real-world query distributions and calibrates refusal behavior.
Positional Bias Mitigation: To counteract the tendency of models to attend disproportionately to the beginning or end of context, the authors randomize the placement of relevant passages among distractors using a mixture of discrete right trapezoidal harmonic decay distributions.
Refusal Calibration: A systematic sweep determined that a 22% ratio of unanswerable examples yields the optimal trade-off, minimizing unsupported answers while avoiding excessive refusal on answerable queries.

3. Model Configuration and Serving

Fine-tuning: Utilizes LoRA (Low-Rank Adaptation) on all attention and MLP layers ( $r=64, \alpha=256$ ).
Quantization: The model is quantized to W4A16 (4-bit weights, 16-bit activations) using SmoothQuant, reducing the footprint from 24GB to 8.4GB for efficient single-GPU deployment with minimal performance degradation.

Key Contributions

Data-Efficient Pipeline: A multi-stage pipeline combining LLM-as-a-Judge filtering, style-conditioned synthetic generation, and privacy-compliant real-data extraction to create a high-quality, grounded training dataset with only 143M tokens.
Calibrated Refusal Mechanism: A strategy based on controlled negative sampling (22% unanswerable examples) that significantly reduces hallucinations while preventing the over-refusal common in general-purpose models.
End-to-End Production Framework: A validated methodology spanning data curation, curriculum learning, and quantized serving, deployed across 40+ financial institutions.

Results

Evaluation was conducted on a proprietary banking test set (258 examples) and the public FinanceBench benchmark.

Answer Quality & Grounding: FinRAG-12B achieved a JudgeLM score of 6.21, outperforming GPT-4.1 (5.72) and the base Gemma 3 model (5.70). It also improved citation quality to 73.1 (vs. 70.8 for GPT-4.1), demonstrating more selective and relevant citation grounding.
Refusal Calibration: The model achieved a 12.0% abstention rate on unanswerable queries. This is a substantial improvement over the base model's unsafe 4.3% rate (under-refusal) and GPT-4.1's 20.2% rate (over-refusal).
Generalization: On FinanceBench, FinRAG-12B achieved the highest F1 score (0.284) with a 97.3% citation rate, outperforming both the base model and GPT-4.1.
Ablation Studies: Joint training on all data sources simultaneously collapsed performance (JudgeLM 3.28, 46.5% over-refusal). The two-stage curriculum was critical to resolving the conflict between learning citation conventions and calibrating refusal.
Latency and Cost: FinRAG-12B operates 3–5x faster than GPT-4.1 (0.57s total time to completion vs. 1.83s) and incurs 20–50x lower costs per query when deployed on a single RTX 6000 Ada GPU.
Production Impact: In a live deployment across a large U.S. credit union, FinRAG-12B improved the query resolution rate by 7.1 percentage points (from 77.4% to 84.5%, $p < 0.001$ ). User satisfaction increased from 59.5% to 62.9%, driven primarily by the higher resolution rate rather than marginal improvements in per-response quality.

Significance

The paper claims that FinRAG-12B demonstrates that data quality, grounding, and training methodology are more critical than model scale for building reliable LLMs in regulated domains. By integrating a data-efficient pipeline with calibrated refusal, the system achieves high accuracy and safety without the latency and cost burdens of large commercial APIs. The work highlights that in production RAG systems, increasing the resolution rate is the primary driver of user satisfaction, and that a unified framework can successfully navigate the trade-offs between answering, citing, and abstaining in high-stakes financial environments.