💬 NLP

Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection

This paper introduces FinFRE-RAG, a two-stage framework that combines importance-guided feature reduction with retrieval-augmented in-context learning to enable Large Language Models to effectively detect financial fraud by generating interpretable explanations while narrowing the performance gap with traditional tabular models.

Original authors: Xuwei Tan, Yao Ma, Xueru Zhang

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuwei Tan, Yao Ma, Xueru Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Super-Genius" vs. The "Spreadsheet"

Imagine you have a Super-Genius Detective (this is the Large Language Model, or LLM). This detective is amazing at reading novels, understanding jokes, and solving mysteries based on stories. They can explain why a suspect is guilty in beautiful, clear sentences.

However, financial fraud detection usually happens in a massive spreadsheet (tabular data). This spreadsheet has thousands of rows and columns of boring numbers: Transaction Amount: $12.50, Card ID: 4492, Time: 14:02, Location: Ohio.

The Conflict:
If you hand this spreadsheet directly to the Super-Genius Detective, they get confused.

Too much noise: The spreadsheet has hundreds of columns. It's like trying to read a book where every 10th word is a random number. The detective gets overwhelmed.
No story: The spreadsheet doesn't tell a story. It just lists facts. The detective doesn't know which numbers matter.
The "Needle in a Haystack" problem: Fraud is rare (maybe 1 in 1,000 transactions). If the detective just guesses, they might say "everything is safe" and miss the one bad guy, or say "everything is bad" and annoy everyone.

In the past, companies used Robot Accountants (traditional AI models like XGBoost) to read these spreadsheets. They are great at math but terrible at explaining why they flagged a transaction. They just say "Flagged" without a reason.

The Solution: FinFRE-RAG (The "Smart Assistant" System)

The authors created a new system called FinFRE-RAG to help the Super-Genius Detective do their job. Think of it as a two-step process that acts like a Personal Research Assistant for the detective.

Step 1: The "Highlighter" (Feature Reduction)

Before the detective looks at the spreadsheet, a Highlighter Bot scans the thousands of columns and says: "Hey, Detective, you don't need to read all 500 columns. Just look at these top 10 that actually matter for fraud, like 'Amount' and 'Location'."

Why? This stops the detective from getting overwhelmed by irrelevant data. It turns a 500-page book into a 10-page summary.

Step 2: The "Case File" (Retrieval-Augmented Generation)

Now, the detective needs to decide if the current transaction is fraud. Instead of guessing, the system pulls up a Case File of similar past transactions.

It finds 20 past cases that look very similar to the current one.
It tells the detective: "Here is Case A: It looked like this, and it was Fraud. Here is Case B: It looked like this, and it was Safe."

The detective then reads the current transaction, compares it to these past cases, and says: "This looks a lot like Case A. The amount is weird, and the location matches the fraud pattern. I'm 80% sure this is fraud."

Why This is a Game-Changer

The paper tested this system on four different datasets and found some cool things:

From "Random Guessing" to "Expert Detective":
Without the system, the LLM was basically guessing (like a coin flip). With the system (FinFRE-RAG), the LLM became almost as good as the best Robot Accountants, but with a superpower: It can talk.
The "Why" Matters:
If a Robot Accountant flags a transaction, you just see a red light. If the LLM flags it, it says: "I flagged this because the transaction amount is small, but the location is in a high-risk country, and the card was used 5 times in an hour, which matches our past fraud cases."
- Analogy: It's the difference between a security guard yelling "STOP!" and a guard explaining, "Stop! You're wearing a mask and running in a bank lobby."
Less is More:
The study found that giving the LLM fewer features (just the important ones) actually made it smarter. Giving it more data just confused it. It's like giving a chef the best 5 ingredients instead of a whole grocery store; they make a better dish.
No Training Needed:
Usually, to make an AI smart at a new job, you have to "train" it for weeks (teaching it from scratch). This system is like hiring a genius who already knows everything but just needs a briefing (the case files) to do the specific job. You don't need to retrain the model; you just give it the right context.

The Catch (Limitations)

The authors are honest about the flaws:

It's not perfect yet: The best Robot Accountants are still slightly better at pure math than this new system.
It needs good data: If the "Case Files" (retrieved examples) are bad or biased, the detective will make bad decisions.
It's not a magic wand: It's a tool to help human analysts, not replace them entirely. Humans still need to check the work.

The Bottom Line

This paper shows that we can teach Super-Genius AIs to understand boring financial spreadsheets by:

Summarizing the data (highlighting the important bits).
Showing them examples of similar past crimes (retrieval).

This turns the AI from a confused guesser into a reasoning expert that can not only catch fraud but also explain why it caught it, making the whole process faster, cheaper, and much more transparent for humans.

1. Problem Statement

Financial fraud detection traditionally relies on tabular machine learning models (e.g., XGBoost, LightGBM) that require extensive feature engineering and offer limited interpretability. While Large Language Models (LLMs) excel at reasoning and generating human-readable explanations, they struggle when applied directly to structured financial transaction data due to two primary misalignments:

Tabular Input Misalignment: Financial data consists of high-dimensional numeric and categorical features. LLMs, pretrained on natural language, often fail to interpret the semantic meaning of raw tabular values or handle the precision required for numerical reasoning. Furthermore, including hundreds of features in a prompt exceeds context limits and introduces noise.
Fraud Ambiguity and Rarity: Fraud definitions vary by context, and fraudulent instances are extremely rare (often <1% of data). Without domain-specific guidance, LLMs treat tabular inputs as arbitrary numbers, leading to predictions indistinguishable from random guessing.

2. Methodology: FinFRE-RAG

The authors propose FinFRE-RAG (FINancial Feature-REduced in-context learning with Retrieval-Augmented Generation), a two-stage framework designed to adapt LLMs to structured fraud detection without task-specific fine-tuning.

Stage 1: Offline Feature Reduction

To address high dimensionality and noise, the framework reduces the feature space before inference:

Importance-Guided Selection: A Random Forest model is trained on the external historical dataset to rank features by importance.
Top-k Selection: Only the top $k$ most impactful features are retained to form a compact subset ( $F_{sel}$ ).
Normalization: Numeric features in $F_{sel}$ are standardized (z-score normalization) using global statistics from the historical dataset to create a vectorized representation for efficient retrieval.

Stage 2: Online Retrieval-Augmented In-Context Learning

At inference time, the framework constructs a prompt for the LLM using a hybrid retrieval strategy:

Instance Retrieval:
- Categorical Filtering: The system filters the historical dataset by matching categorical attributes (e.g., state, card type) in descending order of importance until a non-empty candidate pool is found.
- Numeric Similarity Search: Within the candidate pool, the system retrieves the top- $n$ most similar transactions based on cosine similarity of the normalized numeric features.
Prompt Construction:
- The retrieved examples are converted into natural language sentences (e.g., "The client state is 5, credit balance is 10540... It is a fraud.").
- The current query transaction is serialized similarly but without a label.
- Task Instruction: The LLM is instructed to act as a financial expert, analyze the query against the few-shot examples, and output a 5-point risk score (1–5) along with a natural language explanation. A score $\ge 4$ is classified as fraud.

3. Key Contributions

Framework Design: Introduction of FinFRE-RAG, a novel approach that bridges the gap between LLM reasoning capabilities and structured tabular data via feature reduction and retrieval-augmented in-context learning (ICL).
Systematic Evaluation: Comprehensive benchmarking across four public fraud datasets (CCF, CCFRAUD, IEEE-CIS, PAYSIM) and six open-weight LLMs (Qwen, Gemma, GPT-OSS families).
Empirical Insights:
- Demonstrated that LLMs can match or exceed traditional baselines (Random Forest, XGBoost) in specific settings when guided by FinFRE-RAG.
- Quantified the optimal number of features ( $k$ ) and retrieved examples ( $n$ ) required for effective reasoning.
- Showed that risk scoring (5-point scale) outperforms direct binary classification, allowing models to better handle uncertainty.
- Proved that importance-guided feature reduction is critical; random feature selection significantly degrades performance.

4. Experimental Results

Performance Gains: Direct prompting of LLMs resulted in near-zero or negative MCC (Matthews Correlation Coefficient) scores, indicating random guessing. FinFRE-RAG improved F1 scores dramatically (e.g., Qwen3-14B on CCF improved from 0.00 to 0.31; Gemma 3-12B on CCF improved from 0.00 to 0.79).
Comparison with Baselines: FinFRE-RAG enabled LLMs to achieve performance competitive with strong tabular baselines like XGBoost and TabM in several datasets, particularly on IEEE-CIS and CCF.
Fine-Tuning vs. ICL: The study compared FinFRE-RAG against LoRA fine-tuned LLMs. Surprisingly, FinFRE-RAG (ICL) often outperformed fine-tuned models, particularly in recall. Fine-tuned models tended to overfit to the majority class or narrow binary heads, whereas ICL allowed models to leverage heterogeneous historical examples for more robust reasoning.
Model Behavior: Reasoning models (e.g., Qwen3, GPT-OSS) benefited from larger retrieval sets ( $n$ ), while non-reasoning models (Gemma 3) relied heavily on the label distribution of retrieved neighbors, sometimes leading to shortcuts in synthetic datasets.

5. Significance and Implications

Interpretability: Unlike "black box" tabular models, FinFRE-RAG provides human-readable rationales explaining why a transaction is flagged, aligning with regulatory requirements for explainable AI.
Practical Deployment: The framework operates without the computational cost of fine-tuning large models, making it feasible for on-premise deployment with open-weight models to ensure data privacy.
Paradigm Shift: The paper suggests that for tabular fraud detection, instance-based reasoning (comparing a query to similar historical cases) is a more effective strategy for LLMs than attempting to learn global decision boundaries via parameter updates.
Future Direction: Highlights the potential of LLMs as assistive tools for fraud analysts, narrowing the performance gap with specialized classifiers while offering transparency.

Limitations Noted

The authors acknowledge limitations including reliance on public datasets (which may lack the complexity of real-world production data), the lack of latency/throughput optimization for production, and the fact that generated explanations are heuristic rather than causal. They also note that the method has not yet been tested on domains beyond financial fraud.