🤖 machine learning

K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance

This paper introduces K-FinHallu, the first benchmark designed to detect hallucinations in multi-turn Retrieval-Augmented Generation systems within the Korean financial domain, revealing that even advanced large language models struggle with fine-grained diagnostics and justified abstention.

Original authors: Eunbyeol Cho, Yunseung Lee, Mirae Kim, Jeewon Yang, Youngjun Kwak, Edward Choi

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eunbyeol Cho, Yunseung Lee, Mirae Kim, Jeewon Yang, Youngjun Kwak, Edward Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A "Lie Detector" for Money Talk

Imagine you hire a very smart, well-read assistant to answer your questions about your bank account, loans, or taxes. You give them a stack of official bank documents to read first. Usually, they do a great job. But sometimes, they get confident and make things up, mix up numbers, or ignore the documents entirely. In the world of finance, a single made-up number can cost someone their life savings.

This paper introduces K-FinHallu, which is essentially a training gym and a test exam designed to teach computers how to spot when their "money assistant" is lying or making mistakes. It is specifically built for Korean financial conversations, which have their own unique rules and language quirks that other tests miss.

The Problem: Why Current Tests Fail

The authors say that existing tests for AI "hallucinations" (making things up) are like driving tests done only on empty, straight highways in English-speaking countries. They don't prepare the AI for:

Long Conversations: Real banking isn't just one question and one answer. It's a chat where you say, "How much is my loan?" and then later, "Can I pay that off early?" The AI has to remember what "that" refers to. Current tests mostly look at single, isolated questions.
Korean Money Rules: Korea has unique financial systems (like a specific type of rental deposit called Jeonse) and complex legal language. Translating an American test doesn't work because the rules and words are totally different.

The Solution: Building a "Fake Bank Chat"

The team built a new benchmark called K-FinHallu. Here is how they made it:

The Source Material: They took real, official Korean financial documents (like laws from the Financial Supervisory Service).
The "Good" Chat: They used AI to simulate a perfect conversation between a customer and a bank agent, where the agent answers every question correctly based only on the documents.
The "Bad" Injection (The Trick): This is the clever part. They systematically broke the perfect answers to create "hallucinations." They didn't just delete words; they made subtle, dangerous errors, such as:
- The "Wrong Number" Trick: Changing a loan interest rate from 3% to 4%.
- The "Wrong Word" Trick: Swapping a "secured loan" (backed by a house) with an "unsecured loan" (no collateral).
- The "Ghost Answer" Trick: Answering a question when the documents actually said, "We don't have enough info to answer this."
- The "Refusal" Trick: Saying "I can't answer that" when the documents clearly had the answer.

They organized these mistakes into a taxonomy (a family tree of errors) to see exactly how the AI failed.

The Experiment: Who is the Best Detective?

The researchers took the world's most advanced AI models (like GPT-5, Gemini, and Llama) and asked them to play the role of the Quality Control Inspector. Their job was to look at a chat turn and say, "Is this answer a lie?" or "Is this answer honest?"

The Results:

The Big Models Struggle: Even the most powerful, expensive AI models had a hard time. They were good at spotting obvious lies but terrible at spotting subtle financial errors or knowing when to say, "I don't know."
The "Refusal" Problem: The hardest part for all models was Justified Abstention. This is the ability to say, "I cannot answer this because the documents don't say." Most models tried to guess anyway, which is dangerous in finance.
The Small Model Surprise: The researchers took a smaller, open-source model (Qwen3-8B) and gave it a "cheat sheet" (a training set with explanations of why an answer was right or wrong). After this training, this small model became better than the giant, expensive models at spotting these specific financial lies.

The Key Takeaways

Context is King: In finance, you can't just look at one sentence. You have to look at the whole conversation history and the specific documents provided.
Subtlety Matters: The most dangerous hallucinations aren't wild inventions; they are tiny changes to numbers or legal terms that sound correct but are wrong.
Knowing What You Don't Know: The most important skill for a financial AI isn't just answering correctly; it's knowing when to stay silent because the evidence isn't there. Currently, most AIs are too eager to talk.

What They Did NOT Claim

They did not say this system is currently deployed in real banks to stop fraud.
They did not claim this solves all AI problems in healthcare or law (only finance).
They did not say the small model is perfect; they just said it performed surprisingly well compared to giants.

In short, K-FinHallu is a specialized "driving test" for AI in the Korean financial sector, revealing that even the smartest cars (AI models) need specific training to navigate the tricky, narrow roads of financial regulations without crashing.

Technical Summary: K-FinHallu

Problem Statement

While Large Language Models (LLMs) have advanced financial automation through Retrieval-Augmented Generation (RAG), hallucinations remain a critical barrier to deployment in high-stakes financial environments. Existing benchmarks for hallucination detection suffer from two primary gaps when applied to real-world financial services:

Single-Turn Limitation: Most benchmarks focus on single-turn tasks, failing to capture the complexities of multi-turn dialogues where hallucinations arise from context-tracking failures, anaphora resolution errors, or logical inconsistencies with prior turns.
Linguistic and Regulatory Gaps: Existing datasets are predominantly English-centric and Western-institutional, neglecting the unique nuances of the Korean financial domain. This includes specific regulatory frameworks (e.g., the Jeonse lease system), agglutinative morphology, and polysemous Sino-Korean technical terminology that cannot be addressed through simple translation.

Consequently, there is a lack of rigorous evaluation frameworks to detect domain-specific hallucinations in multi-turn Korean financial RAG systems, particularly regarding "justified abstention" (knowing when not to answer).

Methodology

The authors introduce K-FinHallu, the first benchmark for hallucination detection in multi-turn Korean financial RAG, constructed from authentic Korean financial documents.

Dataset Construction

Source Corpus: The dataset is built upon the "Korean Financial and Legal Document Machine Reading Comprehension" dataset from AI-Hub, filtered for financial categories. To prevent data leakage, the test set is drawn from the Financial Supervisory Service (FSS) and Korea Consumer Agency, while the training set uses disjoint institutions.
Faithful Dialogue Generation: A pipeline using GPT-4o simulates multi-turn dialogues (4–8 turns) between a user and an assistant. The assistant generates responses strictly grounded in gold passages and clue text from the source documents. The pipeline includes "reconfirm" turns to test context tracking.
Retrieval Simulation: To isolate hallucination detection from retrieval errors, the system simulates RAG by providing controlled retrieval contexts ( $P_n$ $P_{n}$ ).
- Answerable Scenarios: $P_n$ contains the gold passage.
- Unanswerable Scenarios: $P_n$ consists entirely of "hard negatives" (semantically similar passages lacking the answer).
Hallucination Injection: An automated pipeline injects subtle hallucinations into faithful dialogues based on a proposed hierarchical taxonomy. Strategies include minimal edits (e.g., swapping financial terms, altering numbers/modifiers) to preserve naturalness while introducing critical errors. Two injection sources (GPT-4o and Gemini-2.5-Flash) are used to mitigate model-specific artifacts.
Quality Control: Dialogues and injected hallucinations undergo rigorous filtering via LLM-as-a-judge and human annotation, ensuring high agreement (Gwet's AC1 > 0.70) on criteria like answerability, plausibility, and naturalness.

Hallucination Taxonomy

The paper proposes a taxonomy based on context answerability:

Answerable Scenarios:
- Contradictory: Response conflicts with the context or dialogue history.
- Unverifiable: Response includes external claims not supported by the context.
- Irrelevance: Response is logically disconnected from the user's intent.
- False Refusal: Model refuses to answer despite sufficient evidence.
Unanswerable Scenarios:
- False Acceptance: Model provides an answer despite lacking evidence.
- True Refusal: Model correctly abstains (the desired behavior).

Key Contributions

K-FinHallu Benchmark: The release of the first multi-turn hallucination detection benchmark for Korean financial RAG, built on authentic documents and covering diverse financial domains (e.g., Financial Economics, Insurance).
Hierarchical Taxonomy: A novel taxonomy that explicitly accounts for justified abstention in multi-turn dialogues, distinguishing between "False Refusal" and "True Refusal."
Scalable Construction Pipeline: Demonstration that the injection pipeline can automatically generate a training split, enabling the fine-tuning of smaller models (8B) to match or exceed frontier LLMs.

Experimental Results

The authors benchmarked frontier and open-source LLMs (including GPT-5, Gemini-2.5, Llama-3, Qwen3, and Korean-centric models) as hallucination detectors.

Binary Detection: Closed-source models led performance, with Gemini-2.5-Flash achieving the highest F1 (0.860). Open-source models like Qwen3-32B (F1 0.732) showed competitive performance, while Korean-centric models lagged behind global counterparts of similar size.
Multi-Class Detection: Performance dropped significantly when moving from binary detection to multi-class diagnosis (identifying the specific type of hallucination). Even the strongest models struggled with fine-grained diagnostics.
Refusal Behavior: "Justified abstention" remained the weakest axis across all models. Most base models failed to distinguish between True and False Refusal, often predicting one type exclusively.
Fine-Tuning Impact: Fine-tuning an 8B model (Qwen3-8B) with rationale supervision (training the model to output a reasoning trace before the label) yielded an overall accuracy of 0.896, outperforming all base models, including GPT-5 (0.855). This approach significantly improved hallucination detection accuracy (from 0.446 to 0.864).
Sub-type Analysis: "Financial Term Misunderstanding" persisted as a primary bottleneck, showing minimal improvement even with larger model scales (up to 32B), suggesting a need for deeper domain expertise beyond lexical matching.

Significance and Claims

The paper claims that K-FinHallu addresses a critical gap in evaluating financial AI, specifically the need for multi-turn, linguistically nuanced, and regulation-aware benchmarks. The authors emphasize that:

Justified Abstention is Critical: The ability to correctly refuse answering when evidence is missing is a vital but under-evaluated dimension in financial RAG.
Small Models Can Compete: With rationale supervision, a fine-tuned 8B model can achieve performance competitive with or superior to frontier LLMs, offering a practical path for open-source detector development.
Domain Specificity Matters: The persistence of errors in financial term understanding highlights that general scaling laws do not automatically solve domain-specific semantic shifts in Korean financial jargon.

The work serves as a stress test for current LLMs, revealing that even state-of-the-art models struggle with the fine-grained diagnostics and refusal behaviors required for reliable deployment in high-stakes financial environments.