💬 NLP

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval

The DS@GT team proposes a language-routed, retrieval-augmented pipeline for multilingual financial QA that combines BGE-M3 embeddings with weighted Reciprocal Rank Fusion and direct option scoring via next-token log-probabilities, demonstrating that optimal performance requires language-specific model selection and careful avoidance of chain-of-thought prompting for certain languages.

Original authors: Justice Ayela, Kabir Sahni

Published 2026-07-28✓ Author reviewed ⓘ

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Justice Ayela, Kabir Sahni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a tricky puzzle, but instead of just using your own brain, you have a massive library of reference books right next to you. This is the world of Retrieval-Augmented Generation (RAG). Think of it like a student taking a test who is allowed to look up facts in a textbook before answering. Usually, computers are great at this if the questions are in English, but things get messy when the questions are in Spanish, Greek, or Hindi. The "library" might be full of English books but empty for other languages, and the student might not know how to read the foreign ones even if they find them.

This paper tackles a very specific, high-stakes version of this problem: Financial Certification Exams. These aren't just trivia; they are complex tests for becoming a Certified Financial Analyst or Accountant, requiring deep logic about money, laws, and accounting rules. The big question the authors ask is: Can we build a computer system that passes these hard money exams in five different languages (English, Spanish, Greek, Chinese, and Hindi) just as well as it does in English? The answer matters because finance is global, but most computer brains are trained mostly on English, leaving huge gaps in how they handle money talk in other languages.

The Team's Big Experiment: A Smart, Multilingual Detective

The researchers from Georgia Tech, calling their team DS@GT, built a digital detective system to solve these financial exam questions. Instead of just guessing, their system follows a strict three-step process, which they call a "pipeline."

Step 1: The Language Detective
First, the system looks at the question and asks, "What language is this?" It's like a bouncer at a club checking IDs. If the question is in Greek, the system knows it needs to pull books from the "Greek Section" of its library. If the library is empty for that language (which happens with smaller languages like Greek or Hindi), it grabs books from the "Global Section" that might have similar ideas, even if they are in a different language. They used a special tool called BGE-M3 to translate the meaning of the questions into a universal code so the system could find relevant examples, even across language barriers.

Step 2: The Model Router (The "Right Brain" for the Right Job)
Here is where the team found something surprising. They didn't just use one giant computer brain for everything. They discovered that different computer brains are better at different languages.

For English, they used a model called Qwen2.5-14B.
For Greek, they surprisingly found that a smaller model, Llama-3.1-8B, was actually the superstar, beating the bigger models by a huge margin (about 19.5 percentage points!).
For Chinese, Hindi, and Arabic, they used Qwen3-14B.

If they had just used the "best" model for everyone, the system would have failed miserably on English and Greek. It's like realizing that while a marathon runner is great for long distances, you wouldn't want them to play chess; you need a different specialist for each task.

Step 3: The Scorer (No More Guessing Games)
Usually, when computers answer questions, they try to "write" the answer, like "The answer is C." This often leads to mistakes where the computer writes a long paragraph and forgets to say which letter it picked. The DS@GT team used a clever trick called Retrieval-Augmented Direct Scoring (RADS). Instead of writing the answer, the system simply checks the math: "How likely is the letter 'A' to be the next word? How likely is 'B'? How likely is 'C'?" It picks the letter with the highest math score. This is like a multiple-choice test where the computer doesn't have to write an essay; it just points to the right bubble. This method eliminated almost all "parse failures" (mistakes in reading the computer's own answer), achieving a 100% success rate on Chinese questions where other methods struggled.

The Surprising Discoveries

The team ran thousands of tests and found some rules that broke the usual ideas about how AI works:

Thinking Too Much Can Be Bad: For most people, "Chain-of-Thought" (asking the AI to explain its reasoning step-by-step) helps. But for Greek questions, this actually destroyed the performance. The accuracy dropped from a fantastic 90.7% down to a terrible 20.9%. It turns out that for these specific Greek questions, which are more about categorizing facts than solving math problems, making the AI "think out loud" confused it. It started writing stories instead of picking the right category.
The "Thinking Mode" Trap: The model Qwen3 has a default setting where it writes a "thinking" section before the answer. The team found that if they left this on, the system's ability to pick the right letter (RADS) collapsed to near-random guessing for Arabic. They had to manually turn off this "thinking mode" for the system to work.
Translation is a Trap: They tried translating questions from other languages into English, solving them, and translating back. This was a disaster. The translation lost important financial details, and the scores dropped by nearly 20 percentage points compared to solving them in the original language.
Data Matters, But Models Matter More: For Hindi, adding more native Hindi examples to their library helped, but only by a small amount (about 1 percentage point). However, the gap between their system and the top commercial AI (Claude Opus 4.7) was huge (17 percentage points). This suggests that for low-resource languages, the problem isn't just having more books in the library; it's that the computer brain itself needs to be trained more deeply on that specific language's financial terms.

The Final Scorecard

When the team submitted their system to the official FinMMEval 2026 competition, they did well, ranking in the top 10 for most languages.

Arabic: 76.0% (10th place)
Hindi: 73.5% (7th place)
English: 69.5% (9th place)
Chinese: 64.5% (9th place)

While they didn't win first place (the top teams scored over 90%), their work proved a vital lesson: One size does not fit all. To build a smart financial AI that works globally, you can't just use one model and one strategy. You need to route different languages to different models, choose different thinking strategies based on the task, and score answers mathematically rather than by writing them out. The future of global finance AI isn't about a single super-brain; it's about a team of specialists, each speaking the right language and using the right tools.

tokens) is explicitly disabled (enable_thinking=False`). This is necessary because the reasoning tokens displace the answer letter from the immediate next-token position, breaking the scoring mechanism.

3. Retrieval-Augmented Direct Scoring (RADS)

Instead of generating free-form text, the system uses Retrieval-Augmented Direct Scoring (RADS):

Mechanism: The model receives a prompt containing the query and top-5 retrieved exemplars. It performs a single forward pass and calculates the next-token log-probability for each valid option letter (A, B, C, D, E).
Prediction: The answer is the argmax of these log-probabilities.
Advantage: This approach eliminates parse failures (e.g., regex extraction errors) and formatting biases inherent in generation-based methods.

Key Contributions

Structured Multilingual RAG Pipeline: A tailored architecture for financial exam reasoning that enforces language consistency in retrieval and model selection.
Two-Layer Exemplar Retrieval: A strategy combining per-dataset and global indices using weighted RRF to handle low-resource languages effectively.
Empirical Analysis of Asymmetry: The paper documents significant, non-intuitive performance gaps:
- Model Routing Necessity: No single model is optimal across all languages. Qwen3-14B outperforms Qwen2.5-14B on Chinese (+4.2 pp) but regresses on English (-22.9 pp). Llama-3.1-8B outperforms all Qwen variants on Greek by +19.5 pp.
- Reasoning Strategy Sensitivity: Chain-of-Thought (CoT) prompting significantly degrades performance on Greek (90.7% $\to$ 20.9%) and Hindi (-19 pp), while only marginally helping Chinese.
- Scoring Strategy: RADS achieves 100% parse rates and outperforms few-shot generation, particularly in eliminating the ambiguity of free-form decoding.

Results

Dev-Leaderboard Performance: The system achieved accuracies of 76.0% (Arabic), 73.5% (Hindi), 69.5% (English), and 64.5% (Chinese) on the final test set.
Impact of Native Data: Expanding the Hindi KB from cross-lingual proxies to a native shard of 5,982 exemplars yielded a modest +1.0 pp improvement, suggesting that for Hindi, the bottleneck is base-model capacity rather than retrieval quality.
Comparison to Frontier Models: A 17 percentage-point gap remains between the best local system (Qwen3-14B + RADS) and frontier API models (Claude Opus 4.7) on Hindi, indicating that retrieval improvements alone cannot close the gap for low-resource languages.
Solver Ablation: RADS (Solver B) consistently tied or exceeded few-shot generation (Solver A) across languages. Translation-based solvers (Solver C) were the worst performers, and CoT with self-consistency (Solver D) caused severe degradation in Greek and English.

Significance and Claims

The paper argues that effective multilingual financial QA requires deliberate heterogeneity at every layer of the pipeline. It challenges two common assumptions:

Universal Backbone: A single capable model cannot serve as a universal backbone; language-specific routing is a prerequisite for competitive performance.
Uniform Reasoning: More sophisticated reasoning strategies (like CoT) do not yield uniform gains; they can be detrimental depending on the task type (e.g., text classification vs. quantitative derivation) and language.

The authors conclude that implementation details—such as disabling thinking modes in specific models and selecting the correct model per language—are as critical as the retrieval infrastructure. They position their work as establishing the reproducible "open-weight ceiling" for this task, highlighting that the remaining performance gap to frontier models is driven by base-model capacity in low-resource languages rather than retrieval quality.