Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering

This paper presents a case study demonstrating that Hybrid Retrieval-Augmented Generation (RAG) consistently outperforms direct long-context prompting across English, French, and German technical agricultural manuals, achieving over 85% accuracy with models like Gemini 2.5 Flash and Qwen 2.5 7B in cross-lingual question answering.

The Gist

Imagine you are a farmer standing in front of a massive, 165-page instruction manual for a high-tech fertilizer machine. The manual is written in English, French, and German. You have a specific question: "What is the exact torque for the vane lock nuts?"

Now, imagine you have two different ways to find that answer:

1. The "Super-Reader" (Long-Context LLM): You hand the entire 165-page book to a genius student who can read 128,000 words at once. You ask them to find the answer.
2. The "Librarian" (RAG): You ask a librarian to quickly scan the book, find the three most relevant pages, and hand only those pages to the student. Then, the student reads just those pages to answer you.

This paper, "Agri-Query," is a race between these two approaches to see who is better at finding the needle in the haystack.

The Setup: A Real-World Test

The researchers from the Technical University of Munich didn't just use random text. They used a real, complex manual for an agricultural machine (the Kverneland Exacta-TLX). They created a test with 108 questions:

• 54 "Findable" questions: The answer is definitely in the book.
• 54 "Unanswerable" questions: The answer is not in the book (e.g., "How much extra diesel does the tractor use?"). This was a trap to see if the AI would lie (hallucinate) just to be helpful.

They tested this in three languages: English, French, and German. Crucially, they asked all questions in English, even when the manual was in French or German, to see if the AI could bridge the language gap.

The Contenders

They pitted 9 different AI models (including famous ones like Llama, Qwen, and the proprietary Gemini) against each other. Some were huge, some were small. They tested them in two modes:

1. Direct Long-Context: Feeding the whole book to the AI.
2. RAG (Retrieval-Augmented Generation): Using a search system to find the right pages first.

They tested three types of "Librarians" (Retrieval methods):

• Keyword Search: Like using "Ctrl+F" to find exact words.
• Semantic Search: Like asking a librarian, "I need the part about tightening nuts," even if the word "nut" isn't used.
• Hybrid Search: A mix of both, using the best of both worlds.

The Results: The Librarian Wins!

Here is the big takeaway, explained simply:

1. The "Super-Reader" gets overwhelmed.
When the AI tried to read the entire 165-page manual at once, it struggled. It suffered from what researchers call the "Lost in the Middle" effect. Imagine reading a 500-page novel and being asked about a detail on page 250. If you read the whole thing in one go, you might forget the middle parts. The AI got confused by all the "noise" (irrelevant text) and often missed the answer or made things up.

2. The "Hybrid Librarian" is the champion.
The Hybrid RAG approach (Keyword + Semantic search) consistently won. By finding the specific pages first and handing them to the AI, the models performed much better.

• Accuracy: They got over 85% of the answers right.
• Honesty: They were much better at admitting, "I don't know," when the answer wasn't in the book, rather than making up a fake answer.
• Small Models Shine: Surprisingly, smaller, cheaper AI models (like the 7B or 3B parameter versions) performed just as well as the giant models when they had the Librarian helping them. This means you don't need a supercomputer to get great results; you just need a good search system.

3. Language is no barrier.
Even though the questions were in English and the manuals were in French or German, the Hybrid RAG system worked brilliantly. The "Librarian" understood the meaning of the question and found the right French or German pages, then the AI translated the answer back to English.

The "Hallucination" Trap

One of the most important findings was about lying.

• When the AI had to read the whole book (Long-Context), it often tried to guess the answer to the "Unanswerable" questions, leading to hallucinations (confidently wrong answers).
• When the AI used the Librarian (RAG), it was much more honest. If the Librarian couldn't find the page, the AI was more likely to say, "Not found," rather than inventing a story.

The Bottom Line

If you are building an AI system to help people read technical manuals (like for farming, medicine, or engineering):

• Don't just dump the whole book into the AI. It gets confused and loses focus.
• Use a "Search First" strategy (RAG). Find the relevant pages, then ask the AI to read them.
• Mix your search methods. Combining keyword search with "meaning" search gives the best results.
• You don't need the biggest AI. A smaller, cheaper AI works great if it has a good search tool helping it.

In short: Don't ask the AI to memorize the library; ask it to use the card catalog.

Technical Summary

Here is a detailed technical summary of the paper "Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering."

1. Problem Statement

The paper addresses the challenge of deploying Large Language Models (LLMs) for technical Question Answering (QA) in specialized industrial domains, specifically agricultural machinery. Key challenges identified include:

• Document Complexity: Modern agricultural manuals are extensive (approx. 59k tokens), covering mechanical, electronic, and agronomical aspects.
• Cross-Lingual Retrieval: Manufacturers must maintain manuals in multiple languages (English, French, German), but operators may query in a different language than the manual's native language.
• The "Needle-in-a-Haystack" Problem: Answers are often located in a single specific page within a massive document, making retrieval difficult.
• Hallucination Risks: In safety-critical domains, models must accurately identify when information is not present (unanswerable questions) rather than fabricating answers.
• Architectural Trade-off: There is an ongoing debate between using Long-Context (LC) models (feeding the entire document to the LLM) versus Retrieval-Augmented Generation (RAG) (retrieving specific chunks first).

2. Methodology

The authors constructed a benchmark called Agri-Query to evaluate nine open-source LLMs and one proprietary model (Gemini 2.5 Flash).

A. Dataset and Setup

• Source Material: The user manual for the Kverneland Exacta-TLX Geospread GS3 fertilizer spreader, available in English, French, and German. Each version is 165 pages (59k tokens) with identical layouts.
• QA Dataset: 108 questions curated by domain experts:
  • 54 Answerable: Factual questions requiring specific data extraction.
  • 54 Unanswerable: Questions not found in the text to test for hallucinations.
  • Cross-Lingual Design: All questions were posed in English, while the retrieval targets were the English, French, and German versions of the manual.
• Evaluation Metric: An LLM-as-a-Judge framework (using Gemma 2 and Gemma 3) compared model outputs against ground-truth answers, calculating Accuracy, F1, Precision, Recall, and Specificity.

B. Experimental Approaches

The study compared two primary architectures:

1. Direct Long-Context (LC): Models were prompted with the entire manual (or large sections with added "noise" pages) without retrieval. Context sizes ranged from 1k to 59k tokens.
2. Retrieval-Augmented Generation (RAG): Models were provided only with the top 3 retrieved chunks. Three retrieval strategies were tested:
   • Keyword (BM25): Frequency-based matching.
   • Semantic (Embedding): Vector-based matching using gte-Qwen2-7B-instruct.
   • Hybrid: A combination of BM25 and Semantic search, merged using Reciprocal Rank Fusion (RRF).

3. Key Contributions

• Specialized Benchmark: Creation of a rigorous, cross-lingual QA benchmark for industrial agricultural machinery, including a significant set of unanswerable questions to test safety.
• Comparative Analysis: A direct empirical comparison between state-of-the-art Long-Context models and various RAG strategies in a real-world technical scenario.
• Open Framework: Release of a reproducible framework (GitHub) for data preparation, RAG implementation, and evaluation, facilitating future research in industrial AI.
• Cross-Lingual Insights: Demonstration that English queries can effectively retrieve information from non-English technical documents when paired with strong multilingual embedding models.

4. Key Results

A. RAG vs. Long-Context

• Hybrid RAG Superiority: Hybrid RAG consistently outperformed direct long-context prompting across all models.
  • Gemini 2.5 Flash (RAG): Achieved 88.0% Accuracy and 88.9% F1 on English.
  • Gemini 2.5 Flash (Full Context): Dropped to 69.4% Accuracy and 74.4% F1.
• Small Model Viability: Smaller models (e.g., Qwen 2.5 7B, Llama 3.2 3B) achieved high accuracy (>85%) with Hybrid RAG, whereas they failed significantly when fed the full manual context.
• Lost in the Middle Effect: Long-context tests revealed a pronounced "Lost in the Middle" phenomenon, where models struggled to locate facts in the middle of long documents, leading to performance degradation as context size increased.

B. Cross-Lingual Performance

• Robustness: Hybrid RAG maintained high performance when English queries were matched against French and German manuals.
  • Example: Qwen 2.5 7B achieved 85.2% Accuracy on French and 79.6% on German using Hybrid RAG, comparable to its English performance.
• Embedding Importance: The use of a strong multilingual embedding model (gte-Qwen2-7B-instruct) was critical for bridging the language gap, outperforming keyword-only retrieval in cross-lingual scenarios.

C. Failure Modes

• Hallucination (False Positives): Smaller models (e.g., Llama 3.2 1B) showed low Specificity (0.204), meaning they frequently invented answers for unanswerable questions.
• Retrieval Failure: While Hybrid RAG minimized this, pure keyword retrieval failed when synonyms were used, and pure semantic retrieval occasionally missed exact technical terms.

5. Significance and Implications

• Practical Recommendation: For technical documentation in safety-critical industries, Hybrid RAG is the preferred architecture over relying solely on long-context windows. It offers higher accuracy, better hallucination control, and allows for the use of smaller, more cost-effective models.
• Resource Efficiency: RAG enables high-performance QA on resource-constrained hardware (e.g., single GPUs) by avoiding the massive memory overhead of processing 100k+ token contexts.
• Global Applicability: The study proves that multilingual industrial support systems can be built where operators query in their native language (or English) against manuals in other languages, provided the retrieval layer is robust.
• Safety First: The inclusion of unanswerable questions highlights that model size alone does not guarantee safety; architectural choices (RAG) and evaluation metrics (Specificity) are crucial to prevent dangerous misinformation.

In conclusion, the paper demonstrates that while Long-Context LLMs are improving, Retrieval-Augmented Generation (specifically Hybrid RAG) remains the superior, more robust, and cost-effective solution for precise, cross-lingual technical question answering in industrial settings.