LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

This paper introduces LIT-RAGBench, a comprehensive benchmark dataset and evaluation framework designed to systematically assess Large Language Models' capabilities in Retrieval-Augmented Generation across five critical categories—Integration, Reasoning, Logic, Table, and Abstention—using a mix of human-constructed Japanese questions and curated English translations to guide model selection and development for practical RAG deployments.

The Gist

Imagine you are hiring a super-smart research assistant (a Large Language Model, or LLM) to answer questions for your company. You tell them, "Don't just guess; go look up the facts in our company files first." This setup is called RAG (Retrieval-Augmented Generation).

The problem? Even the smartest assistants sometimes make mistakes. They might:

• Hallucinate: Make up facts because they are confident but wrong.
• Miss the point: Find the right file but fail to connect the dots between two different documents.
• Get confused by charts: Look at a spreadsheet and completely misread the numbers.
• Refuse to answer: Or worse, refuse to answer when they could have found the answer.

Until now, there hasn't been a good "driver's test" to see exactly how good these assistants are at all these specific skills at the same time.

Enter: LIT-RAGBench

The authors of this paper built a new, rigorous exam called LIT-RAGBench. Think of it as a multi-skill obstacle course designed to test a research assistant's real-world readiness.

The name stands for Logic, Integration, Table, Reasoning, and Abstention. Here is what each part of the obstacle course looks like, using simple metaphors:

1. Integration (The "Puzzle Master")

• The Test: The assistant is given three different documents. The answer isn't in just one; it's a puzzle where Piece A is in Doc 1, and Piece B is in Doc 2.
• The Metaphor: Imagine asking, "Who won the award?" The assistant has to read a newsletter (Doc 1) that says "Alice won," and a separate email (Doc 2) that says "Alice is from the Marketing team." It must combine these to say, "Alice from Marketing won." If it only reads one, it fails.

2. Reasoning (The "Detective")

• The Test: The answer isn't stated directly. The assistant has to do a "multi-hop" deduction.
• The Metaphor: The document says, "The meeting was moved to Tuesday." Another says, "Tuesday is a holiday." The assistant must deduce, "Therefore, the meeting is effectively cancelled or moved again," even though no one explicitly wrote "cancelled." It also tests math skills, like calculating a total profit from a list of sales, which many AI models surprisingly struggle with.

3. Logic (The "Translator")

• The Test: The question uses different words than the document.
• The Metaphor: You ask, "Is the $10,000 budget approved?" The document says, "The ten thousand dollar fund is greenlit." A human knows these are the same. An AI might get confused and say, "I don't see $10,000," missing the synonym. It also tests if the AI understands boundaries (e.g., "Is a 39-year-old eligible for 'under 40'?").

4. Table (The "Chart Reader")

• The Test: The information is hidden inside messy spreadsheets, HTML tables, or CSV files.
• The Metaphor: Imagine a table where rows and columns are merged together (like a complex Excel sheet). The AI has to find a specific number in a cell that is part of a merged block. This is like trying to find a specific seat in a theater where the aisle signs are missing and the rows are merged. Many AIs get lost here.

5. Abstention (The "Honesty Check")

• The Test: The assistant is asked a question where the documents don't have the answer, or the documents contradict each other.
• The Metaphor: You ask, "What is the CEO's favorite color?" The documents only talk about the CEO's business strategy. A good assistant should say, "I don't know, the files don't say." A bad assistant will make up a color (like "Blue") just to be helpful. This section tests if the AI knows when to shut up and admit ignorance.

The Results: Who Passed the Test?

The researchers ran this test on many of the world's smartest AI models (like GPT-5, Claude, Llama, and Qwen).

• The Big Surprise: No model got a perfect score. In fact, no model even reached 90% accuracy. Even the "smartest" models got stuck on specific types of puzzles.
• The Weak Spots:
  • Math & Logic: Many models struggled with simple calculations or understanding that "10k" means "10,000."
  • Tables: Reading messy spreadsheets was a nightmare for almost everyone.
  • Honesty: Some models were too eager to answer (hallucinating), while others were too scared to answer (refusing to answer even when they had the facts). This is called the "Over-Abstention" problem.

Why Does This Matter?

Think of LIT-RAGBench as a quality control checklist for businesses.

If you are a company trying to build a chatbot for your employees, you can't just pick the "most popular" AI. You need to know:

• "Does this model get confused by our financial spreadsheets?" (Table skill)
• "Will it make up facts if the data is missing?" (Abstention skill)
• "Can it connect dots across different reports?" (Integration skill)

The Bottom Line:
AI is getting incredibly smart, but it's not perfect yet. This new benchmark shows us exactly where it breaks down. It tells us that before we trust AI to run our businesses, we need to fix its ability to read charts, do math, and know when to say, "I don't know."

The authors have made this test open-source, so anyone can use it to train better, more reliable AI assistants for the real world.

Technical Summary

Here is a detailed technical summary of the paper "LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation."

1. Problem Statement

Retrieval-Augmented Generation (RAG) is a critical framework where a Generator (LLM) produces answers based on documents retrieved by a Retriever. While RAG addresses issues like hallucinations and outdated information, practical deployment requires the Generator to possess complex, multifaceted capabilities simultaneously. These include:

• Integrating evidence from multiple documents.
• Performing multi-step (multi-hop) reasoning.
• Interpreting complex tabular data.
• Maintaining logical consistency despite lexical mismatches.
• Knowing when to abstain from answering when evidence is missing or contradictory.

The Gap: Existing benchmarks (e.g., FRAMES, RAGBench, RGB) typically evaluate these capabilities in isolation or lack unified conditions that simulate real-world complexity. There is no standard benchmark that systematically evaluates the combinations of these capabilities (e.g., reasoning over a merged-cell table) or measures the Generator's ability to abstain appropriately under unified conditions.

2. Methodology: LIT-RAGBench

The authors propose LIT-RAGBench (Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), a framework designed to evaluate Generator capabilities independently of retrieval quality.

A. Evaluation Categories

The benchmark defines five core categories, subdivided into specific evaluation aspects:

1. Integration (Main): Requires synthesizing information from multiple sources ($2 \le |C^+| \le 3$).
2. Reasoning (Main):
   • Multi-hop: Inferring conclusions not explicitly stated in a single source.
   • Numerical Calculation: Deriving metrics (totals, averages) without explicit formulas.
3. Logic (Main): Resolving semantic/lexical discrepancies between the query and context.
   • Synonym Interpretation: Handling unit/term variations (e.g., "10k yen" vs "10,000 yen").
   • Numerical Inclusion: Boundary reasoning (e.g., age ranges).
   • Conceptual Inclusion: Hierarchical relations (e.g., "earphones" are "electronics").
4. Table (Main): Extracting data from structured formats (HTML, Markdown, CSV), including complex cases like merged cells ($rowspan/colspan$).
5. Abstention (Exceptional): Evaluating the model's ability to refuse answering when:
   • Insufficient Evidence: No relevant chunks are retrieved.
   • Contradictory Evidence: Retrieved documents conflict.
   • Incomplete Chunks: Semantically connected info is split across chunk boundaries.

B. Dataset Construction

• Structure: The dataset consists of 114 human-constructed questions (54 Japanese + 60 derived/translated for English).
• Design: Uses fictional entities (companies, products, people) to prevent models from relying on pre-trained knowledge.
• Composition: Questions are constructed to test single or dual-category combinations (e.g., Reasoning + Table).
• Process: A hybrid approach involving LLM-assisted generation followed by rigorous human curation and filtering by native Japanese speakers to ensure quality and adherence to evaluation aspects.
• Input Handling: To mitigate position bias, retrieved chunks are randomized during input.

C. Experimental Setup

• Models Evaluated: A diverse set of API-based models (GPT-5 series, o3, o4-mini, Gemini-2.5, Claude-Sonnet-4) and open-weight models (Llama-3, Gemma-3, Qwen3).
• Evaluation Metric: LLM-as-a-Judge (using GPT-4.1) to determine semantic consistency between the generated answer and the reference answer.
• Metrics: Category-wise accuracy and overall accuracy. A specific metric, Over-Abstention Rate, was introduced to measure models that refuse to answer even when sufficient evidence exists.

3. Key Results

The experiments revealed that no model exceeded 90% overall accuracy on either the Japanese or English datasets.

• Top Performers:
  • GPT-5 achieved the highest overall accuracy (0.872).
  • Among open-weight models, Qwen3-235B-Instruct (0.859) and Qwen3-235B-Thinking (0.821) performed best.
  • Smaller models (e.g., Llama-3.1-8B, Gemma-3-27B) showed significantly lower performance, particularly in Reasoning and Table tasks.
• Category-Specific Insights:
  • Table: Models struggled significantly with merged-cell tables and large tables split across chunks. Even high-performing models often failed to recognize the overall structure when chunks were shuffled.
  • Reasoning: While some models (e.g., o3) excelled at multi-hop reasoning, many failed at numerical calculations, making arithmetic errors in intermediate steps.
  • Logic: Models frequently failed at unit conversions and boundary inclusion (e.g., answering "500 MB" instead of "0.5 GB").
  • Abstention:
    • Claude-Sonnet-4 showed the highest accuracy in abstention tasks but also the highest Over-Abstention Rate (25.9%), indicating a tendency to refuse valid answers due to excessive caution.
    • Smaller models (e.g., Llama-3.1-8B) also over-abstained (21.3%), likely due to weak reasoning capabilities in the main categories.

4. Key Contributions

1. Unified Benchmark: Introduced the first benchmark that systematically evaluates combinations of RAG capabilities (Integration, Reasoning, Logic, Table, Abstention) under unified conditions, bridging the gap between isolated skill testing and real-world complexity.
2. Realistic Evaluation Aspects: Defined specific, practical failure modes (e.g., merged cells, incomplete chunks, contradictory evidence) often overlooked in previous benchmarks.
3. Over-Abstention Metric: Quantified the trade-off between safety (refusing to answer) and utility (answering correctly), revealing that high abstention rates do not always correlate with better performance.
4. Open Resources: Released the dataset, prompts, and code to facilitate reproducibility and further research in RAG optimization.

5. Significance

• Model Selection: LIT-RAGBench provides a granular metric for selecting the right LLM for specific RAG deployment scenarios (e.g., choosing a model strong in Table parsing vs. one strong in Logic).
• System Optimization: The results highlight that preprocessing (e.g., table restructuring, chunk reordering) is as critical as model selection, as even top models fail when data is structurally fragmented.
• Future Direction: The benchmark identifies that current models struggle with compound complexity (e.g., reasoning over tables). It suggests that future RAG systems require specialized training or "Agentic RAG" approaches where models can autonomously plan retrieval and reasoning steps to handle these intertwined tasks.
• Safety vs. Utility: The findings on over-abstention warn developers that optimizing for "safety" (hallucination avoidance) can inadvertently degrade the system's usefulness by causing it to miss valid answers.