🤖 AI

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

The paper introduces Q-CARE, a query-agnostic and reference-free framework that decomposes queries and answers into atomic units to establish unified metrics for retriever coverage and generator claim verifiability, demonstrating superior correlation with human judgments compared to existing RAG evaluation methods.

Original authors: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. You have a super-smart assistant (a Large Language Model) who can write stories, answer questions, and explain complex topics. But this assistant has a problem: sometimes it makes things up, or it forgets to check its notes. To fix this, we give the assistant a stack of reference books (retrieved evidence) and tell it, "Only answer using what you find in these books." This is called Retrieval-Augmented Generation, or RAG. It's like giving your detective a library card so they don't have to guess.

But here's the tricky part: How do we know if the detective did a good job? If the question is simple, like "Who is the president?", it's easy to check the answer. But what if the question is complicated, like "Explain the history of coffee and how it affects the economy"? The answer might be long, full of different ideas, and hard to grade. Old ways of checking answers were like a strict teacher who only gave points if the student's answer matched a specific "gold standard" word-for-word. This worked for simple facts but failed miserably for complex stories. It couldn't tell if the student missed a key point or if they made up a fact that sounded good but wasn't in the books. We needed a new way to grade that works for any kind of question, whether it's a simple fact or a deep explanation, without needing a perfect answer key to compare against.

Enter Q-CARE, a new evaluation framework proposed by researchers Jeonghwan Choi and his team at KAIST. Think of Q-CARE as a super-organized, hyper-attentive teaching assistant that doesn't just look at the final essay; it breaks the whole process down into tiny, manageable pieces to see exactly where the detective succeeded or failed.

Instead of looking at the whole answer at once, Q-CARE acts like a master chef deconstructing a complex dish. First, it takes the original question and chops it into smaller, bite-sized "sub-questions." If the question is "How do I bake a cake?", it breaks it down into "What ingredients do I need?", "How long do I bake it?", and "What temperature?". It does the same for the answer, breaking the long paragraph into individual "claims" or facts, like "The cake needs flour" or "Bake at 350 degrees."

Then, the magic happens. Q-CARE plays a game of "connect the dots" using only the reference books the detective was allowed to use. It asks three critical questions:

Did the books cover the sub-questions? (Did the library have the info on ingredients?)
Did the answer cover the sub-questions? (Did the student actually write about the ingredients?)
Can the answer's claims be proven by the books? (Is the claim "Bake at 350 degrees" actually written in the book, or did the student guess?)

This approach allows Q-CARE to measure two main things: Coverage (did the answer address everything the question asked?) and Verifiability (is every single sentence backed up by the evidence?).

The researchers tested this new method on a massive benchmark involving eight different types of datasets, ranging from simple news questions to complex scientific explanations. They compared Q-CARE against four other popular evaluation tools. The results were clear: Q-CARE agreed much more with human judges than the other methods. In fact, it achieved a correlation of 0.55 with human judgments for retrieval on close-ended questions and 0.69 for verifiability on open-ended questions, significantly outperforming rivals like RAGEval and RAGChecker, which often struggled when the questions got complicated or open-ended.

One of the coolest things about Q-CARE is that it doesn't need a "gold standard" answer to work. It's "reference-free," meaning it can grade a student's essay even if the teacher doesn't have the answer key, as long as the student used the right textbooks. The team found that this method works just as well for simple "Who is the president?" questions as it does for "Explain the theory of relativity" essays.

However, the paper also suggests that the size of the "brain" (the AI model) doing the grading matters. When they used a smaller AI model to run Q-CARE, the scores weren't as reliable, especially for tricky open-ended questions. But when they used a larger, smarter model (like the Qwen3-30B), the evaluation became much more accurate, suggesting that a strong "teacher" is needed to effectively use this new grading system.

In short, Q-CARE suggests that by breaking questions and answers into tiny, verifiable pieces, we can finally build a grading system that is fair, consistent, and capable of handling the full range of human curiosity, from simple facts to deep, open-ended exploration. It's a step toward making sure our AI assistants are not just sounding smart, but actually being helpful and truthful.

Technical Summary: Q-CARE

Problem Statement

Retrieval-Augmented Generation (RAG) enhances the factuality of Large Language Models (LLMs) by grounding responses in retrieved evidence. However, evaluating RAG systems remains a significant challenge due to the diversity of user queries, which range from "close-ended" fact-seeking questions to "open-ended" requests requiring multi-faceted explanations. Existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across this spectrum.

Current methods face a structural trade-off between Reference-Dependence and Evaluation Granularity:

Reference-Dependent methods (e.g., RAGEval, RAGChecker) offer fine-grained, claim-level diagnostics but rely on static gold answers or gold chunks, limiting their applicability to close-ended queries.
Reference-Free methods (e.g., DoRAG) can handle open-ended queries but often sacrifice granularity, providing coarse signals that lack claim-level verification.

Consequently, no existing method achieves fine-grained, reference-free evaluation while simultaneously supporting both close-ended and open-ended query types.

Methodology: Q-CARE Framework

The authors propose Q-CARE (Query Coverage and clAim veRifiability for RAG Evaluation), a query-agnostic, fully reference-free framework. Q-CARE reinterprets answer correctness not as an isolated check of relevance or reference matching, but as the joint satisfaction of two universal criteria:

Coverage: The answer addresses all information needs implied by the query.
Verifiability: Every statement in the answer is supported by the retrieved evidence.

Q-CARE operates through three distinct stages:

Stage 1: Query-Answer Decomposition

The framework decomposes complex inputs into minimal, self-contained atomic units without information loss:

Query Decomposition: Queries are broken down into sub-queries based on complexity.
- Single-focus (close-ended): Kept as-is.
- Multi-aspect (open-ended): Decomposed into minimal independent perspectives.
- Multi-step reasoning: Decomposed into sequential logical steps.
Answer Decomposition: Generated answers are split into atomic claims. Each claim is self-contained and understandable without the original query (e.g., converting "Jenna Ortega" to "Jenna Ortega plays Harley on Stuck in the Middle").

Stage 2: Coverage and Verifiability Check

Using an LLM-based "Checker," the framework performs three alignment tasks to verify relationships between decomposed units and retrieved chunks ( $C$ ):

Sub-query – Chunk Alignment: Determines if a retrieved chunk contains sufficient information to fully answer a specific sub-query.
Sub-query – Claim Alignment: Identifies which answer claims are relevant to addressing specific sub-queries.
Claim – Chunk Alignment: Verifies if specific answer claims are supported by the retrieved evidence.

Stage 3: Metric Computation

The atomic-level alignment signals are aggregated into five complementary evaluation dimensions:

Retriever Metrics (Coverage-Aware):
Replacing binary human relevance labels with soft coverage scores derived from sub-query coverage:

C-Prec@k: The proportion of retrieved chunks containing query-relevant information, measured by the fraction of sub-queries they cover.
C-nDCG@k: The ranking quality of retrieved chunks, weighting soft coverage scores by their retrieval position.

Generator Metrics:

Completeness: The proportion of sub-queries fully addressed by the generated answer.
Conciseness: The proportion of answer claims that are relevant to the covered sub-queries (penalizing irrelevant verbosity).
Verifiableness: The proportion of answer claims supported by the retrieved evidence.

Key Contributions

Unified Evaluation Principle: The paper redefines answer correctness as the joint satisfaction of query coverage and claim verifiability, creating a principle applicable to both close-ended and open-ended queries.
Q-CARE Framework: A decompositional framework that enables end-to-end, fine-grained, and fully reference-free evaluation by jointly assessing retrieval and generation.
Human-Annotated Benchmark: The construction of a new benchmark spanning eight diverse datasets (covering general, news, finance, biomedical, science, and technology domains) with human annotations for sub-query coverage and claim grounding.
Empirical Validation: Demonstration that Q-CARE achieves higher agreement with human judgments than existing RAG evaluation metrics.

Results

The authors evaluated Q-CARE against four existing RAG metrics (RAGEval, RAGAs, RAGChecker, DoRAG) and traditional metrics (Precision, nDCG, ROUGE-L, EM) on their human-annotated benchmark.

Correlation with Human Judgments: Q-CARE achieved the highest Pearson correlation with human judgments across all evaluation dimensions (Retrieval and Generation) and for both query types. For instance, on open-ended queries, Q-CARE's Verifiableness metric correlated at 0.69, significantly outperforming RAGChecker (0.56) and RAGEval (0.05).
Query Agnosticism: Unlike baselines that showed performance degradation when moving from close-ended to open-ended queries, Q-CARE maintained high correlation across both types.
Retriever Benchmarking: When applied to 21 retriever strategies, Q-CARE's coverage-aware metrics (C-Prec, C-nDCG) revealed significant rank discrepancies compared to traditional metrics, particularly for sparse retrievers and query expansion methods. Coverage-aware metrics increased the Spearman rank correlation between close-ended and open-ended retriever rankings from 0.55 to 0.89.
Generator Benchmarking: Evaluation of 8 LLMs (including GPT-5, Claude-Sonnet-4, and various open-source models) revealed that no single model dominates all dimensions. For example, proprietary models excelled in Completeness, while performance in Conciseness and Verifiableness varied significantly by model and query type.
Generalization: Q-CARE successfully generalized to multi-turn conversational RAG (MTRAG benchmark) without framework modification, achieving the highest correlation on three of four datasets.
Ablation Studies: Removing query decomposition significantly reduced evaluation quality, confirming that decomposition is a critical component rather than an auxiliary step.

Significance and Claims

The paper claims that Q-CARE addresses a critical bottleneck in RAG evaluation: the inability to provide consistent, fine-grained diagnostics across diverse query types without relying on expensive or unavailable reference answers.

By shifting the evaluation focus from "is this chunk relevant?" or "does this match the reference?" to "does the answer satisfy the query's intent and is it grounded in evidence?", Q-CARE offers a reliable, automated, and query-agnostic evaluation framework. The authors posit that query coverage is a critical factor often overlooked in reliable RAG benchmarking, and their framework provides the necessary tools to measure it effectively. The work establishes that fine-grained, reference-free evaluation is achievable and superior to current state-of-the-art methods in aligning with human judgment.