💬 NLP

Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation

This paper proposes a comprehensive factuality evaluation framework for long-form LLM generation that jointly measures precision and importance-aware recall, revealing that current models significantly struggle with factual completeness despite strong performance on precision.

Original authors: Nazanin Jafari, James Allan, Mohit Iyyer

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nazanin Jafari, James Allan, Mohit Iyyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a tour guide for a trip to a famous city. You want them to tell you the truth about the landmarks, but you also want them to tell you everything important about the city, not just the parts they feel like mentioning.

This paper is about a new way to grade these "AI tour guides" (Large Language Models) on how well they tell the truth and how complete their stories are.

The Old Way: The "Spot Check" (Precision Only)

Previously, when we checked if an AI was telling the truth, we used a method called Precision.

Think of this like a teacher giving a student a pop quiz. The teacher picks 10 random sentences the student wrote and checks if they are true.

If the student wrote 10 sentences and 9 were true, they get a 90% score.
The Problem: This ignores what the student didn't write. If the student wrote 10 true sentences but missed the 50 most important facts about the city (like the location of the museum or the price of the train ticket), they still get a 90% score. They are technically "accurate," but they are a terrible guide because they left out the essentials.

The New Way: The "Complete Map" (Precision + Recall)

The authors of this paper say, "We need to check two things at once":

Precision: Are the things you did say true? (The Spot Check)
Recall: Did you say all the important things you should have said? (The Complete Map)

To do this, they built a system that acts like a super-researcher:

The Researcher: Before the AI answers a question, the system searches the internet (like Wikipedia) to find a "Master List" of all the true facts about that topic.
The Grader: The system then compares the AI's answer against this Master List.
- It checks: "Did the AI get the facts right?" (Precision)
- It checks: "Did the AI miss any facts from the Master List?" (Recall)

The "VIP" Factor: Importance-Aware Scoring

Here is the clever twist. Not all facts are created equal.

Fact A: "The Eiffel Tower is in Paris." (Very Important)
Fact B: "The Eiffel Tower was painted a specific shade of brown in 1982." (Less Important)

The authors realized that if an AI misses Fact A, it's a huge failure. If it misses Fact B, it's a minor slip-up. So, they gave their grading system a VIP Pass.

They teach the system to weigh the "Important Facts" (Salience) and the "Relevant Facts" (Relevance) more heavily.
If the AI gets the VIP facts right but misses the boring ones, it still gets a decent score. If it misses the VIP facts, the score tanks.

What They Found: The "Chatty but Incomplete" Problem

They tested this on several AI models and found a funny pattern:

The AI is great at not lying: When the AI does make a statement, it is usually correct. It rarely says something that is clearly false (like "The moon is made of cheese").
The AI is bad at being thorough: The AI often forgets to mention the big, important things. It gives you a short, accurate story but leaves out the whole second half of the chapter.

The Analogy:
Imagine the AI is a student taking a history test.

Old Grading: The teacher only looks at the answers the student wrote. "You got 5/5 right! A+!"
New Grading: The teacher looks at the whole test. "You got 5/5 right, but you only answered 5 out of 50 questions. You missed the most important ones. Your grade is a C."

The Trade-Off: More Words vs. Better Words

The paper also looked at what happens when AI models try to talk more to get a better score.

Some models just start rambling, writing huge paragraphs hoping to accidentally hit the right facts. This increases their "Recall" (they cover more ground) but lowers their "Precision" (they start making up nonsense or repeating themselves).
The best models are the ones that say less, but say the right important things. They are efficient, not just verbose.

The Bottom Line

This paper introduces a better way to judge AI. It stops us from being fooled by AI that sounds confident and accurate but is actually leaving out the most critical information.

In short: We don't just want AI that tells the truth; we want AI that tells the whole truth, especially the parts that matter most. The authors built a tool to measure exactly that.

1. Problem Statement

Evaluating the factuality of long-form Large Language Model (LLM) outputs remains a significant challenge. Current evaluation paradigms predominantly rely on factual precision:

Method: Decomposing generated responses into atomic claims and verifying them against external knowledge sources (e.g., Wikipedia, search engines).
Metric: Calculating the ratio of supported claims to total generated claims.
Limitation: This approach inherently ignores factual recall. It fails to measure whether the model omitted critical, relevant information that should have been included. A response can be 100% precise (no hallucinations) but 0% complete (missing key facts), yet traditional metrics would rate it highly. Furthermore, existing recall approximations often rely on brittle heuristics (e.g., a fixed target number of facts) or query-level coverage rather than fine-grained factual units.

2. Methodology

The authors propose a unified framework that jointly measures Factual Precision and Factual Recall, incorporating an importance-aware weighting scheme.

A. Reference Fact Set Construction ( $F$ )

Instead of relying on model behavior to determine how many facts should be present, the framework constructs a ground-truth reference set using Retrieval-Augmented Generation (RAG):

Retrieval: For a given query $q$ , top- $k$ documents are retrieved from an external knowledge source ( $S$ ).
Fact Extraction: An LLM extracts atomic, non-redundant factual statements from the retrieved passages.
Importance Scoring: Recognizing that not all facts are equally critical, the framework assigns an importance score to each fact $f_k$ $f_{k}$ based on two dimensions:
- Relevance ( $r_k$ ): How directly the fact answers the specific query.
- Salience ( $s_k$ ): How central the fact is to the main topic/entity, independent of the specific query phrasing.
- Composite Score: $imp(f_k; q) = \alpha \tilde{r}_k + \beta \tilde{s}_k$ .
Reference Set ( $F^*$ ): Facts are ranked by this composite score. A "should-include" reference set is formed either by a fixed budget ( $K^*$ ) or an importance threshold.

B. Claim Extraction ( $C$ )

The LLM's generated output ( $G_o$ ) is decomposed into a set of atomic claims ( $C$ ) using a similar extraction process to ensure granularity and independence.

C. Evaluation Metrics

The framework defines two primary metrics:

Factual Precision ( $Prec_F$ ): The fraction of generated claims ( $C$ $C$ ) that are supported by the retrieved evidence.
- Sub-metrics: Rates for "Supported," "Not Supported," and "Contradicted" claims.
Factual Recall ( $Rec_F$ ): The fraction of facts in the reference set ( $F^*$ $F^{*}$ ) that are covered (entailed) by the generated output.
- Importance-Weighted Recall ( $Rec^w_F$ ): A weighted version that prioritizes the coverage of high-importance facts (core details) over peripheral information.

3. Key Contributions

Unified Precision-Recall Framework: A fully automatic mechanism to evaluate both the correctness (precision) and completeness (recall) of long-form generation using retrieval-grounded reference facts.
Importance-Aware Scoring: Introduction of a novel weighting scheme that distinguishes between query-specific relevance and topic-level salience, allowing for a more nuanced assessment of which facts are missing.
Comprehensive Empirical Analysis: Extensive experiments across three diverse long-form domains (Biographies, LongFact, and LongForm) revealing systematic gaps in current LLM capabilities.

4. Experimental Results

The authors evaluated 6 LLMs (including Llama-3.1, Qwen2.5, Mistral, GPT-4o-mini, and Gemini-2.5-flash-lite) across three datasets.

Precision vs. Recall Gap: Current LLMs perform substantially better on precision than on recall. Models are generally good at avoiding hallucinations but struggle to generate comprehensive answers that cover the full set of relevant facts.
Domain Dependency: Factuality varies significantly by domain. For example, in Biography generation, models showed higher contradiction rates (explicit factual errors) compared to other domains.
The "Verbosity" Trade-off:
- Longer generations do not guarantee better factuality.
- Some models (e.g., Gemini-2.5-flash-lite) achieved higher recall by generating significantly more claims (higher $\rho$ , the ratio of generated claims to reference facts), but this often came at the cost of lower precision.
- GPT-4o-mini demonstrated high efficiency, achieving near-maximum recall with a lower claim-to-fact ratio and the highest precision, suggesting better alignment with evidence.
Importance Weighting Insights:
- Models are relatively effective at covering the most important facts (high salience/relevance).
- Performance drops significantly when evaluating coverage of the full set of relevant facts.
- The combined importance score (Relevance + Salience) proved most robust for constructing reference sets, balancing the need for specific answers with core topic coverage.
Error Types: The most common failure mode is "Not Supported" claims (hallucinations without direct contradiction), rather than direct contradictions. However, contradiction rates were notably higher in biography tasks.

5. Significance and Conclusion

This paper shifts the paradigm of LLM factuality evaluation from a "precision-only" focus to a holistic view that includes completeness.

Practical Impact: It highlights that "factual incompleteness" is a major limitation of current long-form generation, distinct from hallucination.
Evaluation Standard: The proposed framework offers a more faithful assessment of model quality, particularly for applications where missing key information is as detrimental as providing false information.
Future Directions: The authors suggest improving the quality of external knowledge sources (filtering noise) and refining the automatic extraction pipelines to further enhance the reliability of these metrics.

In summary, the work argues that for long-form generation to be truly useful, models must not only be correct (high precision) but also comprehensive (high recall), with a specific emphasis on capturing the most important facts.