Technical Summary: Precision Is Not Faithfulness

Problem Statement

Current reference-free faithfulness metrics for grounded generation suffer from a critical blind spot: they measure only precision (the proportion of claims made by a model that are supported by ground truth) but ignore recall (the proportion of relevant ground-truth facts that the model actually stated).

Because these metrics do not penalize omissions, they inadvertently reward abstention. A model can achieve near-perfect faithfulness scores by stating a single, safe fact and omitting all other relevant information. In open-domain settings, measuring recall is structurally difficult because there is no complete, enumerable set of relevant facts to serve as a denominator. Consequently, existing benchmarks cannot distinguish between a model that is highly informative and one that is merely cautious and uninformative.

Methodology

To address this, the authors introduce a complete-oracle approach, utilizing domains where strategic ground truth can be derived deterministically and exhaustively.

1. The Complete Oracle (Formula 1 & Weather)

The primary experimental domain is Formula 1 (F1) strategy explanation. Using the FastF1 API, the authors extract timing and telemetry data to derive a deterministic set of strategic events (pit stops, tire compounds, undercut/overcut attempts, on-track defenses).

• Completeness: For every decision instance, the system enumerates the full set of checkable facts that a faithful explanation must cover (e.g., specific pit lap numbers, compound changes, time gaps).
• Benchmark: A multilingual (English, Spanish, Portuguese) dataset of 7,253 decision instances spanning 150 races (2018–2025) was constructed.
• Secondary Domain: The findings were replicated in a weather forecasting domain using NOAA records, where each forecast record constitutes a complete set of enumerable facts.

2. The Metric: Precision + Recall

The authors propose a reference-free metric that decomposes model generations into atomic claims and verifies them against the structured ground truth:

• Precision (Faithfulness): The fraction of claims made by the model that are supported.
• Recall (Coverage): The fraction of the enumerable ground-truth facts that the model correctly stated.
• F1 Score: The harmonic mean of precision and recall, used to evaluate the balance between accuracy and informativeness.

3. Extraction and Verification

Two extraction backends were employed to ensure robustness:

• A dependency-free regex extractor (for English) and an LLM extractor (for multilingual support) to parse free-form text into typed claims.
• Verification was validated via controlled perturbation (injecting errors) and cross-extractor agreement (Spearman correlation of 1.00 between regex and cross-family LLM extractors at the system level).

4. Verifier-Guided Generation

A generation method was developed where a model iteratively refines its output based on feedback from the structured verifier. The feedback loop provides targeted instructions to fix contradicted claims and add uncovered ground-truth facts, improving both precision and recall without requiring reference texts.

Key Results

1. Precision Rewards Abstention; Coverage Reorders Rankings

The central finding is that the model ranking changes significantly when coverage is required.

• The "Abstention" Effect: The most precise model in the study, grok-4.3 (Precision 0.89 in Portuguese), covered only 0.46 of the relevant facts, resulting in the lowest F1 score (0.61).
• Ranking Inversion: Conversely, more verbose models like DeepSeek-V3.2 had slightly lower precision but significantly higher recall, leading to a much higher F1 score.
• Consistency: This inversion occurred across all three languages (EN, ES, PT) and in the secondary weather domain. The most precise model was never the most complete.

2. Prompting Does Not Solve the Gap

Experiments with an explicit "cover-all" prompt (instructing models to state every supportable fact) failed to close the coverage gap. In fact, mean recall decreased (from 0.60 to 0.47), and only 2 of 5 models showed any improvement. This indicates that low coverage is not merely an artifact of under-prompting but a fundamental limitation of current models in generating comprehensive grounded text.

3. Small Fine-Tuned Models Can Compete

A small, open-source model (Qwen2.5-3B) fine-tuned on grounded explanations achieved the highest F1 score (0.982) in the study, surpassing frontier models. However, the authors note this performance relies on the model faithfully reproducing deterministic templates, raising a caveat about template mimicry versus genuine reasoning.

4. Verifier-Guided Generation Improves Performance

Applying the verifier-guided self-correction method to gpt-5.4-mini increased its precision from 0.640 to 0.881, demonstrating that structured verification signals can effectively guide generation without reference texts.

Significance and Claims

The paper claims that precision is not faithfulness. By introducing a complete-oracle framework, the authors demonstrate that:

1. Reference-free faithfulness metrics are incomplete: They fail to penalize uninformative answers, leading to misleading evaluations where abstention is rewarded.
2. System rankings are unstable: Relying solely on precision reorders system comparisons, potentially promoting models that are accurate but unhelpful.
3. Coverage is measurable: In domains with deterministic ground truth, recall can be measured exactly, revealing a significant gap between precision and coverage in frontier models.
4. The problem is general: The phenomenon is not unique to F1 but appears in other data-to-text domains (weather), suggesting it is a property of precision-only metrics rather than a specific dataset issue.

The authors release the benchmark, structured annotations, the metric, baselines, and an interactive demo to facilitate research into coverage-aware evaluation. They acknowledge limitations, including that their recall metric is bounded by the specific claim ontology of their schema and that fine-tuning results may reflect template mimicry.