Technical Summary: Beyond Questions (BeQu)

Problem Statement

Current benchmarks for evaluating Large Language Model (LLM) knowledge rely heavily on predefined question-answer (QA) pairs. The authors argue this paradigm suffers from a fundamental availability bias: it can only evaluate knowledge that benchmark designers explicitly choose to query. Consequently, facts outside the scope of these questions remain invisible, and high performance on such benchmarks does not necessarily imply broad, general knowledge. Furthermore, existing QA benchmarks face issues of saturation, data leakage, and prompt sensitivity. While recent work on "knowledge materialization" attempts to extract knowledge via open-ended prompts, there is no systematic framework to evaluate the reliability, precision, and recall of such open-ended knowledge elicitation.

Methodology: The BeQu Benchmark

The paper introduces Open Knowledge Evaluation, a paradigm that shifts focus from retrieving answers to fixed queries to characterizing the knowledge models naturally surface in response to open-ended prompts (e.g., "Tell me everything you know about X"). This is instantiated through BeQu (Beyond Questions), a benchmark comprising 10,000 entities and 6 GB of reference corpora.

Benchmark Construction

1. Entity Selection: Three subsets are defined:
   • Random: 10,000 Wikipedia articles filtered for length, Wikidata statement count, and non-ambiguity, then scored by an LLM judge (Llama 4 Scout) on informativeness and suitability.
   • Domains: 100 entities per each of 10 domains (e.g., Person, Organization, Event).
   • Non-existent: 10 manually created fake entities to test hallucination.
2. Reference Corpus Construction: For each entity, a reference corpus is built from the full Wikipedia article and up to 20 web documents (via Brave Search API). An LLM extracts factual triples (subject, predicate, object) from these sources to serve as the ground truth.
3. Evaluation Protocol: The evaluation treats the model's output as a set of elicited triples and compares them against the reference corpus using Textual Entailment (Natural Language Inference) via an LLM judge.
   • Precision: Measures the fraction of model-generated triples that are entailed by the reference corpus.
   • Recall: Measures the fraction of reference triples that are entailed by the model's generated triples.
   • Metrics: Entailment Precision, Recall, and F1-Score.

Key Contributions

1. Paradigm Shift: Introduction of Open Knowledge Evaluation, moving beyond predefined QA pairs to assess knowledge expression via open-ended elicitation.
2. BeQu Benchmark: A large-scale benchmark with 10,000 entities and reference corpora designed to measure both precision and recall of generated knowledge.
3. Empirical Analysis: A comprehensive study across 20 models (commercial and open-source) analyzing the effects of reasoning effort, model scale, prompt format, and knowledge domain.

Experimental Results

1. Model Ranking and Scaling

• Performance: Commercial models generally dominate the top rankings (e.g., Claude Opus 4.6, Kimi K2.5, Gemini 3 Flash), though several open-weight models come remarkably close.
• Scaling: F1 scores generally increase with model size within families (e.g., Anthropic, OpenAI, Google). However, scaling effects vary: larger models often show higher recall but sometimes lower precision.
• Precision-Recall Frontier: A clear trade-off exists; no model simultaneously achieves high precision and high recall. Models cluster into "high-precision/low-recall" (e.g., GPT-5 Nano) or "high-recall/moderate-precision" (e.g., Claude Opus 4.6) groups. The leading F1 scores are driven primarily by recall, suggesting the limiting factor is often the model's failure to assert known facts rather than asserting incorrect ones.

2. Reasoning Effort

• Minimal Impact: Varying reasoning effort (low, medium, high) in models like Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro resulted in negligible changes to overall F1 scores (variations < 0.05). This suggests factual knowledge is largely encoded and retrieved without requiring explicit, effortful reasoning steps.

3. Domain and Hallucination

• Domain Variance: Performance varies by domain. "Event" and "Scientific Concept" yielded the highest scores, while "Person" was the weakest for all tested models, driven primarily by low recall.
• Hallucination: On non-existent entities, only GPT-5.4 completely abstained from generating triples. Other models (DeepSeek V3.2, Llama 4 Scout) hallucinated significant numbers of triples.

4. Prompt Format and Constraints

• Schema Constraints: Enforcing standardized output schemas (e.g., Wikidata, Schema.org) substantially improved precision but drastically collapsed recall.
• Open-ended vs. Constrained: Open-ended prompts yielded the best overall F1 scores.
• Predicate Lists: Predefined predicate lists were the dominant factor in the precision-recall trade-off; removing them recovered substantial recall at a modest precision cost.

5. Triple Range

• Diminishing Returns: Increasing the requested number of triples per entity improved recall and F1 up to a point (e.g., GPT-5.4 peaked at 75–150 triples), after which performance plateaued or degraded due to hallucinations. This indicates models possess latent knowledge not fully expressed when constrained to small output volumes.

Significance and Claims

The paper claims that BeQu provides a necessary complement to existing QA-based benchmarks by addressing availability bias and measuring the full spectrum of knowledge expression (both what models know and what they choose to say).

Key takeaways emphasized by the authors include:

• Benchmark Durability: Open-ended knowledge generation is far from saturated, and open-source models are competitive with commercial ones in this paradigm.
• The "Unreasonable" Task: Unlike other tasks, explicit reasoning effort offers negligible benefits for open-ended knowledge expression.
• Trade-offs: While schema enforcement boosts precision, it significantly harms recall, and explicit attempts to trade precision for recall via prompting have limited effect.
• Standardization: The authors propose that open knowledge evaluation should become a standard dimension of LLM assessment alongside conventional QA.

The authors acknowledge limitations, primarily the reliance on an LLM-as-a-judge (Llama 4 Scout) for entailment, which may propagate systematic biases, and the use of sampled subsets (200 entities) for most experiments due to computational constraints.