Assessing LLM Reliability on Temporally Recent Open-Domain Questions

1. Problem Statement

Large Language Models (LLMs) are increasingly used for open-domain question answering (QA), yet their reliability regarding temporally recent information remains underexplored. Most existing evaluations rely on static benchmarks (e.g., TriviaQA, Natural Questions) containing established facts, which fail to capture the nuances of evolving community knowledge and current events.

The core challenge is determining how well LLMs align with human perspectives on recent topics where training data may be absent or incomplete. Furthermore, there is a lack of consensus on how to evaluate these models: traditional lexical metrics (like BLEU) often penalize valid paraphrasing, while semantic metrics may obscure factual inconsistencies. This paper investigates the alignment between LLM outputs and community-derived consensus on recent Reddit questions.

2. Methodology

Dataset: RECOM (Reddit Evaluation for Correspondence of Models)

The authors introduced a novel benchmark dataset consisting of 15,000 questions sourced from the r/AskReddit subreddit from September 2025.

• Data Collection: 132,728 posts were collected, filtered by engagement (comments/replies), and top 25,000 were selected.
• Reference Construction: Instead of a single "gold" answer, the authors aggregated multiple human responses for each question. These were summarized into a single reference answer using Llama-3.1-8B via a specific summarization prompt. This approach captures diverse community perspectives rather than a single annotator's view.
• Final Set: After filtering out model refusals (e.g., "I am an AI"), the final evaluation set contained 11,515 question-response pairs.

Evaluated Models

Four open-source LLMs were tested using default inference parameters via Ollama:

1. Gemma-2-9B (9B parameters)
2. GPT-OSS-20B (20B parameters)
3. Llama-3.1-8B (8B parameters)
4. Mistral-7B (7B parameters)

Evaluation Framework

The study employed a multi-dimensional evaluation framework to assess alignment across four distinct layers:

1. Lexical Overlap: BLEU-1 to BLEU-4 and ROUGE-1/2/L F1 scores to measure surface-level word matching.
2. Semantic Similarity:
   • BERTScore: Contextual token matching using RoBERTa-large embeddings.
   • Cosine Similarity: Document-level alignment of mean-pooled embeddings.
   • MoverScore: Semantic similarity calculated via optimal transport (Earth Mover's Distance) to account for the cost of moving semantic mass between token distributions.
3. Logical Inference (NLI): Using facebook/bart-large-mnli to classify relationships between model outputs and references as Entailment, Contradiction, or Neutral.
4. Statistical Analysis: Wilcoxon signed-rank tests for significance and Cohen's d for effect sizes.

3. Key Contributions

1. Novel Dataset: Introduction of RECOM, a large-scale dataset of real-world, temporally recent questions paired with consensus-based reference answers.
2. Discovery of the Semantic-Lexical Paradox: The paper identifies a massive divergence where models achieve near-perfect semantic alignment (>99% cosine similarity) despite extremely low lexical overlap (<8% BLEU-1).
3. Scale Independence: Demonstration that model parameter count does not correlate with performance; smaller models (7B) outperformed larger models (20B).
4. Metric Granularity: Validation of MoverScore as an intermediate metric that bridges the gap between strict lexical matching and holistic embedding similarity, offering a more nuanced view of semantic alignment.

4. Key Results

The Semantic-Lexical Paradox

The most striking finding is the ~90 percentage point gap between semantic and lexical metrics:

• Cosine Similarity: All models achieved >99% similarity with references.
• BLEU-1: All models scored <8% (ranging from 0.57% to 7.58%).
• Interpretation: Models preserve meaning through extensive paraphrasing rather than lexical reproduction. High semantic similarity does not imply surface-level text matching.

Model Performance and Scale

• Ranking: Mistral-7B consistently outperformed all other models across all metrics, followed by Llama-3.1-8B.
• Scale Anomaly: GPT-OSS-20B (20B parameters) underperformed both the 7B and 8B models. This challenges the assumption that larger parameter counts automatically yield better alignment in open-domain, abstractive tasks.
• Gemma-2-9B: Trailing significantly in lexical metrics, though semantic alignment was comparable to others.

Logical Inference (NLI)

• Neutral Dominance: 86–90% of responses were classified as Neutral, meaning the model generated content topically related to the reference but not logically entailed by or contradictory to it.
• Contradiction Rates: Contradiction rates remained low (<7%) across all models, suggesting models rarely generate content that directly conflicts with human consensus, even if they do not strictly follow it.
• Entailment: Only 3.4–9.0% of responses showed entailment, indicating that models rarely reproduce the exact logical structure of the reference.

Metric Behavior

• MoverScore: Occupied an intermediate range (51–53%), reflecting the "cost" of aligning token distributions. It provided better discrimination between models than Cosine Similarity (which was near-ceiling for all).
• BLEU/ROUGE: Showed high variance and low absolute scores, confirming their unsuitability for evaluating abstractive generation where paraphrasing is the norm.

5. Significance and Implications

• Evaluation Frameworks: The study argues that relying solely on lexical metrics (BLEU/ROUGE) severely underestimates the quality of modern LLMs in open-domain QA. Evaluation must shift toward multi-dimensional frameworks that combine lexical, semantic, and logical inference metrics.
• Paraphrasing Capability: The results confirm that LLMs have mastered the ability to rephrase community consensus into semantically equivalent but lexically distinct outputs.
• Model Selection: For recent, open-ended questions, model architecture and training data composition appear more critical than raw parameter scale. Smaller, well-tuned models (like Mistral-7B) can outperform larger counterparts.
• Limitations of Alignment: High alignment with community consensus does not guarantee factual correctness. The study notes that models could align with popular but incorrect human opinions. Future work requires external knowledge verification to disentangle alignment from truth.

Conclusion

The paper concludes that LLMs are highly capable of semantic alignment on recent topics but operate via extensive paraphrasing rather than direct retrieval or copying. The "Semantic-Lexical Paradox" necessitates a rethinking of how generative AI is evaluated, moving away from surface-level text matching toward metrics that capture semantic fidelity and logical consistency. The RECOM dataset is made publicly available to facilitate further research in this domain.