An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems

1. Problem Statement

Large Language Models (LLMs) are increasingly integrated into scholarly workflows for search, summarization, and question-answering (QA). However, current evaluation metrics for LLM reliability in scientific domains are predominantly automated (e.g., BLEU, ROUGE, exact match) or rely on generic benchmarks. These approaches suffer from several critical limitations:

• Lack of Contextual Nuance: Automated metrics fail to capture domain-specific requirements for precision, attribution, and logical coherence.
• Overfitting and Gaming: Benchmarks often become optimization targets rather than meaningful measures of real-world utility.
• Blind Spots: Automated systems struggle to detect subtle errors like hallucinated citations, chronological reversals in technical developments, or the conflation of distinct theoretical frameworks.
• The Gap: There is a lack of understanding regarding how domain experts (practicing scientists) actually evaluate LLM outputs, what errors they naturally identify versus miss, and what support structures they need for effective evaluation.

2. Methodology

The authors employed a two-phase, qualitative, and contextual methodology involving practicing scientists to develop and validate an error schema.

System Setup

• Architecture: A Retrieval-Augmented Generation (RAG) system was built using an open-source model (Mixtral-8x7B) to ensure interpretability and local deployment (critical for security and budget constraints in research).
• Pipeline:
  1. Document Processing: PDFs were parsed using PyMuPDF and manual correction to handle complex layouts.
  2. Retrieval: Used Sentence Transformers (All-MiniLM-L6-v2) for vector embeddings and cosine similarity. Implemented iterative query expansion using KeyBERT to handle terminology gaps.
  3. Generation: The model generated answers conditioned on retrieved context, with instructions to cite sources and avoid hallucinations.
• Data: The study utilized papers authored by the participating experts to ensure ground-truth accuracy.

Phase 1: Schema Development

• Participants: A multi-disciplinary team of 3 domain experts (15–40 years of experience), 1 NLP specialist, and 1 HCI researcher.
• Process:
  • Experts generated 68 QA pairs (single and multi-document) based on their own papers.
  • The team performed inductive thematic analysis (open and axial coding) on expert feedback.
  • Result: Identification of 49 initial error codes, consolidated into a preliminary schema.

Phase 2: Schema Validation

• Participants: 10 additional domain experts from diverse STEM fields (aeroacoustics, materials science, etc.) with 10–45 years of experience.
• Process:
  • Experts generated 12 questions each (36 total) regarding their own papers.
  • Contextual Inquiry: Experts engaged in "think-aloud" sessions while evaluating LLM outputs.
  • Two-Part Evaluation:
    1. Unprimed: Experts identified errors freely.
    2. Primed: Experts used a structured inventory based on the Phase 1 schema to check for specific error types.
• Analysis: The team compared unprimed vs. primed results to determine which errors experts naturally miss and how the schema aids detection.

3. Key Contributions: The Expert Error Schema

The primary contribution is a validated schema comprising 20 error patterns organized into 7 major categories. This schema moves beyond simple "correct/incorrect" binary labels to granular failure modes.

1. Incorrect Answers:
   • Completely Incorrect: Fails to capture central claims.
   • Partially Incorrect: High-level concepts correct, but details (parameters, equations) wrong.
   • Self-Contradiction: Internal logic errors.
   • Multimedia Interpretation: Misinterpreting figures, tables, or equations.
2. Hallucinations (Fabricated Content):
   • Terminology: Inventing or misdefining acronyms/terms.
   • Numerical Data: Inventing statistics or metrics.
   • Technical Content: Plausible but fundamentally incorrect explanations of mechanisms.
   • Citation Information: Fabricating DOIs, authors, or page numbers.
   • Document Structure: Inventing non-existent tables or page numbers.
3. Incomplete Answers:
   • Major Omissions: Missing entire sections, main points, or whole papers in multi-doc contexts.
   • Lacking Details: Missing crucial specifications (e.g., calibration parameters) required for reproducibility.
   • Chronological Gaps: Missing key timeline events.
4. Question Interpretation:
   • Misinterpretation: Understanding the prompt incorrectly (e.g., confusing "reconciling data" with "improving simulation").
   • Redirection: Answering a related but different question.
5. Synthesis Issues (Multi-document):
   • Document Linking Failure: Failing to attribute claims to the correct source paper.
   • Source Confusion: Mixing findings from different papers.
   • Disjointed Response: Failing to integrate information into a coherent narrative.
6. Formatting Issues:
   • Verbosity, notation errors, inconsistent referencing, and language issues.
7. System Failures:
   • Retrieval failures (claiming info is missing when present), LLM limitations (context window), and data structure leaks.

4. Key Results

Error Detection Patterns

• Natural vs. Guided Detection: In unprimed evaluations, experts naturally identified Incorrect Answers and Incomplete Answers (specifically major omissions). However, they frequently missed subtle hallucinations (e.g., fabricated citations, invented technical terms) and chronological errors.
• Schema Efficacy: When provided with the structured schema inventory, experts successfully detected previously overlooked issues, particularly citation hallucinations and technical content hallucinations.
• Expert Strategies: Experts employ systematic strategies including:
  • Technical Precision Testing: Cross-referencing numbers, equations, and figures.
  • Value-Based Assessment: Prioritizing honesty about limitations over plausible but unreliable answers.
  • Meta-Evaluation: Reflecting on whether their own question phrasing influenced the error.

Question Type Correlation

The study mapped 11 question types to error patterns, revealing distinct failure modes:

• Higher-Order Questions (Methodological Inquiry, Critical Evaluation): Most prone to Incorrect and Incomplete answers. Experts found the system struggled to reason about study design or critique evidence.
• Procedural Questions: High rate of Incomplete Answers (44%) and Synthesis Issues (33%), indicating difficulty in stitching together scattered steps.
• Future/Numerical Questions: Disproportionately associated with System Failures (e.g., inability to extrapolate or calculate).
• Verification Questions: Higher prevalence of Hallucinations when the model attempts to generalize beyond retrieved text.

5. Significance and Implications

For Evaluation Methodology

• Beyond Automation: The paper argues that automated metrics are insufficient for scholarly QA. A hybrid approach is necessary where automated tools handle low-level checks (e.g., citation existence) and experts handle high-level contextual reasoning.
• Schema-Driven Benchmarks: The proposed schema offers a foundation for creating "ecologically valid" benchmarks that align with expert priorities rather than arbitrary optimization targets.

For System Design

• Multimodal Comprehension: Systems must improve the semantic linking of text with tables, figures, and equations, not just extraction.
• Uncertainty Communication: Experts value systems that admit uncertainty ("I don't know") over confident hallucinations. Designing for transparent uncertainty indicators is critical for scientific integrity.
• Personalized Interfaces: The study suggests that evaluation tools should be adaptive, highlighting error types specific to an expert's blind spots (e.g., if an expert consistently misses citation errors, the tool should prompt a specific check for those).

For the Scientific Community

• Risk of Over-reliance: Even experts miss subtle errors without structured frameworks. There is a risk that early-career researchers or interdisciplinary scientists may accept plausible-sounding but incorrect LLM outputs.
• Interpretive Flexibility: The schema acknowledges that different disciplines may prioritize different error types (e.g., humanities vs. engineering), suggesting the framework should be adaptable to specific epistemological traditions.

In conclusion, this work provides a critical, expert-grounded framework for understanding the specific failure modes of LLMs in scientific contexts, moving the field from generic "hallucination" detection to a nuanced taxonomy that supports the rigorous standards of scholarly research.