DS@GT ARC at LongEval: Citation Integrity and Factual Grounding in Scientific QA

Technical Summary: DS@GT ARC at LongEval

Problem Statement
Scientific knowledge is dynamic, evolving through continuous research and publication. This temporal drift poses significant challenges for Retrieval-Augmented Generation (RAG) systems, which risk retrieving outdated or irrelevant information, missing evidence, or generating answers that are not factually grounded in the provided source documents. While RAG is theoretically ideal for scientific question answering (QA) by leveraging external documents, current systems often suffer from "retrieval-related mistakes," including the inclusion of distractor documents and the generation of ungrounded claims. A critical gap exists between traditional natural language evaluation metrics (which favor fluency and lexical overlap) and the requirement for strict citation integrity and factual grounding in scientific domains.

Methodology
The DS@GT ARC team participated in the CLEF 2026 LongEval Task 4, which utilized the CORE scientific literature corpus. The task provided a query and a fixed set of ten pre-retrieved documents, requiring participants to generate natural language answers supported by specific citations from that set. The team evaluated three distinct architectural approaches:

1. DS@GT Hybrid Baseline: A standard RAG implementation using a hybrid retriever (BM25 + BGE dense embeddings) to rank overlapping semantic chunks. The top-$k$ chunks were fed directly into a Gemma-4-31B instruction-tuned model for answer synthesis.
2. Corrective RAG (CRAG) + CiteFix Pipeline: A two-stage corrective architecture designed to enforce grounding:
   • Pre-Generation: A lightweight cross-encoder (CRAG) evaluates retrieved chunks against the query, dropping those that do not meet an entailment threshold to filter out distractors before generation.
   • Post-Generation: CiteFix parses the generated claims against the cited document chunks. Citations that lack entailment to support the specific claims are pruned, enforcing strict attribution.
3. Frontier Model Benchmarks: Two manually curated runs using state-of-the-art models (GPT-5.5 Thinking and Claude Opus 4.7 Thinking) that bypassed chunking and scoring, synthesizing answers directly from the raw text of the ten candidate documents.

Evaluation Strategy
The team employed a dual-evaluation framework:

• Official Task Metrics: Standard metrics including ROUGE and BERT scores for lexical/semantic similarity against a hidden gold set, Citation Precision, and Nugget Coverage.
• Reference-Free RAGAs Diagnostics: An LLM-as-judge setup (using Claude Sonnet 4.6) to measure Global Faithfulness (grounding of claims against the entire 10-document context), Citation Faithfulness (grounding of claims strictly against cited documents), and Answer Relevancy.

Key Results
The evaluation revealed a significant divergence between traditional performance metrics and factual grounding:

• Frontier Models: GPT-5.5 and Claude Opus 4.7 achieved the highest scores on official metrics (ROUGE, BERT, Answer Relevancy) and Citation Precision. However, RAGAs diagnostics exposed a critical failure mode: these models frequently generated highly relevant answers by relying on their parametric memory rather than the provided context. Consequently, they exhibited low Citation Faithfulness (e.g., GPT-5.5 scored 0.417), indicating they often hallucinated citations or failed to ground specific details in the retrieved documents.
• Corrective Pipeline: The CRAG+CiteFix pipeline achieved the highest Global Faithfulness (0.784) and Citation Faithfulness (0.758) among all submissions. However, this strict adherence to grounding resulted in lower ROUGE and BERT scores compared to the Hybrid Baseline. This drop was attributed to the pipeline's "abstention behavior," where the system refused to generate answers when the retrieved context lacked sufficient evidence, rather than hallucinating from parametric memory.
• Statistical Significance: A Wilcoxon signed-rank test on the RAGAs scores indicated that the improvements in faithfulness provided by the corrective interventions were not statistically significant across the 47 test queries, suggesting potential limitations in the NLI models used for verification or the specific tuning of the interventions.

Significance and Conclusions
The paper argues that the evaluation of trustworthy RAG QA systems requires a shift in metric design. The results demonstrate that frontier models can maximize answer relevance and fluency while failing to utilize the provided context, a failure mode that traditional metrics (ROUGE/BERT) fail to penalize. Conversely, pipelines that enforce strict grounding may appear to underperform on lexical overlap metrics due to safer abstention behaviors.

The authors conclude that securing RAG QA pipelines in scientific domains necessitates evaluation metrics that reward structural safety and proper failure modes (such as abstention) over conversational fluency that relies on parametric memory. They propose that future benchmarks must prioritize metrics that enforce strict answer grounding to ensure that scientific claims are empirically backed by the cited research documents.