Technical Summary: ACL-Verbatim

Problem Statement

Academic researchers require efficient and reliable methods to extract high-quality information from trusted sources. However, current AI-assisted research tools relying on Large Language Models (LLMs) suffer from "hallucinations"—the tendency to produce factually inaccurate, irrelevant, or nonsensical output. While Retrieval-Augmented Generation (RAG) systems attempt to ground LLMs in external documents, they often fail to eliminate hallucinations because the generative component may override retrieved evidence with its own prior knowledge. Furthermore, LLM-generated answers lack transparency and explainability, making independent fact-checking tedious and often omitted due to "algorithm appreciation." The core challenge is to enable question answering from research papers that is strictly verbatim, ensuring that answers are directly traceable to source text without fabrication.

Methodology

The paper introduces ACL-Verbatim, an application of the VerbatimRAG framework tailored for the ACL Anthology. Unlike generative RAG, this system employs an extractive approach, mapping user queries directly to verbatim text spans in retrieved documents.

Corpus and Data Pipeline

1. Data Collection: The system processes over 120,000 papers from the ACL Anthology (as of February 2026). PDFs are converted to Markdown using the docling library, preserving headers, lists, and tables while replacing figures with placeholders.
2. Segmentation: A custom chunking strategy segments papers along section boundaries, prefixing chunks with section titles to improve retrieval. Chunks are constrained between 500 and 5,000 characters and indexed for both full-text and dense vector search using IBM's granite-embedding-english-r2.
3. Query Generation: To create a training and evaluation dataset, the authors adapted the ScIRGen methodology. The pipeline:
   • Selects random papers and chunks.
   • Uses an LLM to generate synthetic question types and questions based on the chunk content.
   • Refines these into shorter, fragmented queries characteristic of real-world user behavior.
4. Annotation: Human annotators (NLP researchers) label the top 5 retrieved chunks for each query. They first determine binary relevance. For relevant chunks, they highlight specific text spans (tokens) that constitute the evidence. This process yielded a ground truth dataset of 100 query-chunk pairs (20 queries, 5 chunks each).

Model Architecture and Training

The authors evaluate three families of extractive systems:

1. LLM-based Span Extractors: Zero-shot models (Mistral, Nemotron, GLM, Qwen) prompted to return verbatim spans.
2. Pruning/Highlighting Baselines: Models like Zilliz Semantic Highlight and Provence that score or prune sentences.
3. Compact Student Model: A custom ModernBERT token classifier (150M parameters) trained on "silver supervision."
   • Silver Supervision: Generated by running the synthetic query pipeline on a larger corpus (2,000 papers) and using a strong LLM (Qwen 3.6 35B) to generate the extraction labels.
   • Architecture: A query-conditioned token classifier over an 8192-token context window, outputting binary labels for each token.
   • Training: Fine-tuned on ~10k positive and ~11k negative rows with specific post-processing rules (dropping spans <10 chars, merging nearby spans).

Key Contributions

• ACL-Verbatim Dataset: A novel ground truth dataset mapping synthetic user queries to relevant text spans in research papers, annotated by domain experts.
• Extraction Pipeline: An end-to-end system for generating synthetic queries and performing manual annotation for extractive QA in the academic domain.
• Efficient Model Performance: Demonstration that a compact, fine-tuned encoder model (ModernBERT) trained on silver data outperforms significantly larger LLMs on word-level extraction tasks.

Results

Evaluation was conducted on the 100 query-chunk benchmark using word-level precision, recall, and F1, as well as containment and coverage metrics.

• Performance: The ModernBERT student model achieved the highest Word-level F1 of 53.63, outperforming the strongest LLM extractor (GLM-5 at 48.71) and other baselines.
• Efficiency: The student model uses 3 to 4 orders of magnitude fewer parameters than the evaluated LLMs while achieving superior results.
• Precision vs. Recall: The student model demonstrated higher precision (65.43%) compared to LLMs, often abstaining from predicting spans on irrelevant chunks. In contrast, LLMs (especially with paragraph-oriented prompts) achieved higher recall but significantly lower precision, frequently extracting evidence from irrelevant chunks.
• Generalization: A multi-domain variant of the model, trained on a combination of ACL data and other benchmarks (RAGBench, Squeez), also outperformed public baselines on the ACL gold set, suggesting the approach generalizes beyond the specific domain.

Significance and Claims

The paper claims that the VerbatimRAG approach, combined with task-oriented training of extractive models, provides a blueprint for hallucination-free question answering. By strictly extracting verbatim spans rather than generating new text, the system ensures that answers are transparent and directly verifiable against the source.

The authors emphasize that while the current dataset is small (100 pairs) and the annotation process is complex, the results indicate that small, specialized models trained on synthetic data can outperform zero-shot LLMs in the critical task of evidence extraction. This offers a path toward deploying reliable, efficient, and trustworthy research assistants that mitigate the risks of hallucination inherent in generative AI. The authors note limitations regarding the dataset size and the potential for synthetic data to propagate LLM biases, suggesting that future work should focus on narrower domains to refine annotation guidelines and reduce bias.