Retrieval-Augmented Question Answering over Scientific Literature for the Electron-Ion Collider

This paper presents a locally deployed, cost-effective Retrieval-Augmented Generation (RAG) system utilizing an open-source LLaMA model and an in-house arXiv database to provide secure, domain-specific question answering for the Electron-Ion Collider (EIC) experiment while ensuring data privacy.

The Gist

Imagine you are a new student joining a massive, world-famous research team called the Electron-Ion Collider (EIC). This team is like a giant library of thousands of books, but instead of stories, the books are filled with incredibly complex, technical science about smashing particles together.

Your goal is to ask questions like, "How does this specific detector work?" or "What was the main finding of that 2023 experiment?"

In the past, if you asked a standard AI (like a chatbot), it might try to answer from memory. But because the AI hasn't read every specific EIC paper, it might confidently make up a wrong answer. This is called a "hallucination"—like a student guessing the answer on a test because they forgot to study, but sounding very sure of themselves.

This paper describes a new, smarter way to build a "Science Assistant" for the EIC team that avoids guessing. Here is how it works, broken down into simple analogies:

1. The Problem: The "Confident Guess"

Standard AI models are like students who have read the entire internet but haven't memorized the specific, secret textbooks of the EIC team. When asked a hard question, they might invent facts to sound smart. The researchers wanted a system that says, "I don't know unless I can find the exact page in the book."

2. The Solution: The "Librarian with a Highlighter" (RAG)

The team built a system called RAG (Retrieval-Augmented Generation). Think of it as a super-smart librarian who never guesses.

• The Library: Instead of the AI memorizing everything, the team built a local, private digital library containing 178 specific scientific papers from a public archive called arXiv.
• The Process:
  1. You ask a question.
  2. The Librarian (Retrieval): The system immediately scans the library, finds the exact pages (chunks of text) that talk about your question, and highlights them.
  3. The Writer (Generation): It hands those highlighted pages to an AI writer (an open-source model called LLaMA) and says, "Write an answer only using what is written on these pages."
  4. The Citation: The system then points you to the exact book and page number so you can verify it yourself.

3. The "Local" Advantage: Why keep it in the basement?

Most companies use expensive, cloud-based AI services that send your data to the internet. The EIC team is working on pre-publication data (secrets they haven't shared with the world yet).

• The Analogy: Imagine you are working on a top-secret recipe. You wouldn't want to email the recipe to a cloud server where anyone could see it.
• The Fix: This new system runs locally (on their own computers). It's like having a private, locked room where the AI and the books live. No data leaves the building. It's also cheaper because they aren't paying for expensive cloud subscriptions.

4. The Experiments: Testing the "Chunks" and the "Brain"

The researchers tried different settings to see what worked best, like tuning a radio for the clearest signal.

• Cutting the Books (Chunking): They had to cut the long PDF papers into smaller pieces (chunks) so the AI could read them.
  • Small pieces (120 characters): Like reading a single sentence. It's precise, but you might miss the context of the whole paragraph.
  • Larger pieces (180 characters): Like reading a full paragraph. The researchers found that larger chunks worked better because the AI understood the "story" of the sentence better.
• Finding the Right Pages (Similarity Search): They tested two ways to find the right pages:
  • Cosine Similarity: Looking for words that match closely.
  • MMR (Maximum Marginal Relevance): Looking for matches but also making sure the results aren't all the same (avoiding redundancy).
  • Result: Both worked about the same speed, but the larger chunks were the real winner.
• The Brain Power (LLaMA Models): They tested two versions of the AI writer: LLaMA 3.2 and LLaMA 3.3.
  • The Result: The bigger model (3.3) was like a genius professor who took 100 seconds to think and sometimes got distracted. The smaller model (3.2) was like a fast, reliable assistant who answered in 15 seconds. For a chatbot, speed and reliability won out over raw "genius."

5. The Verdict: A Trustworthy Tool

The team tested their new assistant against a "Gold Standard" set of questions created by human experts.

• Success: The system was very good at finding the right information (high "Context Recall"). It rarely made things up because it was forced to stick to the text.
• Limitation: Sometimes the answers weren't perfectly factually accurate (scoring lower on "Answer Correctness"). This is likely because the AI model they used is "lightweight" (smaller) to keep it fast and local, and the science is just very, very hard.

Summary

This paper is about building a secure, private, and cost-effective "Science Librarian" for the Electron-Ion Collider team. Instead of letting an AI guess answers from the internet, they gave it a specific, locked library of their own papers and told it to only answer based on what it finds there.

It's a step toward making AI a helpful, honest research partner that respects privacy and doesn't make things up, ensuring that when scientists ask complex questions, they get answers backed by real evidence.

Technical Summary

1. Problem Statement

The Electron-Ion Collider (EIC) is a massive international scientific collaboration involving over 190 institutes, generating vast amounts of specialized technical literature (arXiv articles, technical design reports, etc.). Researchers and new collaborators face challenges in efficiently extracting specific scientific objectives, simulation details, and hardware specifications from this unstructured data.

While Large Language Models (LLMs) offer potential for automated Q&A, they suffer from hallucinations (generating fluent but factually incorrect information) and data privacy concerns when using proprietary, cloud-hosted models. Previous attempts to build an EIC Q&A system relied on proprietary models (OpenAI GPT-4.5) and cloud-based vector databases (Pinecone), which posed cost, privacy, and data sovereignty issues. The authors aimed to develop a cost-effective, locally deployable, and privacy-preserving alternative that grounds answers in specific scientific citations.

2. Methodology and Application Architecture

The authors developed a Retrieval-Augmented Generation (RAG) pipeline using entirely open-source components and local deployment. The architecture consists of five key stages:

• Data Ingestion & Pre-processing:

  • Source: 178 EIC-related research articles from the arXiv repository (covering phenomenology, detector design, accelerator physics, etc.).
  • Chunking: Documents are split into fixed-size segments using RecursiveCharacterTextSplitter. Two configurations were tested: 120 characters and 180 characters, with a 20-character overlap to maintain semantic continuity.
  • Metadata Enrichment: Crucially, metadata (arXiv ID, authors, year) is concatenated with the text chunks before embedding to improve citation accuracy and source differentiation.

• Embedding & Vector Storage:

  • Model: mxbai-embed-large (a transformer-based model by Mixedbread AI) is used to generate 1024-dimensional dense vectors. This model was chosen for its strong MTEB benchmark performance and lack of API dependency.
  • Database: ChromaDB was selected over FAISS, Pinecone, and LanceDB due to its seamless integration with LangChain, local deployment capabilities (ensuring data privacy), and ease of use.

• Retrieval Strategy:

  • User queries are encoded into 1024-dimensional vectors.
  • Two similarity search strategies were compared:
    1. Cosine Similarity: Measures angular alignment.
    2. Maximum Marginal Relevance (MMR): Balances relevance with diversity to reduce redundancy.
  • The top 20 retrieved chunks are selected.

• Answer Generation:

  • Model: The system utilizes locally deployed LLaMA 3.2 and LLaMA 3.3 models.
  • Prompting: A prompt template provides "guardrails" to restrict the model to generating answers only based on the retrieved context, minimizing hallucinations.
  • Observability: LangSmith is used to trace the entire pipeline (query, retrieval, prompt, answer) and link generated answers back to specific arXiv citations for verification.

• Evaluation Framework:

  • The system was evaluated against a "gold standard" dataset of 51 expert-curated questions (generated via GPT-4.0 and vetted by humans).
  • Metrics: The RAGAS framework was used to compute six metrics:
    • Retrieval: Context Entity Recall, Context Precision, Context Recall.
    • Generation: Answer Relevance, Answer Correctness, Faithfulness.

3. Key Contributions

• Local, Privacy-Preserving RAG System: Demonstrated a fully local RAG pipeline for high-stakes scientific domains, eliminating the need to send pre-publication data to public cloud APIs.
• Open-Source Stack: Replaced proprietary models and cloud databases with open-source alternatives (mxbai-embed-large, ChromaDB, LLaMA), offering a cost-effective solution for resource-constrained scientific collaborations.
• Citation Traceability: Implemented a robust mechanism using LangSmith to trace answers directly to source arXiv articles, enhancing scientific rigor and trust.
• Empirical Analysis of Chunking and Retrieval: Provided a comparative analysis of chunk sizes (120 vs. 180) and retrieval metrics (Cosine vs. MMR) specifically for scientific literature.

4. Results and Analysis

• Latency Performance:
  • Retrieval: Chunk size and similarity metrics (Cosine vs. MMR) had negligible impact on retrieval latency (median ~0.11s).
  • Generation: The choice of LLM had a massive impact. LLaMA 3.2 showed stable latency (median 10–20s), whereas LLaMA 3.3 exhibited an order-of-magnitude increase in latency with high variability and extreme outliers, making it unsuitable for interactive chatbots under current compute constraints.
• Retrieval Quality:
  • Context Recall was robust (scores near 1.0), indicating the system successfully retrieved the necessary claims from the ground truth.
  • Context Entity Recall was lower, highlighting a limitation: dense embedding models optimized for general semantics struggle to extract specific scientific named entities.
• Generation Quality:
  • Chunk Size: The 180-character chunk size consistently outperformed the 120-character size. It yielded higher Faithfulness (reduced hallucination) and Answer Relevance scores, suggesting larger chunks preserve better semantic coherence for complex physics topics.
  • Similarity Metric: MMR did not show a significant advantage over Cosine Similarity in this specific domain.
  • Answer Correctness: Scores were generally low across all configurations. The authors attribute this to the complexity of EIC-specific factual details and the limitations of the lightweight LLaMA 3.2 model in handling deep technical reasoning.

5. Significance and Future Work

This work validates the feasibility of deploying secure, local RAG systems for large-scale scientific collaborations. It offers a blueprint for institutions to build domain-specific AI assistants without compromising data privacy or relying on expensive proprietary APIs.

Future Directions:

• Knowledge Base Expansion: Incorporating heterogeneous data sources such as PowerPoint presentations, wikis, white papers, and technical reports.
• Pipeline Orchestration: Migrating from LangChain to the LangGraph framework for more advanced state management and workflow control.
• Model Scaling: Investigating larger models if compute resources become available to improve "Answer Correctness" and reasoning capabilities.