chATLAS: A Retrieval-Augmented Conversational Assistant for the ATLAS Experiment

Technical Summary of "chATLAS: A Retrieval-Augmented Conversational Assistant for the ATLAS Experiment"

Problem Statement
Large-scale scientific collaborations, such as the ATLAS experiment at CERN, generate vast amounts of internal documentation (technical notes, meeting minutes, software guides, wiki pages) spread across heterogeneous systems. This fragmentation creates a significant barrier to knowledge discovery. Information is often inconsistently maintained, written in diverse styles, and difficult to search effectively. Consequently, collaborators face steep learning curves, and experienced members waste time rediscovering existing knowledge. While general-purpose Large Language Models (LLMs) offer conversational interfaces, they lack access to this proprietary, internal documentation, leading to hallucinations or irrelevant responses when applied to domain-specific queries.

Methodology
The authors present chATLAS, a deployed Retrieval-Augmented Generation (RAG) system designed to provide conversational access to over 30,000 documents from the ATLAS knowledge base. The system architecture and methodology are defined by the following components:

• Data Ingestion and Preprocessing: The system aggregates data from multiple sources, including TWiki (over 35,000 pages, filtered for recent activity), MkDocs (GitLab-hosted Markdown), CERN Document Server (CDS), Indico (lectures), and ATLAS Talk (forum threads). Documents are processed into a parent-child hierarchical structure: full documents (parents) are split into smaller chunks (children) for retrieval. This allows the system to retrieve specific relevant segments while passing the full parent document to the generator for broader context.
• Embedding and Fine-Tuning: A domain-adapted embedding model is central to the retrieval process. The base model, sentence-transformers/all-mpnet-base-v2, was fine-tuned using a dataset of 12,000 question-answer pairs. These pairs consist of real user interactions from ATLAS Talk and synthetic pairs generated from other documentation sources. This fine-tuning optimizes the model for ATLAS-specific terminology, acronyms, and jargon.
• Hybrid Retrieval: chATLAS employs a hybrid retrieval strategy combining:
  1. Dense Retrieval: Vector similarity search (cosine similarity) using pgvector with Hierarchical Navigable Small World (HNSW) or Inverted File with Flat Compression (IVFFlat) indices.
  2. Sparse Retrieval: Keyword-based search using PostgreSQL's full-text search (TF-IDF/BM25 style).
  • Ranking: Results from both retrievers are merged using Reciprocal Rank Fusion (RRF) to balance the strengths of semantic and keyword matching.
• Contextual Retrieval: To address the difficulty of interpreting isolated technical chunks (particularly in TWiki), the system implements contextual retrieval. An LLM generates a concise, document-level context description for each chunk, which is embedded alongside the chunk itself. This enriches the vector space without altering the embedding model.
• Query Processing and Safety: Incoming queries undergo intent routing and safety screening via a lightweight LLM. This step filters out-of-scope or dangerous requests, corrects spelling, and resolves conversational references (e.g., pronouns) into self-contained queries before retrieval.
• Generation: The final answer is generated by an LLM conditioned on the retrieved passages. The system supports various models (e.g., gpt-oss-20b, gpt-5-mini) and is designed to be provider-agnostic, with a focus on locally deployable open-source models in the future.
• Evaluation Framework: A custom benchmark, chATLAS_Benchmark, was developed using automatically generated question-answer pairs from the target corpus. It evaluates retrieval accuracy (Top-k) and generation correctness (using an LLM-as-a-judge metric) to guide design choices.

Key Contributions and Results
The paper details several technical insights and performance improvements derived from the development of chATLAS:

1. Domain-Specific Fine-Tuning: Fine-tuning the embedding model on ATLAS-specific data significantly improved retrieval performance. When combined with contextual retrieval, this approach more than doubled the frequency with which the correct document was returned as the top result (Top-1 accuracy) for the most challenging source (TWiki) compared to a baseline model.
2. Contextual Retrieval Efficacy: The study demonstrated that adding document-level context to chunks (contextual retrieval) provided a larger performance gain than embedding fine-tuning alone. The combination of both techniques yielded the highest retrieval accuracy across all documentation sources.
3. Hybrid Retrieval Superiority: Merging dense and sparse retrieval results via RRF improved the weighted document-match score from approximately 50% (baseline) to 65%. While a neural cross-encoder re-ranker could push this to 71%, the computational cost was deemed too high for interactive use, leading to the adoption of RRF as the optimal balance.
4. Cross-Source Generalization: Models fine-tuned on specific sources (e.g., GitLab) showed strong performance on those sources but also retained significant gains when applied to other sources (e.g., TWiki), suggesting the model learned generalizable domain patterns.
5. Deployment and Integration: chATLAS is a fully operational system running on CERN's PaaS infrastructure. It integrates with Mattermost for direct chat interaction and includes an MCP (Model Context Protocol) server to facilitate future agentic workflows and third-party tool integration.

Significance and Claims
The paper claims that chATLAS demonstrates the viability of domain-adapted RAG for large scientific collaborations. Its primary significance lies in providing source-attributed, domain-specific access to proprietary documentation that is otherwise inaccessible to general-purpose LLMs.

The authors emphasize that the system is not a replacement for existing documentation but a unifying interface that flattens the learning curve for new members and improves efficiency for established collaborators. By grounding responses in a controlled corpus, the system ensures responses are trustworthy and verifiable. The work offers a practical foundation for other organizations relying on accurate retrieval from fragmented, proprietary documentation, showcasing a modular architecture that can evolve with new models and data sources without requiring a complete system rebuild. The paper concludes that such systems provide durable infrastructure for AI-assisted scientific work while offering transferable skills to the student developers involved.