MITRA: An AI Assistant for Knowledge Retrieval in Physics Collaborations

The paper introduces MITRA, an on-premise Retrieval-Augmented Generation (RAG) system designed to enhance knowledge retrieval in large-scale physics collaborations like CMS by employing a novel automated pipeline for document processing and a two-tiered vector database architecture to accurately answer context-aware questions while ensuring data privacy.

The Gist

Imagine you are walking into the world's largest, most chaotic library. This library doesn't just have books; it has millions of them, written by thousands of different authors who are all working on different, highly specific projects. Some authors are writing about black holes, others about tiny particles, and some about how to fix a specific type of microscope.

The problem? The library is so big that finding the exact page you need to solve a specific problem takes days. If you ask a librarian, "How do I fix the microscope?" they might hand you a book about black holes because you didn't use the exact right words.

MITRA is the solution to this problem. Think of MITRA as a super-smart, private research assistant who has read every single document in this library and is ready to chat with you.

Here is how MITRA works, broken down into simple concepts:

1. The "Friend" Who Knows Everything (The Core Idea)

The name "MITRA" comes from a Sanskrit word meaning "friend." The goal is to create an AI friend that helps physicists find answers quickly. Instead of forcing you to search for specific keywords (like "transverse momentum"), you can just ask a natural question like, "What are the rules for stopping the particle from moving sideways?" MITRA understands the meaning of your question, not just the words.

2. The Secret Library (Privacy First)

Most AI assistants today are like sending your questions to a giant, public cloud server run by a big tech company. That's risky for scientists because their research data is often secret and sensitive.

MITRA is different. It lives entirely inside the scientists' own building (on their local servers).

• Analogy: Imagine a private tutor who comes to your house, reads your private notes, and answers your questions, but never leaves your house or tells anyone else what you discussed. This ensures no secret data ever leaks out.

3. The Two-Step Detective Process (How it Finds Answers)

One of the biggest challenges is that the library is full of different "stories." If you ask, "What is the biggest danger here?" the answer is totally different if you are talking about a black hole experiment versus a dark matter experiment.

MITRA uses a clever two-step detective strategy:

• Step 1: The "Table of Contents" Check. First, MITRA looks at the short summaries (abstracts) of all the documents to figure out which specific project you are talking about. It's like asking, "Are we talking about the Black Hole book or the Dark Matter book?" It asks you to confirm, "Are you sure you mean the Dark Matter project?"
• Step 2: The Deep Dive. Once you confirm the project, MITRA "locks on" to just that one book. It ignores everything else in the library and dives deep into that specific text to find the answer. This prevents it from getting confused and mixing up facts from different experiments.

4. The "Smart Scanner" (Reading the Documents)

Scientific documents are messy. They have charts, graphs, footnotes, and weird layouts. Standard computer programs often get confused by these and read the page numbers as part of the text.

MITRA uses a high-tech "scanner" (called OCR) that is like a human eye with a magnifying glass. It can look at a page, understand that "Figure 3" is a picture and not text, and separate the main story from the footnotes. This ensures the AI learns from the real content, not the garbage data.

5. Why It's Better Than a Standard Search Engine

The authors tested MITRA against a standard search engine (like a super-charged Google).

• The Old Way: If you asked, "How do we stop the sideways movement?" and the document said "pT cut," the old search engine might say, "I don't know, I can't find those words."
• The MITRA Way: It understands that "stopping sideways movement" means "pT cut." It finds the right answer even if you use different words.

The Bottom Line

MITRA is a privacy-safe, super-smart librarian designed for the world's biggest science projects. It helps new students get up to speed quickly and helps experts find answers without wasting hours searching through mountains of paperwork.

The ultimate dream? To turn MITRA from a simple "question-answering" tool into a proactive research partner that can say, "Hey, I noticed your data looks a bit weird compared to the simulations. Here are three possible reasons why," effectively acting as a co-pilot for scientific discovery.

Technical Summary

Here is a detailed technical summary of the paper "MITRA: An AI Assistant for Knowledge Retrieval in Physics Collaborations."

1. Problem Statement

Large-scale scientific collaborations, such as the Compact Muon Solenoid (CMS) at CERN, generate a massive, ever-growing corpus of internal documentation (analysis notes, wikis, guidelines). This creates significant challenges:

• Information Overload: Researchers (both new and experienced) struggle to navigate terabytes of data to find precise information, hindering knowledge sharing and slowing discovery.
• Limitations of Traditional Search: Standard keyword-based search tools (e.g., BM25) fail to capture semantic context. They rely heavily on exact phrasing matches, making them ineffective when users employ synonyms or paraphrase concepts (e.g., asking for "transverse momentum requirement" when the text uses "pT cut").
• Privacy and Cost Concerns: Existing AI solutions often rely on external API-based services (e.g., OpenAI). While cost-effective for single queries, these models incur substantial cumulative costs for large collaborations and pose a critical risk: proprietary, unpublished research data may leave the secure local network.

2. Methodology: The MITRA Pipeline

MITRA is a Retrieval-Augmented Generation (RAG) system designed to answer context-aware questions about physics analyses. It operates on a fully on-premise infrastructure to ensure data privacy.

A. Automated Document Acquisition & Extraction

• Acquisition: Uses Selenium to automate browser interactions, logging into internal databases and downloading analysis notes (typically PDFs).
• Extraction: Instead of standard PDF-to-text libraries (which struggle with complex layouts), MITRA employs advanced Optical Character Recognition (OCR) engines (Surya and Tesseract). This ensures high-fidelity text extraction that preserves the distinction between main content, figure captions, tables, and page numbers.

B. Two-Tiered Database Architecture

To resolve the ambiguity where different analyses might have conflicting answers to similar questions (e.g., "What is the background?" varies by analysis), MITRA uses a unique two-tier approach:

1. Abstracts Database (Tier 1): Contains only abstracts of all analyses. When a user queries, the system first searches this tier to identify the single most relevant analysis. The user confirms this choice (human-in-the-loop), establishing the correct context.
2. Full-Text Database (Tier 2): Once an analysis is selected, the system "locks on" to a dedicated vector database containing the full, chunked text of only that specific analysis. All subsequent RAG operations occur within this isolated context to prevent cross-analysis confusion.

C. Retrieval and Ranking Pipeline

• Embedding: Document chunks (paragraphs) are encoded into 768-dimensional vectors using the facebook/dpr-question_encoder-multiset-base model.
• Vector Storage: Embeddings are stored in Chroma DB.
• Two-Stage Retrieval:
  1. Dense Retrieval: A similarity search (Cosine Similarity) retrieves the top-$k$ candidates.
  2. Reranking: A cross-encoder model (cross-encoder/ms-marco-MiniLM-L-6-v2) re-ranks these candidates for higher accuracy. This balances computational efficiency with precision.
• Generation: The final ranked passages serve as context for the Large Language Model (LLM), which is explicitly prompted to ground answers strictly in the retrieved text to mitigate hallucinations.

D. On-Premise Deployment

• Hardware: Hosted on local collaboration GPU servers (e.g., NVIDIA Tesla T4).
• LLM: Uses a 4-bit quantized Mistral-7B model served via Ollama and integrated with LangChain.
• Benefits: Eliminates per-token API costs and guarantees that no proprietary data leaves the local secure network.

3. Key Contributions

1. Automated Ingestion Pipeline: A modular system using Selenium and advanced OCR for robust, version-aware document processing.
2. Context-Aware Database Design: A novel two-tiered architecture that isolates specific analysis contexts, preventing the model from conflating information across different physics searches.
3. Privacy-First Deployment: A fully local framework that avoids external API dependencies, ensuring data sovereignty and cost sustainability for large collaborations.

4. Performance Evaluation

The system was evaluated against a standard keyword baseline (Okapi BM25) using two query sets:

• Set 1: Exact keyword matches (source phrasing).
• Set 2: Realistic paraphrased queries (synonyms/conceptual rephrasing).

Key Results:

• Exact Matches (Set 1): Both MITRA and BM25 performed well, with BM25 slightly edging out on Precision@5 due to exact term matching.
• Paraphrased Queries (Set 2): MITRA demonstrated a significant advantage.
  • Precision@1: MITRA achieved 0.75 vs. BM25's 0.13.
  • Mean Reciprocal Rank (MRR): MITRA scored 0.81 vs. BM25's 0.35.
  • NDCG@5: MITRA scored 0.88 vs. BM25's 0.59.
• Hallucination Resistance: Qualitative testing showed MITRA correctly identified when a query was out-of-context (e.g., asking about Higgs bosons in a dark matter search) and refused to hallucinate, instead informing the user of the mismatch.

5. Significance and Future Directions

• Scientific Impact: MITRA addresses the critical bottleneck of information retrieval in high-energy physics, potentially accelerating the onboarding of new researchers and the synthesis of complex analyses.
• Scalability: The system is designed to scale to thousands of users by leveraging existing institutional hardware, avoiding the prohibitive costs of commercial APIs.
• Future Work:
  • Expanding the knowledge base to include diverse document types.
  • Implementing formal user studies and "LLM-as-a-judge" metrics for generation quality.
  • Deploying on production-grade GPU clusters for high concurrency.
  • Evolving from a Q&A tool into a proactive research agent capable of summarizing updates, comparing methodologies, and identifying gaps in search spaces.

In conclusion, MITRA represents a robust, privacy-preserving solution for managing the "knowledge mountain" of modern scientific collaborations, proving that semantic search significantly outperforms traditional keyword methods in complex, domain-specific environments.