ESOFinder: an LLM-powered tool to help users navigate ESO documentation

Technical Summary of ESOFinder: An LLM-powered Tool to Navigate ESO Documentation

Problem Statement
The European Southern Observatory (ESO) maintains a vast and heterogeneous documentation ecosystem essential for the astronomical community, spanning over 3,500 web links and more than 100 PDF manuals (exceeding 6.2 million words). This documentation covers the full observing lifecycle, from Phase 1 (proposal submission) through Phase 2 (observation execution) to Phase 3 (data archival). Locating precise, up-to-date information within this corpus is increasingly difficult for both astronomers and ESO staff. General-purpose Large Language Models (LLMs) are ill-suited for this task due to fixed training cutoffs, an inability to distinguish between current and obsolete operational policies, and a tendency to generate "hallucinated" answers. Furthermore, privacy concerns arise when sensitive or confidential internal data is processed by external commercial models.

Methodology
To address these challenges, the authors developed ESOFinder, an in-house chatbot built on a Retrieval-Augmented Generation (RAG) architecture. The system is designed to run on local servers using open-source LLMs to ensure data privacy and full control over the knowledge base.

• System Architecture: The system operates in two phases: an offline ingestion pipeline and an online inference pipeline.
  • Ingestion: Documentation (PDFs and web content) is parsed, split into overlapping chunks using a hierarchical strategy (respecting document structure), and contextually enriched. Each chunk is passed to an LLM to generate a summary situating it within the broader document context.
  • Indexing: Chunks are indexed via dual indexing: dense vector embeddings (using nomic-embed-text-v1.5 stored in ChromaDB) and keyword-based indexing (using Elasticsearch with BM25).
  • Retrieval: ESOFinder employs a hybrid retrieval strategy. It performs parallel searches using semantic vector search (for meaning and paraphrasing) and keyword search (for exact technical terms like instrument names and acronyms). Results are merged and re-ranked using a cross-encoder model (BAAI/bge-reranker-base) and a weighted scoring formula that prioritizes semantic relevance while boosting specific documentation collections based on query classification.
  • Agentic Workflow: The query-to-answer loop is implemented using LangGraph and a single local LLM (Mistral-Small-3.1-24B-Instruct-2503). The workflow includes:
    1. Query Understanding: Condensing multi-turn conversations and classifying queries into seven observatory-specific domains to adjust collection weighting.
    2. Retrieval & Grading: Executing hybrid search, merging results, and filtering out low-scoring passages.
    3. Self-Correction: If the initial retrieval fails or yields low-confidence results, the system rewrites the query and retries up to two additional times.
    4. Generation & Verification: The LLM generates a structured answer with inline citations. Two subsequent grading steps verify that the answer is factually grounded in the retrieved documents and actually addresses the user's question. If checks fail, the cycle retries; otherwise, a fallback response is provided.

Key Contributions

• Domain-Specific RAG Implementation: A robust pipeline tailored for the specific constraints of astronomical observatory documentation, integrating over 3,500 links and 100+ manuals.
• Hybrid Retrieval Strategy: A deliberate combination of semantic and keyword-based search to handle both natural language queries and precise technical terminology (e.g., instrument acronyms, OB templates).
• Self-Correcting Agentic Loop: A multi-step workflow that includes automatic query rewriting and answer verification to reduce hallucinations and improve response reliability without human intervention.
• Privacy-Preserving Deployment: The use of locally hosted open-source models ensures that sensitive internal data and proprietary documentation remain within ESO's infrastructure.

Results
The system underwent three iterative feedback exercises involving ESO staff, fellows, and students:

1. First Round (User Support Astronomers): An early version achieved an average rating of 3.0/5 stars. Issues included irrelevant answers, confusion with multi-part questions, and overly verbose responses.
2. Second Round (Data Management & Operations): A broader test yielded 3.1/5 stars. While performance improved on straightforward queries, failures persisted with large tables, missing ALMA documentation, and inappropriate source retrieval for general queries.
3. Third Round (Current Version): Incorporating architectural refinements and expanded documentation, the current version achieved an average star rating of 3.84/5 (median of 5 stars) among a representative sample of ESO users. When evaluated specifically by Data Management and Operations (DMO) staff, the average rose to 4.08/5. The distribution of qualitative feedback shifted significantly toward "Perfect answer, accurate" or "Correct answer, but missing information," with a notable reduction in "Incorrect answer" labels.

Significance and Future Directions
The paper positions ESOFinder as a critical interface for navigating ESO's complex documentation ecosystem, supporting both staff and community astronomers. The authors emphasize that the tool's iterative development, driven by user feedback, has successfully addressed initial weaknesses in retrieval coverage and response quality.

The paper modestly acknowledges remaining limitations, specifically the difficulty in retrieving information from documents containing large tables or figures, and the need for better integration of the separate ALMA documentation collection. Future work outlined by the authors includes:

• Using ESOFinder as a tool for documentation quality control to identify contradictions and outdated content.
• Exploring hierarchical multi-agent architectures to improve response speed and navigation of layered documentation.
• Integrating helpdesk ticket history (from the DeskPro system) to provide context-aware support while maintaining data privacy via local LLMs.
• Adapting the framework for Paranal operations staff and eventually making the system available to the broader ESO user community as a standard interface for documentation queries.