Technical Summary: AstroRAG - A PageRank-Based Retrieval-Augmented Generation Pipeline for Astronomy QA

1. Problem Statement

Large Language Models (LLMs) exhibit strong natural language processing capabilities but frequently generate factual errors when relying solely on parametric knowledge. While Retrieval-Augmented Generation (RAG) mitigates this by grounding responses in external evidence, conventional "retrieve-and-dump" pipelines often introduce irrelevant context. This is particularly detrimental in specialized domains like astronomy, where dense technical terminology, mathematical formalism, and precise contextual grounding are required.

Furthermore, recent studies highlight a "U-shaped positional effect" in long-context models, where accuracy drops significantly when key evidence is placed in the middle of a prompt rather than at the beginning or end. Existing domain-specific solutions often struggle with handling equation-rich content, bridging the gap between general and specialized vocabularies, and ensuring traceability and privacy in scientific reasoning.

2. Methodology: The AstroRAG Pipeline

AstroRAG is a lightweight, fully local, and training-free RAG framework designed specifically for astronomy question answering. It operates on a per-instance basis to ensure privacy and prevent cross-task leakage. The pipeline consists of the following stages:

A. Instance-Level Isolation and Preprocessing

• Transient Indexing: For each query, the system ingests only the relevant background documents (.txt, .pdf, .md), processes them, and stores them in an ephemeral Elasticsearch index. This index is deleted immediately after the answer is generated to prevent data contamination.
• Token-Aware Chunking: Documents are normalized and split using a recursive, token-aware text splitter (based on tiktoken with p50k encoding).
  • Maximum chunk size ($L$): 300 tokens.
  • Overlap ($\Delta$): 30 tokens.
  • Metadata (filename, page, chunk ID) is attached to every chunk for provenance.

B. Embedding and Vector Indexing

• Chunks are embedded using a sentence embedding model (e.g., Sentence-BERT).
• Optional HyDE (Hypothetical Document Embeddings): A local LLM can generate a hypothetical passage for the query, which is then embedded to improve retrieval on sparse queries.
• Embeddings are stored in the transient Elasticsearch index.

C. Two-Stage Retrieval

The system employs a two-stage retrieval process to select a compact, mutually supportive context under a strict token budget:

1. Stage 1: Maximal Marginal Relevance (MMR):

   • Retrieves a small, diverse candidate set (default $k=3$) from the vector store.
   • Optimizes for both relevance to the query and diversity among candidates to avoid redundancy.
   • Formula: $d^* = \arg\max_{d \in D \setminus S} [\lambda s(d, q) - (1-\lambda) \max_{d' \in S} s(d, d')]$.

2. Stage 2: Reader-Driven PageRank (PR) Re-Ranking:

   • Constructs a similarity graph where nodes are the $k$ retrieved chunks and edges represent cosine similarity above a threshold ($\tau = 0.01$).
   • Applies a PageRank algorithm with a personalization vector derived from the query-chunk similarity scores.
   • Goal: Identify a compact set of chunks that are not only relevant to the query but also mutually reinforcing (high inter-snippet support).
   • The top $pr$ chunks (default = 3) are selected based on PageRank scores.

D. Answer Generation

• The selected chunks are concatenated and trimmed to a fixed token budget (default $T=1800$) to fit the LLM context window.
• A local LLM (e.g., Mistral-7B, LLaMA 2, AstroSage) generates the final answer in a single pass, providing the response along with exact source citations.
• The system supports multiple backends via OpenAI-compatible APIs or HuggingFace TGI.

3. Key Contributions

1. Instance-Level Isolation and Traceability: The system ingests only relevant files, constructs a transient index per instance, records the answer and supporting snippets, and erases the index post-generation. This ensures privacy and prevents cross-instance contamination.
2. Two-Stage Retrieval with Reader-Driven PageRank: The novel combination of MMR for initial diversity and a reader-driven PageRank re-ranking step selects a small, coherent set of snippets. This balances query relevance with inter-snippet support, optimizing for the "positional effect" in LLMs.
3. Fully Local and Modular Pipeline: The entire workflow (embedding, retrieval, generation) runs on local hardware, supporting offline operation and privacy-sensitive use cases without reliance on external cloud APIs.

4. Experimental Results

The pipeline was evaluated on the AstroQA benchmark (3,082 questions across easy, medium, and hard difficulty levels) and a custom AstroRef-QA dataset.

• Performance on AstroQA:
  • The Mistral-7B + AstroRAG variant achieved 79.49% accuracy and 79.49% F1-score on the total dataset.
  • This represents a near-doubling of performance compared to the non-RAG Mistral-7B baseline (21.74% accuracy).
  • Performance gains were consistent across difficulty levels: Easy (77.50%), Medium (80.28%), and Hard (83.59%).
  • AstroRAG outperformed other RAG baselines like DrQA and Map-Reduce strategies.
• Performance on AstroRef-QA:
  • Using ROUGE-L and ROUGE-N metrics, the Mistral-7B + AstroRAG model achieved the highest scores (30.17% ROUGE-L, 42.12% ROUGE-N), demonstrating improved lexical overlap and content relevance.
• Ablation Studies:
  • Token Budget: An output token limit of 64 provided the optimal balance between latency and accuracy.
  • Hyperparameters: The best performance was observed with a PageRank top-$K$ of 3 and a chunk overlap of 30 tokens.

5. Significance and Claims

The paper claims that AstroRAG establishes a robust foundation for extending RAG to other scientific fields by demonstrating that disciplined retrieval and refinement are critical for boosting domain-specific reasoning.

• Effectiveness of Re-ranking: The results highlight that simply retrieving more passages is less effective than using a two-stage process (MMR + PageRank) to select a compact, mutually supportive context.
• Privacy and Reproducibility: By operating locally and using transient indexing, the system addresses the critical needs of verifiability and privacy in scientific applications.
• Domain Adaptation: The framework successfully bridges the gap between general LLM capabilities and the rigorous requirements of astronomical reasoning, handling complex tasks such as magnitude-brightness reasoning, celestial object categorization, and sidereal time calculations.

The authors acknowledge limitations, including sensitivity to embedding model choices and retrieval hyperparameters, and suggest future work in adaptive chunking, multi-hop aggregation, and handling multilingual inputs.