Technical Summary: A Systematic Evaluation of Retrieval-Augmented Generation and Language Models for Space Operations

Problem Statement

The rapid expansion of space activities has resulted in an unprecedented accumulation of technical documentation, operational guidelines, and scientific literature. This volume creates significant challenges for timely decision-making in space operations, where mission teams must rely on highly specialized and continuously evolving information. Current operational workflows often depend on manual document inspection and legacy toolchains, leading to inefficient, error-prone, and cognitively demanding processes. While Large Language Models (LLMs) offer potential for reducing cognitive load, they struggle with domain-specific tasks and hallucinations when not grounded in authoritative sources. Furthermore, the sensitive nature of mission data often precludes the use of cloud-based services, necessitating solutions deployable within secure, closed environments.

Methodology

This paper presents a systematic evaluation of Retrieval-Augmented Generation (RAG) pipelines tailored for space operations, focusing on two representative subdomains: space debris mitigation documentation and the European Space Agency (ESA) mission Q&A dataset. The study extends previous work by Belo et al. [1] through the following methodological steps:

1. Dataset Construction: The authors extended the existing dataset by adding negative passage samples per query to create a context relevance dataset. They employed two strategies for negative mining: Random Negative Sampling (sampling unrelated in-domain passages) and In-Document Negative Sampling (sampling passages from the same document but unrelated to the query).
2. Reranker Validation: Three open-source rerankers (BGE-M3, GTE-reranker-base, and Jina-reranker-v2) were evaluated as context relevance classifiers. They were used as a proxy for ground truth to assess retrieval performance, given the limitations of synthetic datasets for direct retrieval evaluation.
3. Retrieval Benchmarking: Eight state-of-the-art embedding models and the BM25 algorithm were benchmarked as retrievers. The evaluation utilized four metrics (Recall, Precision, NDCG, and Kendall Tau) across two chunking strategies (512 tokens vs. 2,000 tokens).
4. Impact Analysis: The study analyzed the impact of reranking and chunk size on relevance scores assigned by LLMs.
5. Answer Generation Evaluation: The factual accuracy and quality of generated answers were assessed using a curated ESA dataset (SpaceQA) containing 60 question–passage–answer triplets. The Llama 3 8B model was tested under noisy conditions (introducing four random irrelevant passages) to evaluate robustness.

Key Contributions

The paper makes four primary contributions to the field of space operations and RAG evaluation:

1. Enhanced Dataset: Creation of a context relevance dataset by augmenting the previous work with negative passage samples, enabling more rigorous evaluation of retrieval and reranking components.
2. Reranker Validation: Evaluation of three reranking models, demonstrating their effectiveness in distinguishing relevant from irrelevant passages and validating their utility as a ground-truth proxy for retrieval evaluation.
3. Comprehensive Retrieval Benchmarking: Benchmarking of eight embedding models and BM25 across multiple metrics and chunking strategies (512 vs. 2,000 tokens), providing insights into optimal retrieval configurations.
4. Answer Quality Assessment: Evaluation of factual accuracy and answer quality (faithfulness, relevance, noise robustness) using the ESA SpaceQA dataset, including an analysis of failure modes in the generated answers.

Results

The evaluation yielded several key findings regarding the performance of RAG components in the space domain:

• Reranker Performance: All tested rerankers demonstrated strong classification performance (F1-Scores > 0.93), confirming their reliability for distinguishing relevant contexts.
• Retrieval Models: BM25 and Qwen2-1.5B Instruct showed the most consistent performance across all metrics and k values. Conversely, STS-MPNet v2 and E5 Large performed the worst. The study suggests that pipelines with rerankers should prioritize high-recall retrievers (like BM25), while pipelines relying solely on retrievers should prioritize models with high NDCG (like BGE-M3 or Qwen2).
• Impact of Reranking and Chunk Size: Reranking consistently reduced the proportion of irrelevant documents and increased highly relevant ones across all top-k ranges. The study found that 512-token passages combined with reranking provided the most effective setup. Shorter passages reduced noise, allowing the reranker to more effectively boost the proportion of highly relevant documents compared to 2,000-token chunks.
• Answer Generation: The Llama 3 8B model achieved an answer accuracy of 56/60 when provided with relevant context, compared to only 3/60 without retrieval (RAG). The model demonstrated high robustness to noise (4.52/5.0) and strong faithfulness (3.92/5.0).
• Failure Analysis: Incorrect answers primarily occurred when the connection between the question and the supporting passage was weak or implicit (e.g., missing entity mentions or requiring bridging inferences). The model tended to default to conservative responses rather than speculating, which reduced hallucinations but limited its ability to answer indirect questions.

Significance and Claims

The paper claims that RAG pipelines can significantly enhance knowledge access, reduce uncertainty, and support decision-making in complex space operations. The results demonstrate that current open-source models are promising for controlled deployment in space environments, particularly when:

• Open-source frameworks are used to maintain data privacy and security within closed environments.
• Reranking is employed to maximize context relevance.
• Shorter chunk sizes (512 tokens) are utilized to minimize noise.

The authors emphasize that while the results are indicative rather than definitive due to the limited size of the evaluation dataset (60 examples), the study provides a critical initial benchmark. It highlights that retrieval design, including chunking strategies and metadata inclusion, plays a critical role in overall system performance. The work underscores the necessity of grounding LLM outputs in authoritative sources to improve factual accuracy and reliability, thereby mitigating the risks of hallucination in safety-critical space operations.