AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model

1. Problem Statement

General-purpose Large Language Models (LLMs) often struggle with specialized domain knowledge, particularly in complex scientific fields like astronomy, astrophysics, and cosmology. This limitation hinders their deployment as reliable research agents. While previous work (AstroSage-Llama-3.1-8B) demonstrated that domain specialization could allow smaller models to match larger generalist models, it remained unclear whether scaling this specialization to larger parameter counts (e.g., 70B) could enable specialized models to surpass the most advanced commercial proprietary models in their specific domain while maintaining cost efficiency.

2. Methodology

The authors developed AstroSage-Llama-3.1-70B, a 70-billion parameter domain-specialized AI assistant derived from the Meta-Llama-3.1-70B foundation. The development followed a rigorous three-stage pipeline:

• Continued Pre-training (CPT):

  • Data: The model was trained on a vast corpus of astronomical literature, including ~250,000 arXiv preprints (astro-ph, gr-qc), ~30,000 Wikipedia articles, and textbooks. The data was cleaned using ftfy for Unicode normalization and rule-based repetition removal.
  • Mitigating Catastrophic Forgetting: To preserve general language capabilities, a random selection of samples from the FineWeb dataset was interleaved into every training epoch ("replay").
  • Infrastructure: Trained on the Oak Ridge Leadership Computing Facility (OLCF) Frontier supercomputer using 2,048 AMD Instinct MI250X GPUs.
  • Hyperparameters: Learning rate was scaled to match tokens-per-second metrics from the original Llama-3.1 training; warm-up was extended to 0.15 epochs, and weight decay was increased to 0.1.

• Supervised Fine-Tuning (SFT):

  • Objective: To instill instruction-following, conversational skills, and explicit reasoning capabilities.
  • Data Composition: The dataset was a diverse mixture:
    • NVIDIA Llama-Nemotron-Post-Training-Dataset: Used for scientific reasoning, code, math, and general chat (approx. 22.7% of data).
    • OpenHermes 2.5: For general instruction following.
    • Custom Astronomy Q&A: Derived from previous AstroSage work (approx. 22.0% of data).
  • Reasoning Integration: The SFT dataset explicitly included Chain-of-Thought (CoT) reasoning chains, enabling the model to emit a human-readable thought process before answering.
  • Training: Conducted for 2 epochs on the same supercomputer infrastructure.

• Model Merging:

  • The final model was created using the DARE-TIES method (parameter averaging) via the mergekit library.
  • Mixture: 85% AstroSage-70B-SFT (specialized) + 15% Meta-Llama-3.1-70B-Instruct (general instruction polish). This strategy aimed to retain astronomical expertise while inheriting the robust conversational abilities of the base instruct model.

3. Key Contributions

• Domain Specialization at Scale: Demonstrated that scaling domain specialization from 8B to 70B parameters allows a model to outperform generalist counterparts and match top-tier proprietary models in a specific scientific domain.
• Explicit Reasoning Capability: Integrated a "thinking mode" where the model generates step-by-step analytical processes (enclosed in <thought> tags) before providing an answer, improving reliability on complex multi-step problems.
• Open-Weight Availability: Unlike many state-of-the-art models, AstroSage-Llama-3.1-70B is released openly under the Llama 3.1 Community License, enabling offline deployment and processing of sensitive institutional data (e.g., NASA, NSF).
• Cost-Efficiency Framework: Introduced a "Value Score" metric to quantify the trade-off between performance and inference cost, showing that specialized open models can offer orders of magnitude better value than proprietary APIs.

4. Results

The model was evaluated on the AstroMLab-1 benchmark, a validated subset of 3,846 multiple-choice questions derived from Annual Review of Astronomy and Astrophysics papers (withheld during training).

• Performance: AstroSage-Llama-3.1-70B achieved a score of 89.0%.
  • This places it on par with the latest proprietary flagships: GPT-5.2 (89.4%), Claude 4.5 Opus (89.0%), and Gemini 3.0 Pro (88.9%).
  • It significantly outperformed its base model (Llama-3.1-70B, 82.5%) and the smaller AstroSage-8B (81.9%).
• Cost-Efficiency:
  • The model achieved a Value Score of +5.55 relative to GPT-4.
  • While Gemini 3.0 Flash scored slightly higher (90.3%), it had a lower value score (+4.98) due to higher inference costs.
  • Practical Impact: Processing the entire astro-ph archive would cost approximately $1,000 using AstroSage-Llama-3.1-70B versus $10,000 using Gemini 3.0 Flash.
• Comparison to Humans: Professional astronomers score approximately 67% on this benchmark, indicating the model has surpassed human-level performance on standard knowledge recall tasks in this domain.

5. Significance

• Parity with Proprietary Giants: The study proves that domain-specialized open-weight models can compete with, and in some cost-metrics surpass, the most advanced closed-source commercial models.
• Democratization of AI in Science: By releasing a high-performance, 70B-parameter model openly, the authors enable researchers to run state-of-the-art AI on local infrastructure, addressing data privacy concerns and reducing reliance on expensive APIs.
• Future of Scientific AI: The paper argues that the future of AI in astronomy lies not just in scaling general models, but in specialized training combined with agentic workflows (e.g., querying databases like SIMBAD or executing analysis code).
• Benchmarking Evolution: The work highlights the need for future benchmarks that move beyond simple knowledge retrieval to test complex reasoning, hypothesis formulation, and data interpretation in realistic research scenarios.

In conclusion, AstroSage-Llama-3.1-70B represents a significant leap in specialized AI, demonstrating that targeted domain training on large-scale architectures can deliver superior, cost-effective, and open-access tools for the astronomical community.