STILTS-NLI: A Natural Language Interface for STILTS

1. Problem Statement

STILTS (Starlink Tables Infrastructure Library Tool Set) is a powerful, widely used command-line tool for processing astronomical tabular data. However, its comprehensive and rigid syntax creates a steep learning curve, making it underutilized compared to its graphical counterpart, TOPCAT.

• The Barrier: New users struggle to craft valid commands due to complex parameter combinations and strict syntax requirements.
• The Gap: Existing Large Language Models (LLMs) often fail to generate syntactically correct STILTS commands because the tool's specific syntax represents a minuscule fraction of their general training data, leading to "hallucinations" (invalid functions or arguments).
• Accessibility Issue: Reliance on proprietary, large-scale LLMs (via APIs) creates barriers regarding cost, data privacy, and computational resource requirements, limiting accessibility for the broader astronomical community.

2. Methodology

The authors developed STILTS-NLI, a natural language interface that translates user prompts into valid STILTS commands using a specialized, fine-tuned open-source LLM.

A. Synthetic Data Generation

Since existing documentation provided only ~150 examples (insufficient for fine-tuning), the team created a synthetic dataset:

• Target Tasks: Focused on 12 high-value STILTS tasks (e.g., tpipe, tmatch, tapquery, tcat).
• Generation Strategy: Used NotebookLM (Google) to generate diverse prompt-response pairs. This approach proved superior to other models (like GPT-4.5 or Claude) in generating natural language prompts and avoiding syntax hallucinations.
• Validation Pipeline:
  1. Automated Syntax Check: Used internal STILTS functions to validate command structure.
  2. Manual Inspection: Fixed common errors (e.g., missing quotes in cmd arguments, incorrect unit conversions for radius).
  3. Result: Started with 7,270 pairs; after filtering invalid entries and fixing errors, 6,228 high-quality pairs remained for training.

B. Model Training

The authors utilized Google's Gemma model family, chosen for its compact size and performance.

• Models Tested: Gemma 2B (2 billion parameters) and Gemma 27B (27 billion parameters).
• Two-Phase Training:
  1. Domain-Adaptive Pre-training: Exposed the model to the full STILTS documentation (cleaned HTML) for a single epoch to learn vocabulary and syntax embeddings.
  2. Fine-tuning: Trained on the 6,228 synthetic prompt-response pairs using AdamW optimizer and cross-entropy loss.
• Hardware: The 2B model trained on a single NVIDIA A6000 GPU (30 mins); the 27B model required a cluster of NVIDIA GH200 Superchips (12 hours).

C. Agentic Software Implementation

To improve user experience, the system employs a two-model architecture:

1. Interface Layer: Uses Llama-3.2-3B-Instruct to handle user interaction, manage history, and interpret complex instructions.
2. Core Translation Layer: Routes specific command-generation tasks to the fine-tuned Gemma-2B model.
3. Execution: The system can execute the generated command locally or return the command for user verification.

• Optimization: A "model-only" mode allows the tool to run on local machines with ~5GB RAM, removing reliance on external APIs.

3. Key Contributions

• STILTS-NLI Tool: A fully functional, open-source command-line interface that converts natural language descriptions into executable STILTS commands.
• Curated Dataset: A validated dataset of ~6,000 synthetic prompt-response pairs specifically for STILTS, addressing the lack of domain-specific training data.
• Small-Model Viability: Demonstrated that a compact, open-source model (2B parameters) can outperform massive proprietary models (GPT-5, Gemini 2.5 Pro) on niche, syntax-rigid tasks.
• Accessibility Framework: Proved that high-quality scientific interfaces can be built using local, low-resource hardware, ensuring privacy and zero cost for users.

4. Results

A. Performance Comparison

The fine-tuned 2B model was compared against Gemini 2.5 Pro and GPT-5 on four prompts of increasing difficulty (file conversion, column calculation, null value handling, and complex multi-catalog matching).

• STILTS-NLI (2B): Generated 100% syntactically valid commands for all prompts.
• Proprietary Models:
  • Gemini 2.5 Pro: Failed on the complex 4-way catalog match (Prompt 4) due to hallucinated join arguments.
  • GPT-5: Failed on Prompts 3 and 4, hallucinating invalid functions (e.g., isnan instead of STILTS-native logic) and incorrect syntax.
• Conclusion: Specialized fine-tuning on a curated dataset effectively eliminates hallucinations that plague generalist models.

B. Model Size vs. Performance

• The 2B model and 27B model achieved comparable performance on the evaluation set (Cosine similarity scores: 0.91 vs. 0.89).
• The 27B model required significantly more training time and resources but offered no performance gain, suggesting that for this specific domain, a smaller model is sufficient if the data quality is high.

C. Evaluation Metrics

• Cosine Similarity: The generated commands showed high semantic similarity (mean ~0.90) to the expected ground truth.
• Syntax Validity: The primary metric was the ability to pass STILTS's internal syntax checker, which the fine-tuned model achieved consistently.

5. Significance and Future Work

• Lowering Barriers: STILTS-NLI democratizes access to powerful astronomical data processing tools, allowing researchers without deep command-line expertise to utilize STILTS.
• Open Science: By using open-source models and local execution, the tool ensures reproducibility, data privacy, and independence from commercial API constraints.
• Future Extensions:
  • Expansion: Currently limited to 12 tasks; future work aims to cover the full STILTS suite.
  • Data Quality: The bottleneck is the labor-intensive creation of high-quality synthetic data. The authors propose a community-driven approach where users submit corrected prompt-response pairs to improve the model iteratively.
  • Accessibility: Potential integration of speech-to-text layers to assist users with motor impairments.

In summary, the paper establishes that data quality and domain-specific fine-tuning are more critical than model scale for specialized scientific tools, offering a blueprint for making complex astronomical software accessible to a wider audience.