EVE: A Domain-Specific LLM Framework for Earth Intelligence

1. Problem Statement

Earth Observation (EO) and Earth Sciences research generate vast amounts of high-value knowledge, yet this information remains fragmented across heterogeneous sources (scientific papers, institutional reports, encyclopedias) and formats. This fragmentation creates a "trust gap" for decision-makers who require transparent, scientifically robust insights to support environmental interventions ("Earth Action").

• The Gap: General-purpose Large Language Models (LLMs) lack the specific domain knowledge, rigorous evaluation standards, and grounding mechanisms required for reliable Earth Intelligence (EI). They often hallucinate facts or fail to interpret complex interdisciplinary data.
• The Need: There is a critical demand for systems that do not merely retrieve information but can interpret, reason, and synthesize knowledge across diverse EO and Earth Science domains while maintaining factual accuracy.

2. Methodology

The authors propose EVE (Earth Virtual Expert), an open-source, modular, end-to-end framework designed to build, evaluate, and deploy domain-specialized LLMs. The system consists of four core pillars:

A. Data Curation and Processing

• Corpus Construction: A massive corpus of 5.3 billion tokens was manually curated from 172 trusted sources (22 institutions), including open-access publishers (MDPI, Copernicus) and proprietary content (Wiley).
• Processing Pipeline:
  • Extraction: Used Trafilatura for HTML and Nougat (OCR) for PDFs, selected for superior handling of LaTeX formulas and tables.
  • Cleaning: Applied SHA-256 hashing for exact duplicates and MinHash LSH for near-duplicates. Used rule-based normalization and LLM-based syntax repair for OCR noise.
  • Anonymization: Used Microsoft Presidio to strip PII (names, emails).
  • Metadata Extraction: Extracted DOIs and queried CrossRef for structured metadata.
• Synthetic Data Generation: To augment the training set, the team generated 10.7 billion tokens of synthetic instruction data using an "Active Reading" pipeline. This involved:
  • Long-form text: Reorganizing salient content to reinforce terminology.
  • Instruction-formatted text: Creating diverse QA pairs (ContextQA, SelfQA, Multi-hop QA, LongQA) using high-quality models (Mistral Large, GPT-4o, Qwen, DeepSeek).
  • Filtering: LLM judges filtered data based on domain relevance and factual quality.

B. Model Training (EVE-Instruct)

• Base Model: Built upon Mistral Small 3.2 (24B parameters, 128k context).
• Training Strategy: To balance domain adaptation with general capabilities, the authors employed an interleaved fine-tuning strategy:
  • Mixed Instruction Fine-Tuning (IFT) with long-form text.
  • Combined domain-specific data with general-domain replay data to prevent catastrophic forgetting.
  • Used a learning rate schedule intermediate between standard IFT and Continuous Pre-training (CPT).
  • Ensemble: Merged checkpoints from ten different training runs using uniform parameter interpolation to optimize the trade-off between domain performance and general reasoning.
• Alignment: Applied Online Direct Preference Optimization (DPO) to refine formatting and stylistic consistency.

C. Grounded Generation & RAG Pipeline

• Retrieval-Augmented Generation (RAG): The system retrieves context from a Knowledge Base (KB) of ~365k documents.
  • Chunking: Two-pass strategy (sections, then paragraphs) preserving LaTeX/Markdown.
  • Embedding: Used Qwen3-Embedding-4B with binary quantization in Qdrant.
  • Re-ranking: Used Qwen3-Reranker-4B to select top documents.
• Hallucination Detection Pipeline: A multi-stage process where EVE-Instruct acts as an evaluator to detect hallucinations. If a hallucination is flagged, the system reformulates the query, retrieves new evidence, generates a revised answer, and ranks the original vs. revised output based on factuality.

D. Deployment

• Deployed as a production system via API and GUI, supporting 350 pilot users.
• Architecture: Single-node Qdrant DB, RunPod serverless inference (NVIDIA A100/H100), Amazon DocumentDB for state management, and an AWS CloudFront frontend.

3. Key Contributions

1. EVE-Instruct: A specialized 24B LLM optimized for Earth Intelligence, outperforming comparable models in its size range.
2. Curated Corpus & Synthetic Data: Release of a 2.8B token open-access corpus and a 10.7B token synthetic instruction dataset.
3. First Domain Benchmarks: Creation of the first systematic, manually curated benchmarks for EO and Earth Sciences (5,693 samples), covering MCQA, Open-Ended QA, and Hallucination Detection.
4. Production System: A deployed, hallucination-aware RAG chat system accessible via GUI and API.
5. Open Release: All models, datasets, code, and benchmarks are released under open licenses on Hugging Face and GitHub.

4. Results

The paper evaluates EVE-Instruct against the parent model (Mistral Small 3.2) and other comparable (24B) and larger (up to 180B) models.

• Domain Benchmarks (Table 3):
  • MCQA: Achieved the highest accuracy (86.12% IoU, 77.73% Acc) on both single and multiple-choice questions.
  • Hallucination Detection: Achieved the highest F1 score (84.70%), demonstrating superior ability to distinguish factual from non-factual responses.
  • Open-Ended QA: Led in pairwise win rates against competitors, even those with significantly larger parameter counts (e.g., outperforming GPT-4.1 and Qwen3 in specific win-rate metrics).
• General Capabilities (Table 4):
  • EVE-Instruct maintained or improved performance across general domains (Math, Coding, Tool Calling, Chat Quality) compared to the base Mistral Small 3.2.
  • Notable gains: +4.1 in Math & Reasoning, +3.0 in Tool Calling.
• Efficiency: Despite being a 24B model, EVE-Instruct achieved a competitive rank (1.33 average rank) against models up to 180B parameters, proving that careful domain engineering can rival massive scale.

5. Significance and Future Work

• Scientific Impact: EVE demonstrates that domain-specialized, grounded LLMs can provide reliable, actionable insights for Earth Sciences without requiring massive, resource-intensive models. It bridges the gap between raw data and decision-making.
• Reproducibility: By releasing the first dedicated benchmarks and a significant portion of the training data, the authors enable reproducible research in Earth Intelligence.
• Limitations: The current system is text-only (no direct reasoning over imagery or geospatial vectors) and faces licensing constraints on full corpus release.
• Future Direction: The authors aim to evolve EVE into a multimodal, agentic platform capable of reasoning over satellite imagery, in-situ data, and Earth system models, supporting complex, multi-step scientific workflows.

In conclusion, EVE represents a significant step toward "Earth Action," providing a practical, open, and scientifically rigorous tool for integrating heterogeneous Earth science knowledge into decision-support systems.