FormationEval, an open multiple-choice benchmark for petroleum geoscience

1. Problem Statement

Large Language Models (LLMs) are increasingly applied to scientific and engineering domains, yet their capabilities in specialized fields like petroleum geoscience and subsurface engineering remain difficult to assess.

• Gap in Benchmarks: General benchmarks (e.g., MMLU) cover broad knowledge but lack depth in specialized technical fields. Existing domain-specific benchmarks exist for medicine and law, but none exist publicly for petroleum geoscience.
• Evaluation Challenges: Assessing models in this field requires testing deep understanding of well logging physics, reservoir characterization, and geological interpretation, rather than simple phrase recognition.
• Copyright Constraints: Creating high-quality benchmarks from authoritative textbooks often risks copyright infringement if questions are derived via verbatim copying or close paraphrasing.

2. Methodology

The paper introduces FormationEval, a 505-question multiple-choice benchmark designed to evaluate LLMs on petroleum geoscience knowledge while adhering to strict copyright and quality standards.

A. Dataset Construction & Concept-Based Derivation

• Sources: Questions were derived from three authoritative sources:
  1. Well Logging for Earth Scientists by Ellis & Singer (219 questions).
  2. Petroleum Geoscience by Bjørlykke (262 questions).
  3. TU Delft OpenCourseWare (24 questions).
• Generation Pipeline: A reasoning model (GPT-5.2 with high reasoning effort) was used to generate questions.
  • Concept-Based Approach: Instead of copying sentences, the model extracted concepts from the source text and generated new questions from scratch. This tests understanding rather than pattern matching and avoids copyright issues.
  • Standalone Requirement: Questions must be answerable using general domain knowledge without referencing the specific source chapter (e.g., prohibiting phrases like "according to the text").
• Domains: The dataset covers seven domains: Petrophysics, Petroleum Geology, Sedimentology, Geophysics, Reservoir Engineering, Drilling Engineering, and Production Engineering.
• Difficulty Levels: Questions are categorized as Easy (undergraduate), Medium (graduate/professional), and Hard (specialist).

B. Quality Assurance & Bias Mitigation

• Human Verification: A domain expert (petrophysicist) performed spot checks on all questions to verify answer correctness, distractor plausibility, and the absence of copied text.
• Bias Analysis & Correction:
  • Length Bias: Initial analysis showed correct answers were significantly longer than distractors (>55% of cases). The authors expanded distractors with technical context to reduce this gap (down to 43.2%).
  • Qualifier Bias: Absolute words (e.g., "always") appeared only in distractors, while hedged words (e.g., "may") appeared only in correct answers. Synonyms were introduced, and hedging was added to distractors to break these exploitable patterns.
• Metadata: Each question includes source provenance, contamination risk labels (low/medium/high), and a rationale for the correct answer.

C. Evaluation Setup

• Models Evaluated: 72 models from major providers (OpenAI, Anthropic, Google, Meta, DeepSeek, Mistral, Zhipu, etc.) and open-weight alternatives.
• Metrics: Primary metric is overall accuracy. Secondary metrics include performance by domain, difficulty level, and bias analysis (position and length bias).
• Reproducibility: The evaluation uses a zero-shot prompt format. API responses are cached to allow re-analysis without additional costs.

3. Key Contributions

1. First Open Benchmark for Petroleum Geoscience: Fills a critical gap in domain-specific evaluation, covering seven key sub-disciplines.
2. Copyright-Compliant Generation Methodology: Proposes a "concept-based" derivation pipeline that respects intellectual property while testing deep domain understanding.
3. Comprehensive Evaluation: Provides a leaderboard for 72 models, analyzing performance across different providers, model sizes (3B to frontier), and cost tiers.
4. Community Engagement Tool: Developed an interactive website allowing domain experts to take the quiz and provide feedback, creating a continuous quality assurance loop.
5. Transparency: Publicly releases the dataset, evaluation code, and detailed bias analysis.

4. Key Results

• Top Performers:
  • Gemini 3 Pro Preview achieved the highest accuracy at 99.8% (504/505).
  • GLM-4.7 (Open-weight) led the open models with 98.6%.
  • Other top performers include Gemini 3 Flash Preview (98.2%) and Grok-4.1-fast (97.6%).
• Open-Weight vs. Closed Models: The performance gap is narrower than expected. Several lower-cost open-weight models (e.g., DeepSeek-R1, Qwen3 variants) exceeded 90% accuracy, with some (like GLM-4.7) rivaling the most expensive closed models.
• Domain Difficulty:
  • Petrophysics was the most challenging domain for all models (average 87.5% accuracy), likely due to the complex physics of well logging.
  • Reservoir Engineering had the highest average accuracy (95.6%).
• Difficulty Trends: Contrary to some expectations, Medium difficulty questions showed the lowest accuracy across most models, suggesting they require specific application of concepts that is harder than simple recall (Easy) or complex integration (Hard).
• Model Size Correlation: Smaller models (e.g., Llama-3.2-3b) showed significant variance and lower accuracy (57.6%), while compact models (3B–12B) generally struggled with Petrophysics specifically.

5. Significance and Future Work

• Practical Utility: The benchmark demonstrates that frontier LLMs possess near-expert knowledge in petroleum geoscience, making them viable for assisting in well logging interpretation, reservoir characterization, and educational tools.
• Cost-Effectiveness: The results highlight that high-accuracy performance is achievable with lower-cost open-weight models, democratizing access to domain-specific AI tools.
• Living Resource: FormationEval is designed as a "living" benchmark. Future work includes expanding to additional languages (Norwegian, Russian), balancing domain coverage (currently Petrophysics-heavy), and developing better contamination detection methods.
• Limitations: The paper acknowledges residual biases (length and absolute words) and the inability to fully verify training data contamination, though these are documented for transparency.

In conclusion, FormationEval establishes a rigorous standard for evaluating AI in the energy sector, proving that concept-based generation can create high-quality, copyright-safe benchmarks that reveal nuanced performance differences between state-of-the-art language models.