Technical Summary: PETRA – Transforming Web Text for Petroleum-Engineering Domain Adaptation

Problem Statement

Petroleum engineering is a knowledge-intensive industry where operational decisions rely on scattered technical evidence (e.g., standard operating procedures, equipment manuals, incident reports). While retrieval-augmented systems offer a path to ground answers in source documents, they face a critical supervision gap: relevant evidence exists in public web text, but high-quality, domain-specific relevance labels are scarce.

Existing solutions fail to bridge this gap effectively:

• Curated domain datasets are often too small or narrow to cover the full spectrum of terminology, procedures, and equipment contexts required by deployed retrievers.
• Large web corpora provide scale but suffer from off-domain noise, duplication, weak structure, and a lack of query-passage relevance labels.
• Public benchmarks measure general retrieval quality but do not provide the specific supervision needed to adapt production search stacks for petroleum engineering, where relevance depends on domain conventions (acronyms like EUR, OOIP; polysemous terms like "well" or "field") rather than simple lexical overlap.

Methodology: The PETRA Pipeline

The authors introduce PETRA (Petroleum Engineering Text for Retrieval Adaptation), a large-scale dataset and pipeline designed to convert noisy public web data into domain-specific retrieval supervision. The pipeline operates in two primary stages:

1. Corpus Curation

The first stage transforms open-source documents into a high-recall, curated petroleum-engineering corpus.

• Source Selection: The pipeline processes three public source families: FinePDFs-Edu, peS2o scientific papers, and an English Wikipedia slice focused on petroleum engineering.
• Filtering & Cleaning: Documents undergo rigorous cleaning to remove references, OCR artifacts, and low-quality text using document-level heuristics (word count, non-alphanumeric ratios).
• Domain Classification: A LoRA-tuned Llama-3.1-8B classifier, trained on labels distilled from Mistral-Large-3-675B, acts as a high-recall energy gate. It achieves 98.4% test accuracy and 99.69% recall, ensuring that potentially valuable in-domain evidence is not discarded (minimizing false negatives).
• Chunking & Validation: Documents are deduplicated (exact and semantic) and split into chunks. A two-step majority-vote gate using Mistral-Large-3-675B validates chunks for informativeness, self-containment, and answerability, assigning them to a 13-field petroleum-engineering taxonomy.
• Scale: The resulting curated corpus contains 1.36 million chunks (approx. 2 billion token equivalents).

2. Synthetic Retrieval Supervision

Since the corpus lacks labeled relevance pairs, PETRA synthesizes training data for both first-stage retrieval (embedding) and reranking.

• Query Generation: An instruction-tuned LLM generates three query styles per chunk (natural-language questions, fact statements, keyword searches). A filter rejects items unsupported by the chunk or too vague.
• Hard Negative Generation:
  • LLM-Written: For each anchor, an LLM generates topically close passages containing logical or semantic errors (e.g., entity substitutions, omitted key values). These are validated to ensure they do not accidentally answer the query.
  • Retrieval-Mined: A baseline RAG stack is deployed to generate adversarial candidate lists. If the target chunk is not at rank 1, the retrieved candidates are assigned to "failure buckets."
• Reranker Training Data: Retrieved candidates and generated hard negatives are re-scored by a zero-shot LLM ranker (Qwen3.6-35B-A3B) to produce teacher relevance scores. This creates teacher-scored candidate rows that match the inference-time candidate distribution.
• Data Construction: The final output includes contrastive triples for embedding models and teacher-scored candidate rows for rerankers.

Model Training & Adaptation

The authors adapt a two-stage retrieval stack (Qwen3-Embedding-8B and Qwen3-Reranker-8B) using LoRA adapters:

• Embedding Adapter: Trained with InfoNCE contrastive loss on synthetic triples.
• Reranker Adapter: Trained with pointwise binary cross-entropy (BCE) on teacher-scored candidate rows.
• Inference Strategies:
  • Score Fusion: The first stage runs both base and adapted encoders, fusing scores with a fixed weight ($\alpha=0.7$) to balance in-domain gains with out-of-domain retention.
  • Adapter Merging: TIES (Task-Interference-Resolving) merging is used to combine domain-specific adapters with general-retrieval anchors (e.g., MS MARCO) or pilot adapters, creating single checkpoints that trade specialization for broader retention.

Key Results

The authors evaluate PETRA on an internal operator benchmark (SOP) and public benchmarks (Earth Science, BEIR tasks, and a reasoning-intensive panel).

• First-Stage Retrieval:

  • Adapting the embedding model with energy-only data improves in-domain nDCG@10 from 0.703 to 0.757 but causes a significant drop in out-of-domain performance (0.603 $\to$ 0.498).
  • Score fusion (base + adapter) achieves the best in-domain performance (0.763 nDCG@10) while retaining strong out-of-domain quality (0.570).
  • TIES merging offers the best single-checkpoint compromise, recovering 0.082 of the out-of-domain loss compared to the energy-only adapter.

• Reranking:

  • Adapting the reranker with PETRA's synthetic data yields a 44% relative improvement on the Earth Science benchmark (0.302 $\to$ 0.436) and a 23% improvement on a six-task reasoning panel.
  • The final checkpoint maintains or slightly improves performance on the internal SOP benchmark and out-of-domain tasks compared to the base reranker.

• Negative Findings (Lessons Learned):

  • High train-holdout accuracy on synthetic labels does not guarantee retrieval gains.
  • Retrieval-mined data is only effective when repackaged as teacher-scored candidate lists sampled from the inference-time distribution, rather than as standalone hard-negative triples.
  • Pairwise margin-ranking on mined triples caused regression on SOP and out-of-domain probes.

Significance and Claims

The paper claims PETRA is the first publicly described large-scale dataset and pipeline specifically for petroleum-engineering retrieval adaptation. Its significance lies in:

1. Scalability: Demonstrating that a scalable pipeline can transform noisy public web text into high-quality, domain-specific retrieval supervision without requiring massive human annotation.
2. Methodological Insight: Showing that the packaging of supervision data (specifically, using teacher-scored candidate lists that match inference distributions) is more critical than the raw accuracy of generated labels.
3. Practical Impact: Providing a working solution that pushes a deployed retrieval stack past its strong base models in-domain while preserving general retrieval capabilities, currently being validated with enterprise Oil and Gas customers.

The authors note limitations, including the proprietary nature of the primary internal benchmark (SOP), the English-only scope, and the fact that the system is currently in user-testing deployment rather than full production with online A/B testing. They emphasize that the system is designed as a human-in-the-loop assistant, surfacing source procedures for operators rather than making autonomous decisions.