From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise

This paper introduces a deterministic, automated pipeline that transforms raw domain corpora into completion-style benchmarks to provide scalable, unbiased, and LLM-independent evaluations of domain expertise in both base and instruction-tuned models, effectively addressing issues of benchmark contamination and multiple-choice bias.

The Gist

Here is an explanation of the paper, translated into everyday language with some creative analogies.

The Big Problem: How Do We Really Know What an AI "Knows"?

Imagine you have a library of books on Quantum Physics. You want to know which of your students (or AI models) actually understands the material, and which one just memorized the answers to a specific test.

Currently, the way we test AI is like giving them a Multiple Choice Quiz (like the MMLU benchmark). The paper argues this is a terrible way to test real expertise for three reasons:

1. The "Order Matters" Bug: If you shuffle the order of the answers (A, B, C, D), the AI's score changes wildly. It's like a student who only knows how to pick "C" because it's always in the middle, not because they know the answer.
2. The "Cheating" Problem: Many of these tests are already in the AI's training data. It's like giving a student a final exam they've already seen the answers to online. They aren't showing knowledge; they're showing memory.
3. The "Generalist" Trap: Standard tests ask broad questions. They don't tell you if an AI is a genius at neuroscience but terrible at cardiology.

The Solution: A Custom "Fill-in-the-Blank" Factory

The authors built a deterministic pipeline (a step-by-step machine) that turns raw, messy text from a specific field (like medical journals or physics papers) into a custom test.

Think of it like this:

• Old Way: You buy a pre-made, generic trivia game and hope it covers the specific topic you care about.
• New Way: You take a stack of raw textbooks, feed them into a machine, and it automatically generates a unique "Fill-in-the-Blank" test based only on the words and concepts found in those books.

How the Machine Works (The 4-Step Recipe)

1. Mining for Gold (Keyword Extraction): The system reads thousands of papers and pulls out the most important "buzzwords" (e.g., "reinforcement learning," "phylogenetic tree"). It filters out boring words like "the" or "study."
2. Finding the Context (Sentence Matching): For every buzzword, it finds sentences in the text that talk about that word.
3. Building the Puzzle (Prompt-Target Pairs): It takes a sentence and cuts it off right before a key term.
   • The Prompt: "Prior attempts at improving data efficiency in reinforcement learning involved the use of an Experience..."
   • The Target: "...Replay."
   • The AI has to guess the missing word.
4. The Scorecard (Ranking): Instead of asking "Did it get it right or wrong?", the system asks: "How high up on the AI's list of guesses was the correct word?"
   • If the AI puts the correct word as its #1 guess, that's a perfect score.
   • If it puts it as #500, it's struggling.
   • Why this matters: This measures confidence and knowledge without needing the AI to be a perfect conversationalist.

Why This is a Game Changer

The paper validates this method with some cool experiments:

• The "Expert" Check: They compared their auto-generated test against a test written by human experts from a famous textbook. The results matched almost perfectly (99% correlation). This proves their machine is as good as a human expert at building a test.
• The "Learning" Tracker: They watched an AI learn during its training.
  • Old Metric (Perplexity): Like checking a student's handwriting. It gets neater over time, but doesn't tell you if they actually learned the math.
  • New Metric (Their Rank): Like watching the student solve problems. You can see exactly when they start understanding the specific topic.
• The "Chatbot" Surprise: They tested "Base" models (raw, smart but blunt) vs. "Chat" models (polished, helpful, aligned with human rules).
  • The Shock: The "Chat" models often performed worse on specific domain knowledge than the raw "Base" models.
  • The Analogy: It's like taking a brilliant, grumpy professor (Base Model) and hiring a PR team to make them polite and friendly (Chat Model). In the process, the PR team accidentally made the professor forget some of their deep technical details to sound more "safe" and "general." The authors call this the "Alignment Tax."

The Bottom Line

This paper gives us a scalable, cheat-proof, and unbiased way to test if an AI is actually an expert in a specific field (like Law, Medicine, or Physics).

Instead of relying on multiple-choice questions that can be gamed or contaminated, they built a machine that turns raw domain data into a "Fill-in-the-Blank" exam. This allows researchers to see exactly how much an AI knows, how it learns, and whether making it "safer" (via instruction tuning) accidentally makes it "dumber" at its actual job.

Technical Summary

Here is a detailed technical summary of the paper "From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise."

1. Problem Statement

The rapid proliferation of Large Language Models (LLMs) has created a need to accurately assess domain-specific expertise (e.g., in healthcare, law, or specific scientific fields). However, current evaluation methods suffer from critical limitations:

• Perplexity: Aggregates prediction quality over all tokens, failing to distinguish between general linguistic fluency and genuine domain knowledge.
• Multiple-Choice Questions (MCQs): Benchmarks like MMLU are susceptible to biases (e.g., answer ordering, prompt formatting), often rely on data that may be contaminated in training sets, and disadvantage base models that lack instruction-following capabilities.
• Static Benchmarks: Existing domain-specific benchmarks are often manually curated, expensive to update, and prone to contamination as models are trained on public datasets containing benchmark questions.
• Lack of Granularity: Existing benchmarks often cover broad categories rather than fine-grained subdomains.

The authors propose a need for a scalable, contamination-resistant, and automated framework to generate domain-specific benchmarks from raw text corpora without relying on human annotation or other LLMs.

2. Methodology

The authors present a deterministic pipeline that transforms raw domain corpora (e.g., academic papers) into completion-style benchmarks. The process consists of six stages:

1. Data Curation:

   • Inputs: Raw domain text (e.g., full arXiv papers) and metadata (abstracts).
   • Source: The pipeline was instantiated using the RedPajama-Data-1T dataset (1.56M arXiv papers) and linked with Kaggle metadata.
   • Target Domains: CS.AI, Physics (Soc-Ph), Quantitative Biology (Q-Bio.PE), and General Economics (Econ-GN).

2. Keyword Generation:

   • Preprocessing: Abstracts are standardized (lowercasing, removing brackets, handling contractions).
   • N-gram Extraction: Gensim's Phrases model generates 2–7 token n-grams.
   • Filtering: Stopwords, generic academic terms, and quantitative expressions are removed.
   • Selection: Adaptive length-based filtering targets ~300 high-quality keywords per domain.
   • Deduplication: Semantic similarity (cosine similarity > 0.85 using all-MiniLM-L6-v2) is used to merge redundant keywords.

3. Keyword-Sentence Matching:

   • Sentences from full papers are embedded and matched against keywords using cosine similarity (threshold 0.5).
   • Sentences are cleaned (LaTeX removal, citation stripping) to ensure high-quality context.

4. Target Vocabulary Construction:

   • Two variants are created to capture different levels of specificity:
     • TF (Term Frequency): Captures broadly relevant domain terms.
     • TF-IDF (Term Frequency-Inverse Document Frequency): Surfaces rarer, more niche terminology by down-weighting terms common across all keyword corpora.

5. Prompt-Target Pair Construction:

   • For each keyword, sentences containing target vocabulary are selected.
   • Prompt: The sentence up to (but not including) the target term.
   • Target: The domain-specific term following the prompt.
   • Constraints: Prompts must be >10 tokens; targets must appear after the 10th token.
   • Output: Hundreds of prompt-target pairs per keyword.

6. Model Evaluation:

   • Metric: Prediction Rank (the rank of the correct target token in the model's output distribution).
   • Rationale: Rank is preferred over probability because LLM probabilities are often poorly calibrated, especially after instruction tuning (RLHF). Rank isolates what the model knows from how confidently it expresses it.
   • Aggregation: A 20% trimmed mean of ranks is used to reduce the impact of outliers.

3. Key Contributions

• Automated Benchmark Generation: A fully deterministic pipeline that generates benchmarks from any raw text corpus without human curation or reliance on other LLMs.
• Contamination Resistance: Benchmarks can be regenerated on-demand from fresh or held-out corpora, eliminating the risk of training-data contamination by design.
• Unified Evaluation Framework: The completion-style format allows for fair comparison between base models and instruction-tuned (chat) models, avoiding the formatting biases inherent in MCQs.
• Granularity: Enables evaluation at arbitrary levels of granularity, from broad disciplines down to specific research subdomains.

4. Results and Validation

The authors validated their pipeline through extensive experiments:

• Validation against Expert Benchmarks:

  • The pipeline was compared against a manually curated expert benchmark derived from the textbook Understanding Deep Learning.
  • Correlation: The automated pipeline showed near-perfect correlation with the expert benchmark (r=0.99, p<0.001).
  • Claude Proxy: A benchmark generated by Claude Sonnet 4 also correlated strongly with the expert benchmark (r=0.91), validating the use of high-quality LLMs as a scalable proxy for human curation.

• Domain Adaptation Experiments:

  • Models adapted to domains semantically close to the target (e.g., CS.CL for CS.AI) achieved significantly better prediction ranks than those adapted to unrelated domains.
  • Baseline Comparison: The proposed rank metric showed strong correlation with the expert benchmark, whereas Perplexity and Last-Layer Attribution Rate showed weak or no correlation, failing to capture genuine domain knowledge acquisition.

• Pretraining Dynamics:

  • The benchmark successfully tracked knowledge acquisition during general pretraining (OLMo-2) and continual pretraining (Llama2-7B).
  • Unlike MCQs, which often saturate early in training, the rank metric revealed nuanced, continuous improvements in domain knowledge throughout the training process.

• Base vs. Chat Models (The "Alignment Tax"):

  • Evaluating six model families across four domains revealed that instruction tuning generally degrades domain-specific knowledge compared to base models.
  • This "alignment tax" was highly heterogeneous; for example, Llama2-7B and Mistral-7B showed significant performance drops, while others (e.g., Qwen2-1.5B) showed minimal degradation or slight improvements.

5. Significance

This work provides a principled alternative for evaluating LLMs in specialized fields. By shifting from static, biased MCQs to dynamic, completion-based benchmarks derived directly from domain corpora, the authors offer a tool that:

1. Eliminates Contamination: Ensures evaluations are based on data the model has not seen.
2. Reduces Bias: Removes formatting and ordering biases associated with MCQs.
3. Enables Scalability: Allows researchers to instantly generate benchmarks for emerging fields or specific sub-domains.
4. Improves Insight: Reveals that instruction tuning may come at the cost of raw domain expertise, a critical insight for practitioners selecting models for specialized tasks.

The framework is particularly valuable for high-stakes domains (medicine, law, finance) where accurate, up-to-date, and contamination-free assessment of model capabilities is essential.