💬 NLP

HistoRAG: Embedding Historical Methodology in Retrieval-Augmented Generation Through Critical Technical Practice

This paper introduces HistoRAG, a framework that adapts Retrieval-Augmented Generation for historical research by integrating historiographical principles—such as temporal windowing, separated retrieval and generation, and transparent LLM-as-judge evaluation—to address standard RAG's factual biases and better support interpretive scholarly practices.

Original authors: Noah J. Kim-Baumann, Torsten Hiltmann

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Noah J. Kim-Baumann, Torsten Hiltmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a historian trying to understand how people felt about computers in Germany between 1950 and 1979. You have a massive library of 100,000+ newspaper articles.

If you use a standard AI tool (called "RAG") to help you, it acts like a very fast but slightly clumsy librarian. You ask, "What did people think about computers?" and the librarian runs to the shelves, grabs the books that have the word "computer" in them, and hands them to you.

The problem? This librarian has a few bad habits that ruin historical research:

They only speak modern slang. If you ask about "computers" using 1970s words, they ignore all the 1950s books because back then, people called them "electronic brains." The librarian misses the most important early history.
They grab the loudest books. If a topic was discussed a lot in 1970, the librarian grabs a huge pile of 1970s books and ignores the quieter, but equally important, discussions from 1950.
They don't understand why you are asking. They just grab books that look similar to your question, even if the books don't actually answer your specific research question about human fears or hopes.

The paper introduces HistoRAG (Historical RAG). Think of this not as a faster librarian, but as a new way of working with a research assistant that respects the rules of history.

Here is how HistoRAG works, using three simple tricks:

1. The "Two-Step Dance" (Separated Retrieval and Generation)

In a normal AI system, you ask a question, and it instantly gives you an answer. It's like a magic 8-ball.
In HistoRAG, the process is split into two distinct steps:

Step 1: The Hunt (Heuristik). The AI goes out and gathers a lot of potential sources. But it stops there. It puts them on a table and says, "Here are the candidates. You, the historian, must look at them and decide which ones are worth reading."
Step 2: The Analysis (Interpretation). Only after you have picked the best sources does the AI help you write a summary or find patterns.

The Analogy: Imagine you are a detective. A normal AI is like a detective who finds a suspect and immediately writes a closing argument. HistoRAG is a detective who finds a list of suspects, hands the list to you, and says, "You decide who to interview. Once you've picked the right people, then I'll help you write the report." This ensures the human historian stays in control of the story.

2. The "Time-Traveler's Grid" (Temporal Windowing)

Standard AI tools are biased toward the present. If you ask about "computers," the AI grabs mostly recent articles because the language matches better. It's like trying to find old recipes by only searching for modern cooking terms; you'll miss the old ones.

HistoRAG forces the AI to be fair. It divides the 30-year period into small time blocks (like 5-year chunks). It then says, "No matter how many 'computer' articles exist in 1970, you must grab an equal number of articles from 1950, 1955, 1960, etc."

The Analogy: Imagine you are making a fruit salad. A normal AI grabs a huge bucket of strawberries (because they are everywhere right now) and ignores the blueberries from last season. HistoRAG is a strict chef who says, "We need exactly one cup of fruit from every season of the year, even if the strawberries are more popular." This ensures you taste the full history, not just the most popular era.

3. The "Human-Like Judge" (LLM-as-Judge)

Even if the AI grabs the right time periods, it might still grab the wrong types of articles. It might grab a dry technical manual when you wanted a letter from a worried citizen.

HistoRAG adds a second layer of checking. After the AI grabs the articles, it runs them through a "Judge" (another AI) that acts like a human historian. This Judge reads the articles and asks: "Does this actually talk about fear or hope?" It gives the article a score and writes a reason why.

The Analogy: Imagine a talent show. The first AI is the "Casting Director" who finds everyone who can sing. The "Judge" is the "Music Critic" who listens to them and decides: "Okay, you can sing, but you aren't singing the emotional ballad we are looking for." The Judge doesn't just count notes; it understands the feeling of the song.

The Result: "Zwischentexte" (Intermediate Texts)

Finally, the paper talks about what the AI actually writes. In normal systems, the AI tries to give you a final answer. In HistoRAG, the AI produces what the authors call Zwischentexte (Intermediate Texts).

The Analogy: Think of these not as a finished novel, but as a rough draft or a sketch. The AI says, "Here is a possible interpretation of how people felt in 1964. Here is a pattern I see. But I am just a tool. You, the historian, must check my work, verify the sources, and decide if my sketch is accurate."

Why This Matters

The paper argues that for history and other interpretive fields, we shouldn't just want the "fastest" answer. We need a system that:

Doesn't forget the past just because the language changed.
Doesn't let the most popular era drown out the quiet ones.
Makes its reasoning visible so a human can say, "I disagree with that conclusion."

HistoRAG is a blueprint for building AI tools that act like humble research assistants rather than all-knowing oracles, ensuring that the human historian remains the one who decides what the past means.

Technical Summary: HistoRAG

Problem Statement
Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding Large Language Model (LLM) outputs in external evidence. However, dominant RAG paradigms are optimized for factual question-answering, assuming that "relevance" is a fixed property of documents and that similarity equates to utility. For interpretive disciplines like historical studies, these assumptions conflict with scholarly practice. Standard RAG systems conflate source discovery with interpretation, rely on similarity metrics that introduce temporal biases (due to vocabulary drift and thematic density), and treat the selection of sources as an opaque, automated process. This threatens the epistemic agency of researchers, who must be able to scrutinize, contest, and justify the basis of their source selection and interpretive framing.

Methodology: The HistoRAG Framework
The authors introduce HistoRAG, a framework that embeds historical methodology into RAG architecture through three specific interventions grounded in Critical Technical Practice (Agre, 1998) and the concept of Kontinuitätsannahme (Welskopp, 2008). These interventions are implemented in SPIEGELragged, a system applied to 102,189 articles from Der Spiegel (1950–1979).

Separated Retrieval and Generation (Heuristik vs. Analyse):
The framework formally decouples source discovery (Heuristik) from interpretation (Analyse). In standard RAG, a query triggers a seamless pipeline to an answer. In HistoRAG, the retrieval phase produces a curated corpus of chunks based on descriptive topic queries, not interrogative ones. A human researcher must explicitly review, accept, or reject these sources before any LLM-assisted analysis begins. This restores the distinction between finding sources and asking questions of them, ensuring the historian retains control over which materials enter the interpretive phase.
Temporal Windowing:
To counter the "vocabulary alignment bias" and "thematic density bias" inherent in similarity-based retrieval, HistoRAG enforces balanced source representation across the research period. Instead of retrieving the top- $N$ most similar chunks globally, the system divides the corpus into temporal windows (e.g., five-year intervals) and retrieves an equal quota from each. This ensures that periods with lower thematic density or distinct historical vocabulary (e.g., 1950s terms like Elektronenhirn vs. 1970s terms like Computer) are not systematically under-represented by modern query formulations.
LLM-as-Judge Evaluation:
Following retrieval and temporal balancing, the system employs an LLM to evaluate each chunk against explicit, researcher-defined criteria (a rubric). Unlike standard RAG which relies on vector similarity scores, this layer produces a numerical score and a written justification for relevance. This transforms the selection process from a black-box algorithmic filter into a transparent, argumentative process where the basis for relevance is contestable. The evaluation is performed independently (not comparatively) to avoid batch-ranking biases.

Key Contributions

Architectural Translation of Epistemology: The paper demonstrates how specific historiographical commitments (e.g., the need for balanced temporal sampling, the distinction between discovery and interpretation) can be translated into concrete software design decisions.
The Concept of Zwischentexte: The authors propose Zwischentexte (intermediate texts) as a framework for LLM outputs. Rather than treating generated text as final findings or answers, these outputs are positioned as "interpretive proposals" or scaffolding that require historian validation. This reframes the LLM's role from an oracle to a tool for hypothesis generation and pattern recognition within a documented research process.
Multi-Layer Retrieval Architecture: The system integrates keyword search (with corpus-trained vocabulary expansion), semantic retrieval, and LLM-based interpretive filtering as complementary layers, rather than competing alternatives.

Results and Evaluation
Using the SPIEGELragged implementation, the authors empirically validated the deficiencies of standard RAG and the efficacy of their interventions:

Temporal Bias: Standard similarity retrieval exhibits severe temporal skew. Queries using 1970s terminology retrieved zero chunks from the 1950s, while 1950s terminology retrieved 48% of results from the 1950s (double the corpus baseline). Temporal windowing successfully enforced balanced representation, accepting lower aggregate similarity scores to ensure coverage of formative periods.
Divergence of Relevance Metrics: There is a weak correlation (Spearman $\rho = 0.275$ ) between vector similarity scores and LLM-assessed relevance. Sources highly similar to a query (e.g., using the word "Computer") were often rated low by the LLM for interpretive relevance, while sources using different vocabulary (e.g., "Rationalisierung") but addressing the core research question (fears of automation) received high relevance scores.
Disjoint Source Pools: Keyword-based retrieval and semantic retrieval surface largely disjoint sets of articles. In the Der Spiegel test case, only 12.9% of articles overlapped between the two methods. The semantic layer successfully surfaced vernacular texts (reader letters) that keyword filters missed, but this introduced noise that required the LLM-as-judge layer to filter effectively.
Model Robustness: While absolute score calibration varied significantly across different LLMs (e.g., Mistral vs. Claude), the relative ranking of sources remained highly consistent ( $r > 0.96$ ). This suggests that while absolute thresholds are model-dependent, the relative ordering of relevance is portable.

Significance and Claims
The paper claims that HistoRAG offers a model for how domain-specific epistemological commitments can be operationalized in AI systems. It argues that for interpretive disciplines, the goal is not merely to retrieve the "best" passage for a factual query, but to construct a transparent, contestable, and temporally balanced corpus for interpretation.

The authors modestly position their work not as a final solution, but as an opening in a conversation about AI in historical research. They emphasize that the architecture is designed to preserve epistemic agency, ensuring that the historian remains the locus of accountability for source selection and argumentation. The system does not replace the historian but augments their capacity to engage with large corpora while maintaining the critical distance required for hermeneutic scholarship. The paper concludes that the tools should not determine the questions; rather, the questions (and the historian) should determine the tools.