💬 NLP

Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature

This paper introduces "Loci Similes," a benchmark dataset of 545 expert-verified parallels between Late Antique and classical Latin authors, designed to overcome the scarcity of standardized resources for training and evaluating large language models on the detection of intertextualities.

Original authors: Julian Schelb, Michael Wittweiler, Marie Revellio, Barbara Feichtinger, Andreas Spitz

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julian Schelb, Michael Wittweiler, Marie Revellio, Barbara Feichtinger, Andreas Spitz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a giant, ancient library. The mystery? Figuring out which old stories were secretly copied, tweaked, or whispered about by later writers. In the world of Latin literature, this is called "intertextuality." For centuries, scholars had to read thousands of dusty scrolls by hand, relying on their memory to spot when a writer like Jerome (a Christian monk) was secretly borrowing a cool phrase from Virgil (a pagan poet).

But here's the problem: writers didn't just copy-paste. They changed the word order, swapped synonyms, or hid the reference so well that a simple "search for this word" tool would miss it. It's like trying to find a specific song in a playlist just by humming a few notes, but the singer changed the tempo and the lyrics slightly.

Enter Loci Similes, a new digital tool created by a team of computer scientists and Latin experts to help solve this. Think of it as a massive, super-organized "cheat sheet" for the library.

The Big Library and the Cheat Sheet

The team built a dataset containing about 176,000 small chunks of text. They split these into two piles:

The "Query" Pile: Texts written later (by authors like Jerome, Lactantius, and Valerius Flaccus).
The "Source" Pile: Texts written earlier (by classics like Cicero, Virgil, Ovid, and Horace).

To make sure their computer tools were actually learning, they created a "ground truth" cheat sheet. This sheet contains 1,490 specific connections that human experts verified were real. It's like having an answer key where the experts said, "Yes, this sentence in Jerome's book is definitely a nod to this sentence in Virgil's book."

The Great Computer Race

The researchers asked: Can computers find these hidden connections better than old-school search tools?

They ran a race with three types of computer models:

The "Word Counter" (Lexical Models): These look for exact words or word stems. They are like a librarian who only checks if the exact same words appear in both books.
The "Deep Thinker" (Dense Neural Models): These use advanced AI to understand the meaning of a sentence, even if the words are different. They are like a detective who understands the vibe of a story.
The "Two-Step Team" (Retrieve-and-Rerank): This team first uses the "Word Counter" to grab a shortlist of suspects, then uses the "Deep Thinker" to double-check them.

What They Found (The Plot Twist)

Here is where the story gets interesting, because the results weren't what everyone expected.

1. The "Deep Thinker" didn't win the first round.
You might think the fancy AI that understands meaning would be the best at finding hidden references. But the paper suggests that for Latin, the old-school "Word Counter" (specifically a method called BM25 using lemmas, which are the dictionary forms of words) actually did a better job at the first stage.

Why? Latin is a language with crazy word endings that change based on grammar. The fancy AI sometimes got confused by these changes, while the "Word Counter" was good at stripping them away to find the core word.
The Result: The "Word Counter" found about 78% of the direct, word-for-word copies (verbatim references) when looking at the top 100 candidates. The fancy AI models were close, but didn't beat it.

2. The "Deep Thinker" is needed for the hard stuff.
While the "Word Counter" was great at finding exact copies, it failed miserably at finding allusions (subtle hints where the meaning is similar but the words are totally different).

The Problem: If Jerome wrote a sentence that felt like Virgil but used completely different words, the "Word Counter" saw zero connection.
The Solution: The "Deep Thinker" (specifically a model called XLM-R Large) was much better at spotting these subtle, low-overlap connections. It could tell, "Hey, even though the words are different, these two sentences are talking about the same scary feeling."

3. The Winning Strategy: The Two-Step Team.
The paper suggests that the best way to solve the mystery isn't to pick just one detective, but to use a team.

Step 1: Use the "Word Counter" to scan the whole library and pull out the top 100 most likely suspects for every query.
Step 2: Hand that shortlist to the "Deep Thinker" to decide which ones are real references and which are just coincidences.
The Result: This team approach managed to recover 63% of the true hidden references while shrinking the list of things a human scholar would need to read by 97%. It's like turning a haystack of a million needles into a small basket of 1,900 needles, making the job much easier.

What the Paper Rules Out

The authors are very clear about what doesn't work well:

Just using fancy AI alone isn't enough. If you rely only on the "Deep Thinker" without a good first filter, you miss too many connections.
Just using simple word matching isn't enough. If you only look for exact words, you miss the subtle, clever references that are the most interesting part of the research.
It's not a "solved" problem. The paper explicitly states that finding these subtle links is still "challenging." The AI is good, but it still makes mistakes, especially when the connection is very short or the words are very different.

The Bottom Line

The paper suggests that while AI is a powerful new tool for studying ancient literature, it's not a magic wand that solves everything instantly. The best approach right now is a hybrid one: use simple, fast tools to narrow down the search, and then use smart, deep-learning tools to make the final call.

The authors admit that their dataset isn't perfect (it's not every single connection that ever existed), and that defining what counts as a "reference" can be tricky even for humans. But with 1,490 expert-verified examples and a new benchmark called Loci Similes, they have given future researchers a solid starting line to build even better tools. The mystery of the ancient library is still being solved, but now we have a much better flashlight.

Technical Summary: Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature

Problem Statement
Tracing intertextual connections in Latin literature is a fundamental task in classical philology, essential for reconstructing an author's "virtual library" and understanding cultural reception, particularly in Late Antiquity where Christian authors recontextualized pagan texts. However, automated detection remains constrained by the scarcity of standardized benchmarks and accessible datasets. Existing methods often rely on exact $n$ -gram matching or heuristic filtering, which fail to capture subtle allusions, paraphrases, and morphological variations that do not share significant surface vocabulary. While language models offer potential for capturing semantic similarity beyond lexical overlap, their development for this specific domain is hindered by the lack of curated, expert-verified ground truth data.

Methodology
The authors introduce Loci Similes, a comprehensive benchmark designed to evaluate computational approaches for Latin intertextuality detection. The methodology involves three primary components:

Dataset Construction:
- Corpus: The dataset comprises $\sim$ 176,000 text segments divided into a Query Corpus (later authors: Jerome, Lactantius, and Valerius Flaccus) and a Source Corpus (earlier canonical authors: Cicero, Virgil, Ovid, etc.).
- Ground Truth: A merged dataset of 1,490 expert-verified intertextual links was constructed by combining 545 newly curated references with 945 labeled references from a previous study (Burns et al., 2021).
- Annotation Scheme: Links are classified into three categories: No Match, Verbatim Reference (cit.) (including marked, unmarked, and paraphrased reuse), and Allusion (cf.) (thematic or stylistic echoes with low lexical overlap). Annotation was performed by four Latin literature experts using criteria based on uncommon vocabulary, attested frequency, and conduit function.
Experimental Setup:
- Tasks: The study evaluates three method families: (i) Retrieval-only (ranking source segments), (ii) Classification-only (distinguishing match types on a fixed candidate set), and (iii) Retrieve-and-Rerank pipelines.
- Models:
  - Retrievers: Sparse lexical methods (TF-IDF, BM25 with surface tokens and lemmas), word embeddings (Word2Vec, LatinBERT), and dense bi-encoders (multilingual models like BGE-M3, E5, and Latin-specific SPhilBERTa).
  - Classifiers: Lexical baselines (LogReg, GBDT) and cross-encoder transformers (mmBERT, XLM-R, mBERT, and Latin-specific variants).
- Evaluation: The study uses 5-fold cross-validation with realistic retrieval scenarios (query vs. in-domain distractors). Metrics include Recall@k, Precision, Recall, F1, and error rates (SMR, FPR, FNR).

Key Contributions

Curated Benchmark: The release of a dataset containing $\sim$ 176k segments and 1,490 labeled links, distinguishing between verbatim references and allusions, addressing a critical gap in standardized resources for Latin intertextuality.
Evaluation Framework: An alignment of computational evaluation with practical philological workflows, specifically testing the ability to detect subtle allusions alongside direct quotations.
Baseline Results: Comprehensive performance baselines for retrieval and classification models, establishing a foundation for future comparative research.

Results

Retrieval Performance: Contrary to trends in general NLP, sparse lexical methods outperformed dense neural retrievers. Specifically, BM25 with lemmatized unigrams achieved the best Recall@100 (0.78 for verbatim, 0.58 for allusions). Dense multilingual encoders (e.g., BGE-M3) performed better than Latin-specific baselines but still lagged behind lexical methods.
Classification Performance: Transformer cross-encoders significantly outperformed lexical classifiers, particularly for allusions. XLM-R Large achieved the best F1 for verbatim references (0.76), while mmBERT performed best for allusions (F1 0.13), though allusion detection remained challenging with low precision.
Retrieve-and-Rerank Pipeline: Combining BM25 retrieval with a cross-encoder reranker yielded the most practical results. At $k=100$ , the pipeline recovered 63% of true intertextual references while reducing the candidate set by 97% compared to retrieval-only baselines, significantly narrowing the scope for human inspection.
Error Analysis: Performance is heavily correlated with lexical overlap for verbatim references but decouples for allusions. False positives often arise from high-overlap generic phrases (e.g., omnia timeo), while false negatives occur when short, low-overlap allusions are lost in pooled sentence embeddings.

Significance and Claims
The paper claims that while language models offer a promising path for detecting semantic similarity in Latin, the primary challenge remains distinguishing meaningful reuse from coincidental lexical overlap. The authors assert that:

Hybrid approaches are necessary: The rich morphology of Latin fragments under subword tokenizers, giving sparse, lemma-based methods an advantage in the retrieval stage.
Neural models excel at nuance: Cross-encoders provide a distinct advantage over lexical classifiers for detecting subtle allusions where lexical features fail.
Current limitations: Detecting short, low-overlap parallels remains difficult. The authors suggest future work should explore token-level matching (late-interaction retrievers) and span-level analysis to recover signals lost in sentence-level pooling.

The study concludes that Loci Similes provides a necessary infrastructure for advancing intertextuality research, but further architectural developments are required to fully automate the detection of subtle, recontextualized echoes in Latin literature.