MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection

Technical Summary: MiqraBERT

Problem Statement
Textual reuse is pervasive in the Hebrew Bible (HB), yet computational detection methods have historically relied on lexical overlap (shared words and string similarity). These pre-neural approaches fail when parallels involve paraphrase, lexical substitution, or syntactic restructuring, which are common in biblical transmission. While neural network methods have advanced in other classical languages (e.g., Latin, Greek), no model has been specifically finetuned for verse-level semantic similarity in Biblical Hebrew (BH). Furthermore, pre-trained transformer models suffer from "anisotropy," a geometric flaw where sentence embeddings cluster into narrow cones or drift toward dominant directions, causing unrelated texts to appear semantically similar simply because they share a register (e.g., all Biblical Hebrew verses clustering together regardless of content).

Methodology
The paper introduces MiqraBERT, a regression-based Sentence-BERT model finetuned from AlephBERT, a state-of-the-art Modern Hebrew (MH) language model. The methodology addresses the anisotropy problem and the lack of BH-specific semantic similarity tools through the following steps:

• Base Model Selection: AlephBERT was chosen over multilingual models (like E5) or ancient Hebrew-specific models (like Embible) because it preserves the triconsonantal root system and verbal conjugation patterns of BH while demonstrating superior baseline performance in separating parallel and non-parallel distributions.
• Training Data: The training set consists of 1,650 labeled verse and half-verse pairs:
  • 825 Positive Pairs: True parallels drawn from Chronicles synoptic material (narrative) and four foundational studies of poetic parallelism.
  • 825 Negative Pairs: Randomly sampled non-parallel verses from the entire HB.
• Finetuning Architecture: The model employs a Siamese network structure with shared weights. Both input verses are encoded by AlephBERT, mean-pooled into 768-dimensional vectors, and compared via cosine similarity.
• Training Objective: Unlike binary classification, the model uses cosine similarity regression. It minimizes the mean squared error between the predicted cosine similarity and a continuous target label (1.0 for parallels, 0.0 for non-parallels). This allows the model to learn a gradient of similarity rather than forcing a discrete decision boundary.
• Evaluation Protocol: The study utilizes a multi-seed validation framework (10 random seeds) across various training/testing splits (ranging from 50% to 90% training data). Performance is measured using distribution-based metrics—Wasserstein Distance (WD) and Overlap Coefficient (OVL)—which assess the separation of similarity distributions without relying on arbitrary thresholds. Retrieval performance is also evaluated using Recall@k.

Key Results

• Distributional Separation: MiqraBERT achieves a 2.7-fold improvement in Wasserstein Distance (from 0.276 to 0.751) compared to the pre-trained AlephBERT baseline. The ambiguous overlap region between parallel and non-parallel distributions is reduced from approximately 24% to 6%.
• Retrieval Performance:
  • Narrative Parallels: The model achieves a Recall@10 of 87.1% for narrative synoptic parallels (e.g., Chronicles vs. Samuel-Kings), successfully placing the true parallel in the top ten candidates for nearly 90% of queries.
  • Poetic Parallels: Performance on poetic parallelism is significantly lower, with a Recall@10 below 9%.
• Qualitative Improvements: The model successfully distinguishes between genuine parallels with theological or lexical variations (e.g., 2 Sam 24:1 vs. 1 Chr 21:1, where "YHWH" is replaced by "Satan") and unrelated texts that share only formulaic or thematic elements (e.g., shepherd imagery in Ps 23 vs. Ezek 34), which the pre-trained model incorrectly scored as highly similar.
• Optimal Configuration: A 70%-15%-15% train-validation-test split was identified as optimal, balancing high distributional separation with stable confidence intervals.

Significance and Claims
The paper claims that MiqraBERT demonstrates the viability of transfer learning from Modern Hebrew to Biblical Hebrew for semantic similarity tasks, even with a relatively small training set (~1,100 pairs). The study establishes that:

1. Anisotropy Correction: Contrastive regression finetuning effectively reshapes the embedding space, allowing the model to prioritize textual similarity over register-based clustering.
2. Genre-Dependent Limitations: The model's utility is currently confined to narrative textual reuse. The failure to detect poetic parallelism is attributed to the nature of poetic correspondence (often lacking shared lexemes) and the limitations of co-occurrence-based embeddings, rather than a flaw in the finetuning process itself.
3. Evaluation Rigor: The paper argues that distribution-based metrics (WD, OVL) combined with multi-seed validation provide a more honest and rigorous assessment of model behavior on small datasets than single-split classification metrics.

The authors conclude that while MiqraBERT is a reliable tool for detecting narrative synoptic parallels, its application to poetic or highly abstract intertextual relationships remains an open challenge requiring further methodological development. The model is publicly available for use in biblical scholarship.