Normalisation-Based Likelihood Ratio Estimation for Forensic Authorship Verification

Technical Summary: Normalisation-Based Likelihood Ratio Estimation for Forensic Authorship Verification

Problem Statement
Authorship Verification (AV) in forensic contexts aims to determine if two texts share a common author. The legal standard for evaluating such evidence is the Likelihood Ratio (LR) framework, which quantifies the relative probability of the evidence under competing prosecution ($H_p$) and defence ($H_d$) hypotheses. While score-based AV methods exist, they typically output uncalibrated similarity scores or raw likelihood ratios that do not reliably reflect empirical frequencies. To render these scores interpretable as valid forensic evidence, a separate calibration model—most commonly logistic regression—is required. This calibration process necessitates large, case-specific datasets (same-source and different-source text pairs) that are often difficult, time-consuming, and expensive to compile. Furthermore, the selection of calibration data involves subjective expert decisions (e.g., anchoring strategies) that require further validation, adding complexity and opacity to the forensic process.

Methodology
This study investigates whether the raw scores produced by LambdaG, a state-of-the-art binary extrinsic AV method based on n-gram language modelling, can be transformed into well-calibrated LRs without a separate training model. The authors hypothesize that the raw LambdaG score ($\lambda_G$) suffers from "score inflation" due to the method's reliance on summing independent token-level contributions. This aggregation assumes conditional independence between features, an assumption that natural language violates due to repetition. Consequently, longer or highly repetitive texts accumulate disproportionately high scores, overstating evidential strength.

To address this, the paper proposes two novel normalisation techniques that scale the raw $\lambda_G$ score directly:

1. Square Root Correction: This method scales the score by the square root of the number of tokens in the questioned document ($N(Q)$). It is grounded in the statistical principle that variance in aggregated features often grows proportionally to $\sqrt{N}$. The formula is:
   $$\Lambda_G = \frac{\lambda_G}{\sqrt{N(Q)}}$$
   This aims to mitigate the inflation caused by text length, acknowledging that additional tokens in long texts contribute diminishing independent evidential weight.

2. Hapax Correction: This method incorporates a measure of lexical diversity and redundancy. It scales the score by the ratio of hapax legomena (words appearing exactly once, $V_1(Q)$) to the total token count ($N(Q)$). The formula is:
   $$\Lambda_G = \lambda_G \times \frac{V_1(Q)}{N(Q)}$$
   This correction targets the redundancy of repeated patterns. In texts with low diversity (high repetition), the ratio is low, reducing the score to counteract the artificial inflation. This aligns with Herdan-Heap's Law, which describes the sub-linear growth of vocabulary as text length increases.

Experimental Protocol
The authors evaluated these corrections against the standard baseline of logistic regression calibration across 15 diverse corpora (including academic papers, news, blogs, reviews, forums, emails, chat logs, and tweets). The evaluation utilized the Log-Likelihood Ratio Cost ($C_{llr}$) metric, where lower values indicate better calibration and discrimination.

• Data: The study included 13,146 authorship verification problems.
• Variables: For a subset of corpora (Avocado, Twitter, Bolt), the authors systematically varied the lengths of the questioned text ($Q$) and the known-author text ($K$) from 100 to 9,500 tokens to test robustness across text lengths.
• Baseline: Logistic regression models were trained on 40–50% of the data for each corpus and tested on the remainder. The proposed normalisations were applied directly to the raw scores without training.

Key Results

• Overall Performance: Across the 15 corpora, the Hapax Correction outperformed logistic regression calibration in 45.4% of tests. In the remaining cases where logistic regression performed better, the Hapax Correction's performance was within 5% of the baseline in 56.91% of instances.
• Text Length Robustness:
  • On longer texts (e.g., Avocado and Twitter with $K > 5,500$ tokens), logistic regression often achieved slightly lower peak $C_{llr}$ values, likely due to more stable parameter estimation with larger datasets. However, the Hapax Correction consistently produced well-calibrated LRs ($C_{llr} < 0.1$) in these conditions.
  • On shorter texts, the Hapax Correction frequently outperformed logistic regression, demonstrating greater robustness when data is scarce.
  • The Square Root Correction was generally outperformed by both the Hapax Correction and logistic regression (losing in over 70% of tests), though it still offered improvements over raw scores.
• Anomalies: Logistic regression calibration occasionally produced misleading results ($C_{llr} > 1$) on datasets with small calibration sets (e.g., Twitter, Avocado), suggesting sensitivity to noise and overfitting. The proposed normalisations exhibited lower variability and greater stability.
• Bolt Corpus (Short Texts): On the SMS/Chat corpus with very short texts (100–500 tokens), the Square Root Correction outperformed the Hapax Correction in 84% of conditions, likely because the Hapax ratio becomes noisy and unreliable in extremely short samples where vocabulary growth is rapid.

Significance and Claims
The paper claims that these normalisation techniques offer a functionally equivalent, yet simpler alternative to logistic regression calibration for the LambdaG method. The primary significance lies in the elimination of the calibration model, which:

1. Reduces Data Requirements: Removes the need for large, case-specific training datasets that are difficult to obtain.
2. Increases Transparency: Replaces the "opaque" logistic regression transformation with simple, interpretable mathematical corrections grounded in lexical statistics and usage-based linguistic theory.
3. Enhances Accessibility: Simplifies the forensic workflow, making the derivation of calibrated LRs more accessible to experts and easier to explain to legal fact-finders.

The authors conclude that while the Hapax Correction does not universally dominate logistic regression, its ability to maintain comparable performance across varying text lengths and registers—without the computational and data burdens of training a model—supports its adoption in forensic settings for English-language texts. The study posits that these corrections effectively model the diminishing evidential value of repeated linguistic patterns, offering a more scientifically valid approach than standard calibration for this specific class of AV methods.