💬 NLP

Fusing Stylometric and Embedding Systems to Estimate Authorship Likelihood Ratios in Japanese

This study pioneers the application of the likelihood ratio framework to Japanese authorship attribution by fusing stylometric features with embedding-based systems, demonstrating that this hybrid approach significantly improves discriminability and calibration compared to individual methods.

Original authors: Praju Ghatpande, Satoru Tsuge, Shunichi Ishihara, Wataru Zaitsu, Mitsuyuki Inaba

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Praju Ghatpande, Satoru Tsuge, Shunichi Ishihara, Wataru Zaitsu, Mitsuyuki Inaba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out who wrote a mysterious note. In the past, experts would look for specific "fingerprints" in the writing, like how often someone uses commas or specific short words. This is called stylometry.

However, in the digital age, we have powerful new tools called AI models (specifically Large Language Models) that can read a text and understand its "vibe" or context in a way that feels more like how a human reads, rather than just counting words.

This paper is about a team of researchers who asked a very specific question: Can we combine the old-school "word-counting" detective work with the new-school "AI vibe-checking" to get a better answer? And, crucially, can they do this for Japanese, a language that has never been tested with this specific legal framework before?

Here is the breakdown of their experiment using simple analogies:

1. The Goal: The "Likelihood Ratio" Scale

In court, experts don't just say, "I am 100% sure this is Author A." Instead, they use a scale called the Likelihood Ratio (LR).

Think of this like a weather forecast. Instead of saying "It will rain," they say, "It is 10 times more likely to rain than not."
The researchers wanted to build a system that gives this kind of "odds" for Japanese text.
They wanted to see if mixing the "old school" methods (counting characters) with the "new school" methods (AI embeddings) would make the weather forecast more accurate.

2. The Ingredients: Two Kinds of "Detectives"

The researchers set up two teams of detectives to analyze Japanese blog posts (about 1,000 characters long):

Team A (The Stylometric Features): These are the traditional detectives. They count specific things:
- Character Bigrams: How often do two specific characters appear next to each other? (e.g., how often does "te" follow "shi"?)
- Function Words: How often do they use tiny words like "the," "of," or Japanese particles like "no" or "ga"?
- Comma Usage: Where do they put commas? (In Japanese, comma placement is very personal and artistic).
- Part-of-Speech: What kind of words are they using? (Nouns, verbs, adjectives).
- Character Types: Do they mix Kanji, Hiragana, and Katakana in a unique way?
Team B (The Embedding Systems): These are the AI detectives. They use pre-trained AI models (like a super-smart robot that has read millions of books) to turn the text into a mathematical "fingerprint" (a vector).
- Word-Level AI: Looks at whole words.
- Character-Level AI: Looks at individual characters.

3. The Experiment: The "Fusion" Kitchen

The researchers didn't just let Team A and Team B work separately. They put them in a kitchen to fuse their opinions.

They tried mixing Team A's results with Team B's results using a mathematical recipe called Logistic Regression.
Think of this like a jury deliberation. Instead of one juror deciding, they combine the votes of 5 traditional jurors and 2 AI jurors to reach a single, stronger verdict.

4. The Results: The "Super-Detective"

The paper claims that the Fused System (the jury) was the best detective of all. Here is what they found:

The AI was strong: The AI-only team was actually better than the traditional word-counting team on its own.
The Fusion was strongest: When they combined the AI and the traditional methods, the system became even better.
- Better Accuracy: It was much better at telling the difference between two different authors.
- Better Calibration: The "odds" it gave were more reliable. It didn't overreact or underreact.
- The "Sweet Spot": The best combination didn't use every single feature. It used a specific mix: Character bigrams, Comma bigrams, Character types, and both Word and Character AI embeddings.

5. The "Real World" Test

To prove it worked, they looked at two specific examples:

Case 1 (Same Author): They found two blog posts by the same person who used a very strange, repetitive style (lots of commas and specific weird phrases). The fused system gave a massive "odds" that these were written by the same person.
Case 2 (Different Authors): They found two posts by different people. One was weird and chaotic; the other was standard and calm. The fused system correctly said, "These are definitely different people," with a very strong negative score.

6. The Bottom Line

This paper is the first time this specific legal framework (Likelihood Ratios) has been successfully applied to Japanese text.

They proved that:

You can use this legal math on Japanese.
Mixing the "old school" counting methods with "new school" AI methods creates a system that is more accurate and reliable than using either one alone.

What they did NOT claim:

They did not say this system is ready for use in real courtrooms tomorrow.
They did not test it on emails or social media (only blogs).
They did not claim it works for all types of Japanese writing, just the specific blog excerpts they tested.

In short: Mixing the old and the new makes for a smarter, more reliable way to guess who wrote a Japanese text.

Technical Summary: Fusing Stylometric and Embedding Systems to Estimate Authorship Likelihood Ratios in Japanese

Problem Statement
The likelihood ratio (LR) framework is the established standard for evidential analysis in forensic sciences, offering a logically sound method to quantify the strength of evidence under competing prosecution and defense hypotheses. However, its application in forensic text comparison (FTC) has been almost exclusively confined to English-language texts. Furthermore, while authorship attribution has traditionally relied on discrete stylometric features (e.g., word counts, character n-grams), the advent of pre-trained large language models (LLMs) has introduced contextual embedding approaches. Although ensemble learning (fusion) is known to improve performance in natural language processing, no prior study has attempted to fuse traditional stylometric-feature systems with LLM-based embedding systems within the LR paradigm, particularly for non-English languages with distinct writing systems like Japanese.

Methodology
This study addresses these gaps by applying the LR framework to Japanese digital texts (approximately 1,000-character excerpts from Yahoo! Blogs). The research design involves the following components:

Dataset: A corpus of 4,574 blog posts from 2,287 unique authors, derived from the Balanced Corpus of Contemporary Written Japanese (BCCWJ). The data was partitioned into five batches for five-fold cross-validation, utilizing Training, Validation, Test, and Calibration sets.
System Architecture: Seven distinct systems were developed and evaluated:
- Stylometric-Feature Systems (5 systems): These utilize a Dirichlet-multinomial model to handle count-based, discrete inputs. The features include Character bigrams, Function-word unigrams, Part-of-Speech (POS) bigrams, Comma bigrams (comma and preceding character), and Character types (hiragana, katakana, kanji, etc.).
- Embedding-Based Systems (2 systems): These utilize pre-trained RoBERTa models fine-tuned as Siamese networks. One system uses word-level tokenization (word-embeddings), and the other uses character-level tokenization (character-embeddings). Scores are derived using cosine distance.
Scoring and Calibration:
- Stylometric systems generate scores based on the ratio of probabilities under the prosecution vs. defense hypotheses.
- Embedding systems generate similarity scores based on cosine distance.
- All raw scores are converted to log-likelihood ratios (log LR) using logistic regression calibration to ensure the outputs are well-calibrated evidential values.
Fusion Strategy: The study employs logistic regression to fuse the scores from multiple systems. This method simultaneously performs fusion and calibration, accounting for correlations between the different evidence sources. All possible combinations of two to seven systems were tested.

Key Results
The study evaluated system performance using the log-likelihood ratio cost ( $C_{llr}$ ), where lower values indicate better discrimination and calibration.

Individual System Performance: Among single systems, the word-embedding system (System 6) achieved the best performance ( $C_{llr} = 0.38757$ ), outperforming the best stylometric system (Character bigrams, $C_{llr} = 0.47115$ ).
Fusion Performance:
- Stylometric Fusion: Fusing Character-bigrams, Comma-bigrams, and POS-bigrams yielded the best stylometric-only result ( $C_{llr} = 0.43964$ ).
- Embedding Fusion: Fusing word and character embeddings improved performance to $C_{llr} = 0.36186$ .
- Heterogeneous Fusion: The optimal result was achieved by fusing five systems: Character-bigrams, Comma-bigrams, Character types, Word-embeddings, and Character-embeddings. This fused system achieved a $C_{llr}$ of 0.32484.
Impact of Fusion: The best-performing fused system demonstrated:
1. Excellent calibration (calibration loss $C_{llr}^{cal} \approx 0.03$ ).
2. Increased magnitudes of consistent-with-fact LRs (stronger evidence when authors match).
3. Decreased magnitudes of contrary-to-fact LRs (weaker, less misleading evidence when authors do not match).
4. Improved overall discriminability compared to any single system or homogeneous fusion.

Significance and Claims
The authors claim this study is the first to apply the LR-based forensic text comparison framework to Japanese digital texts. The primary significance lies in demonstrating the feasibility of the LR framework for a language with a fundamentally different writing system (Japanese) compared to English.

Furthermore, the paper establishes that fusing heterogeneous systems—combining traditional, discrete stylometric features with modern, contextual LLM embeddings—yields superior performance within the LR paradigm. The results suggest that while individual systems have strengths, their combination captures complementary dimensions of authorship (e.g., raw character sequences vs. syntactic patterns vs. semantic context), leading to more robust and discriminative forensic evidence. The study concludes that the LR framework is viable for Japanese FTC and that fusion across diverse modeling approaches is a critical strategy for enhancing evidential strength.