💬 NLP

LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotations

This paper proposes LLM-based Multi-Reference Evaluation (LMRE), a scalable method that generates multiple valid phrasings to overcome the limitations of single-reference evaluation and aligns more closely with human judgment for assessing phrase break annotations.

Original authors: Younghan Park, Hoyeon Lee, Hawon Jeong, Jong-Hwan Kim

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Younghan Park, Hoyeon Lee, Hawon Jeong, Jong-Hwan Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to read a story out loud. For the robot to sound natural, it needs to know exactly where to pause, just like a human does. These pauses are called "phrase breaks." If the robot pauses in the wrong spot, the sentence might sound confusing or robotic.

The problem is: How do we know if the robot's pauses are good?

The Old Way: The "One Right Answer" Trap

Traditionally, to check if the robot is doing a good job, researchers used a method called Single-Reference Evaluation.

Think of this like a strict teacher who has a single, perfect answer key.

The Scenario: You ask the teacher, "Is this sentence broken up correctly?"
The Problem: In real life, there isn't just one way to pause a sentence. You could pause after the first word, or after the second, and both might sound perfectly natural.
The Flaw: The "One Right Answer" teacher only has one specific way written in their answer key. If the robot pauses slightly differently than that one specific key, the teacher marks it wrong, even if the robot sounded great.
The Human Alternative: You could hire a human to listen to the robot. Humans are flexible and understand that multiple pauses can be correct. But hiring humans is slow, expensive, and hard to scale up for thousands of sentences.

The New Solution: LMRE (The "Creative Librarian")

The authors of this paper propose a new method called LMRE (LLM-based Multi-Reference Evaluation). They use a Large Language Model (LLM)—a super-smart AI—to act as a Creative Librarian.

Here is how it works:

The Setup: Instead of asking the AI for just one answer, the researchers give it a few examples of good pauses (like showing it a few books with good chapter breaks).
The Magic: The AI uses these examples to generate many different, valid ways to pause the same sentence. It creates a "library" of correct answers, not just a single one.
The Check: When the robot's pauses are tested, the system checks: "Does the robot's pause match any of the many valid ways in our library?"
The Result: If the robot's pause matches even one of the valid options, it gets a "Pass."

Why This Matters (The Analogy)

Imagine you are judging a cooking competition where the dish is "Pasta."

Single-Reference Evaluation is like a judge who says, "The only correct pasta is spaghetti with tomato sauce." If you make penne with pesto, you fail, even though it's delicious.
Human Judgment is like having a panel of 100 food critics taste every dish. It's accurate, but it takes forever and costs a fortune.
LMRE is like a judge who says, "I know there are many ways to make great pasta. I will generate a list of 20 delicious variations. If your dish matches any of those 20, you pass."

What They Found

The researchers tested this on 1,356 Korean sentences (the "testbed"). They compared their new "Creative Librarian" method against the old "One Right Answer" method and actual human listeners.

Better Agreement with Humans: The LMRE method agreed much more with human listeners than the old method did. It stopped rejecting good pauses just because they didn't match a single, rigid answer key.
Scalable and Fast: Unlike hiring 100 humans, the AI can do this instantly and cheaply.
Handles Variety: It works well even when sentences are long and complex, where there are many ways to pause naturally.

The Bottom Line

The paper claims that by using AI to generate multiple valid options instead of forcing a single "gold standard," we can evaluate speech systems more fairly and accurately. It solves the problem of being too strict (rejecting good answers) while avoiding the cost of hiring humans for every single test. It's a way to make speech technology sound more natural, faster and cheaper.

Technical Summary: LLM-Based Multi-Reference Evaluation for Phrase Break Annotations

Problem Statement

Reliable evaluation of phrase break annotations is critical for Text-to-Speech (TTS) systems, as prosodic boundaries directly influence speech clarity and naturalness. However, existing evaluation frameworks suffer from two primary limitations:

Single-Reference Evaluation: This conventional approach assumes a one-to-one correspondence between an utterance and a unique "gold" annotation. Since prosodic phrasing is inherently one-to-many (multiple valid phrasings exist for a single sentence), this method risks incorrectly rejecting valid alternative annotations that differ from the specific reference. Furthermore, it lacks scalability, as the reference lookup must be manually expanded for every new utterance.
Human Judgment: While flexible enough to recognize multiple valid phrasings, human evaluation is labor-intensive, unscalable, and requires significant linguistic expertise, making it impractical for large-scale or iterative system development.

Previous attempts to use Large Language Models (LLMs) as judges have shown limited effectiveness for this specific task, often failing to capture subtle prosodic variations.

Methodology: LLM-Based Multi-Reference Evaluation (LMRE)

The authors propose LMRE, a framework designed to model the one-to-many nature of prosodic phrasing. Unlike "black-box" LLM-as-a-judge approaches, LMRE constructs a reusable, deterministic multi-reference lookup to enable reproducible evaluation.

Core Components:

Multi-Reference Lookup Construction: Instead of a single gold standard, LMRE uses an LLM to generate a set of $k$ plausible reference annotations ( $R(W) = \{B^{(1)}_r, \dots, B^{(k)}_r\}$ ) for a given utterance $W$ .
Diversification via Few-Shot Prompting: The LLM is prompted with a few-shot pool ( $P_{FS}$ ) of existing annotations. To ensure diversity, the system resamples few-shot examples for each query and performs multiple iterations ( $N_{iter}$ ).
Deterministic Filtering: To mitigate noise and spurious outputs, the system retains only those annotations generated more than $N_{iter}/10$ times across iterations.
Evaluation Logic: A hypothesis annotation $B_h$ is accepted if its similarity score (using Exact Match or F1) with at least one reference in the generated set exceeds a threshold $\theta$ .
Implementation: The system employs batch prompting (batch size 32) to reduce latency and cost, using temperature 0.0 for reproducibility.

Experimental Setup

The authors validated LMRE on a curated Korean testbed comprising 1,356 phrase break annotations.

Data Diversity: The dataset covers five annotation strategies (rule-based, comma-driven, audio-driven, text-driven by experts, and synthetic) and eleven configurations.
Utterance Lengths: Data is stratified into short (<7 words), medium (7–10 words), and long (≥11 words) categories.
Baselines: Comparisons were made against Single-Reference Evaluation (using two human expert references) and Human Judgment (from four independent judges).
LLM Configurations: Experiments utilized GPT-4.1, Claude Sonnet, and a combined approach, varying the few-shot pool size ( $|P_{FS}|$ ) and iteration counts ( $N_{iter}$ ).

Key Results

Alignment with Human Judgment: LMRE demonstrates significantly stronger alignment with human judgments than single-reference evaluation.
- Acceptance Behavior: Single-reference evaluation systematically under-accepts valid annotations, showing an average gap of 26.59% from human judgment in the "Acceptable" score group. LMRE (Combined setting) reduced this gap to 7.04%, and to 1.75% for high-quality (score 5) annotations.
- Correlation: LMRE achieved higher Pearson ( $r$ ) and Spearman ( $\rho$ ) correlations with human scores. In the best configuration (Combined, $N_{iter}=40$ , $|P_{FS}|=128$ ), the correlation with human scores reached $r=0.68$ and $\rho=0.70$ , compared to $r=0.56$ and $\rho=0.53$ for single-reference evaluation.
Metric Sensitivity: The F1 score metric consistently showed better alignment with human judgment than Exact Match (EM) across all methods. This is attributed to F1's ability to reward partial overlaps, accommodating the subtle variations inherent in prosodic phrasing.
Scalability and Generalization: LMRE maintained strong performance even with small few-shot pools ( $|P_{FS}|=128$ ) and generalized well to unseen utterances without additional human labeling effort. The performance gap between different few-shot pools narrowed as the pool size and iteration count increased, indicating robustness to inter-annotator variability.
Length Dependency: The relative improvement of LMRE over single-reference evaluation was most pronounced for longer utterances under the EM metric, supporting the intuition that longer sentences permit greater phrasing variability.

Significance and Contributions

The paper claims that LMRE offers a scalable and reliable framework for prosodic evaluation in TTS systems by effectively addressing the trade-off between scalability and multi-reference support. The specific contributions are:

Framework Proposal: Introducing LMRE, which overcomes the one-to-one assumption of conventional methods by modeling the one-to-many nature of prosodic phrasing through LLM-generated references.
Comprehensive Testbed: Constructing a diverse Korean testbed of 1,356 annotations spanning five strategies and eleven configurations to validate the method under heterogeneous quality conditions.
Empirical Validation: Demonstrating that LMRE achieves substantially improved alignment with human judgments compared to single-reference evaluation while remaining fully automatic.

The authors conclude that LMRE highlights the potential of LLMs for evaluation in the speech domain, providing a solution that is both efficient (automated) and robust (capable of handling multiple valid phrasings).