Gradient boundaries through confidence intervals for forced alignment estimates using model ensembles

1. Problem Statement

Forced alignment is a standard tool in phonetics and speech science used to align audio signals with orthographic and phonetic transcriptions. However, existing tools (e.g., Montreal Forced Aligner, standard MAPS) typically provide only point estimates for segment boundaries (e.g., a single timestamp at 123.4 ms).

This approach presents two major limitations:

• Theoretical Mismatch: Speech production is inherently gradient and overlapping (as described in Articulatory Phonology), yet standard data formats like Praat TextGrids enforce discrete, non-overlapping intervals. Point estimates fail to capture the natural uncertainty and transition periods between segments.
• Lack of Uncertainty Quantification: Traditional aligners do not quantify the confidence or uncertainty of a boundary placement. This makes it difficult for researchers to identify ambiguous boundaries, detect outliers, or prioritize segments for manual review.

2. Methodology

The author proposes a method to generate gradient boundary regions by leveraging model ensembles and order statistics to derive confidence intervals.

A. The Base System (MAPS)

The method extends the Mason-Alberta Phonetic Segmenter (MAPS), a deep recurrent neural network (RNN) trained as a phone recognizer.

• Architecture: 3 LSTM layers with 128 units each.
• Input: 13 MFCCs (with log energy replacing the zeroth coefficient) plus delta and delta-delta features.
• Training Data: A mix of the TIMIT and Buckeye corpora (American English).
• Ensemble Construction: The system was trained 10 separate times with different random initializations. These 10 models form the ensemble.

B. Generating Gradient Boundaries

Instead of running a single model, the alignment process is repeated for all 10 models in the ensemble for a given utterance.

1. Point Estimate: For each segment boundary, the median of the 10 models' predicted timestamps is selected as the final point estimate. The median is chosen over the mean to ensure robustness against outliers (cascading errors).
2. Gradient Region (Confidence Interval): To define the boundary region, the author uses order statistics on the sample of 10 predictions.
   • The 2nd lowest and 9th highest values from the 10 predictions are selected as the lower and upper bounds.
   • This construction yields an approximate 97.85% confidence interval for the boundary location.
   • Note: This interval represents the uncertainty of the model's approximating function, not necessarily a "true" phonetic boundary, as true boundaries are theoretically undefined in continuous time.

C. Output Formats

To accommodate the new data structure, the system outputs:

• JSON Files: Contain segment data with explicit start/end bounds for the gradient regions, allowing for overlapping events.
• Statistical Tables: For programmatic analysis.
• Praat TextGrids (with caveats): Uses a "point tier" to represent the edges of the boundary regions, though the author notes TextGrids are not an ideal format for representing overlapping uncertainty.

3. Key Contributions

• Gradient Representation: Introduces a method to move from discrete point estimates to gradient boundary regions, better reflecting the continuous nature of speech transitions.
• Uncertainty Quantification: Provides a statistical measure (confidence interval width) of model uncertainty, allowing researchers to identify "risky" alignments that require manual review.
• Robustness via Ensembling: Demonstrates that using the median of an ensemble reduces the impact of outlier predictions caused by cascading alignment errors.
• Theoretical Alignment: Bridges the gap between computational alignment outputs and phonetic theories that acknowledge overlap and gradience (e.g., Browman & Goldstein, 1992).

4. Results

The system was evaluated on the TIMIT and Buckeye corpora.

• Accuracy Improvement: The ensemble approach showed a slight overall improvement in boundary placement accuracy compared to a single model (Kelley et al., 2024).
  • Test Set Median Absolute Error: Reduced from ~7.14 ms (single model) to 7.12 ms (ensemble).
  • The improvement is attributed to the median's resistance to outliers.
• Boundary Region Analysis:
  • Ambiguity Correlation: The width of the confidence intervals correlates with phonetic intuition. Sequences known to be difficult to segment (e.g., vowel-vowel, approximant-vowel) exhibited wider boundary regions (higher uncertainty).
  • Clearer Transitions: Sequences like stop-stop or stop-silence showed narrower regions (lower uncertainty).
  • Outlier Sensitivity: The mean width of regions was significantly larger than the median width, indicating that a few poor alignments in the ensemble can expand the region, but the median width remains stable (often within one 10ms frame).
• Coverage: Approximately 34% of the reference "gold standard" boundaries fell within the calculated 97.85% confidence intervals. The author cautions that this metric reflects the model's internal consistency rather than a guarantee of phonetic correctness.

5. Significance and Future Directions

• Scientific Rigor: The method treats alignment as a statistical estimation problem, providing a "measure of model certainty" rather than a false sense of precision. This allows for more responsible data annotation and analysis.
• Practical Utility: Researchers can filter data by confidence interval width to automatically flag segments for manual correction, improving the efficiency of corpus creation.
• Format Limitations: The paper highlights the inadequacy of current formats (TextGrids) for representing overlapping or gradient data and advocates for new formats (like JSON) or extensions to existing ones.
• Computational Cost: The primary drawback is a 10x increase in computation time due to running 10 models. The author suggests future work should explore parallelization or Bayesian neural networks to estimate uncertainty more efficiently.

In conclusion, this work provides a statistically grounded framework for acknowledging the inherent uncertainty in speech segmentation, offering a more realistic and useful tool for phonetic research than traditional point-estimate aligners.