findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embedding

1. Problem Statement

Syllable-level units are highly effective for spoken language modeling and unsupervised word discovery because they offer a compact representation that reduces sequence length while preserving linguistic structure. However, research in this domain is currently fragmented:

• Lack of Standardization: Implementations of syllabification methods (both classical signal-processing and modern neural approaches) are scattered across isolated scripts, often tuned to specific datasets.
• Reproducibility Issues: It is difficult to reproduce prior results or compare different methods fairly because they lack a common interface and evaluation protocol.
• Inability to Isolate Variables: Researchers cannot easily disentangle whether performance gains come from the representation (e.g., the feature extractor) or the segmentation algorithm (e.g., the clustering method), as most tools treat these as fixed, monolithic pipelines.
• Resource Disparity: While high-resource languages benefit from self-supervised learning (SSL) models, low-resource and under-documented languages often rely on classical methods that are hard to integrate into modern pipelines.

2. Methodology: The findsylls Toolkit

The authors introduce findsylls, a modular, language-agnostic Python toolkit designed to unify classical and neural syllabification under a single framework. The toolkit decouples the processing pipeline into three interoperable modules:

A. Envelope Computation (Classical Cues)

This module generates amplitude envelopes to capture syllabic rhythm using classical signal processing. It supports:

• RMS energy and low-pass filtered energy.
• Hilbert-transform envelopes.
• Spectral Band Subtraction (SBS) envelopes.
• Neurophysiologically inspired theta (oscillator) envelopes.
• Output: Smoothed energy contours where syllable nuclei are detected as local maxima.

B. Feature Extraction (Representations)

This module handles frame-level feature extraction, decoupled from segmentation logic. It supports:

• Classical: MFCCs and log mel-spectrograms.
• Self-Supervised (SSL): Encoders like HuBERT, VG-HuBERT, and Sylber, which have been shown to encode syllabic structure.
• Flexibility: Extractors return high-dimensional feature matrices without imposing a specific segmentation strategy.

C. Segmentation Algorithms

This module applies algorithms to either envelopes or feature sequences to output syllable intervals:

• Peak Detection: Implements Billauer's peakdetect algorithm on envelopes (or pseudo-envelopes derived from features).
• Greedy Cosine Merging: Merges adjacent frames based on cosine similarity thresholds.
• MinCut: An optimized dynamic programming algorithm applied to feature self-similarity matrices (featSSM).
• CLS-Thresholding: Uses attention weights from [CLS] tokens.

Key Innovation: Modular Recombination
The toolkit allows researchers to "mix and match" components. For example, one can apply the peakdetect algorithm (usually for envelopes) to a Sylber cosine-similarity trace or a VG-HuBERT feature matrix. This enables controlled ablation studies to determine the specific contribution of the representation versus the segmentation algorithm.

D. Evaluation Pipeline

The toolkit includes a unified evaluation framework that quantifies performance against time-aligned annotations (TextGrids) at three granularities:

1. Nuclei: Detection of syllable peaks.
2. Boundaries: Detection of onset/offset.
3. Spans: Recovery of the full syllable interval.
   Metrics include Precision, Recall, and F1 scores, alongside token rate (tokens/second) and computational cost (Inverse Real-Time Factor).

3. Key Contributions

1. Unified Open-Source Toolkit: findsylls is the first tool to unify classical amplitude-envelope algorithms and recent neural syllabifiers under a single interface.
2. Reusable Evaluation Pipeline: Provides standardized tools for aggregating nuclei, boundary, and span detection metrics across large-scale corpora.
3. Systematic Benchmarking: Conducts a comprehensive comparison of envelope-based vs. SSL-based methods across diverse datasets (English, Spanish, and the under-documented Kono language).
4. Shared Infrastructure: Positions the toolkit as a community resource to facilitate reproducible research and method recombination across high-resource and low-resource settings.

4. Experimental Results

The authors benchmarked 10 configurations (classical baselines and neural setups) across 7 corpora (including adult read speech, child-directed speech, and fieldwork data).

• Accuracy vs. Granularity: Nuclei detection is significantly easier than full span recovery. For instance, the SBS envelope achieved a 91.2% F1 for nuclei but dropped to 37.1% F1 for full spans, indicating that errors compound when reconstructing full intervals.
• Best Performers:
  • Nuclei: Sylber with cosine-threshold segmentation achieved the highest nuclei F1 (93.3%).
  • Boundaries: VG-HuBERT with MinCut achieved the best boundary F1 (65.0%).
• Impact of Modular Recombination:
  • Applying peakdetect to Sylber's cosine-similarity envelope improved boundary F1 from 63.1% to 69.9% and span F1 from 45.7% to 47.0%.
  • Swapping the cue for VG-HuBERT (using local cosine similarity instead of featSSM) improved span recovery while maintaining competitive boundary performance.
  • Conclusion: Treating syllabifiers as fixed pipelines hides potential gains; recombining components yields measurable improvements.
• Efficiency Trade-offs:
  • Envelope-based methods are computationally efficient (high RTFx, e.g., 684x) but offer lower boundary accuracy.
  • Neural methods incur higher computational costs (lower RTFx, e.g., 6x–72x) but generally provide better boundary and span recovery.
  • All methods produce syllabic tokens at 3.1–5.8 tokens/second, a massive reduction compared to standard 50–100 Hz frame-level features.

5. Significance

• Bridging the Resource Gap: By supporting both classical signal processing and SSL models, findsylls enables robust syllable-level research on under-documented languages (demonstrated here with Kono, a Central Mande language) without requiring massive pretraining datasets.
• Design Space vs. Fixed Method: The results challenge the notion of a single "best" syllabifier. Instead, syllable tokenization is a design space where researchers can optimize for specific constraints (e.g., streaming speed vs. boundary fidelity).
• Reproducibility: By standardizing the interface and evaluation metrics, the toolkit allows the community to fairly compare new algorithms and reproduce existing ones, accelerating progress in speech tokenization and unsupervised word discovery.

In summary, findsylls provides the necessary infrastructure to move syllable-level speech processing from fragmented, isolated experiments to a unified, reproducible, and modular scientific discipline.