💬 NLP

FreyaTTS Technical Report

Freya-TTS is a compact, tokenizer-free, Turkish-first text-to-speech model that leverages a non-autoregressive conditional flow-matching Diffusion Transformer in a continuous latent space to achieve high-quality, real-time conversational synthesis with superior efficiency and accuracy compared to larger open-source systems.

Original authors: Ahmet Erdem Pamuk, Ömer Yentür, Ahmet Tunga Bayrak, Yavuz Alp Sencer Öztürk, Mustafa Yavuz

Published 2026-07-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmet Erdem Pamuk, Ömer Yentür, Ahmet Tunga Bayrak, Yavuz Alp Sencer Öztürk, Mustafa Yavuz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a robot that speaks Turkish. Most people try to build this robot by teaching it a massive dictionary of sounds, breaking every word down into tiny, pre-defined pieces (like Lego bricks), and then stacking those bricks together one by one, left to right. This is how many current "state-of-the-art" speech robots work. They are huge, they take forever to build, and if they stumble on a long sentence, they often trip over their own feet, mixing up numbers or getting the order wrong.

The FreyaTTS team decided to try a completely different, much smaller, and smarter approach. They didn't build a giant dictionary or a brick-by-brick stacker. Instead, they built a 183.2-million-parameter "imagination engine" that learns to speak Turkish by listening to audio and guessing the whole sentence at once, like a jazz musician improvising a full solo rather than reading sheet music note by note.

Here is how their magic trick works, broken down into three simple parts:

1. The Frozen Blueprint (The "AudioVAE2")

Think of the FreyaTTS team as architects who found a perfect, pre-built house (called AudioVAE2) that already knows how to turn a sketch into a beautiful 48 kHz sound wave. This house is so good that the team decided never to touch it. They froze it in place.

Because they didn't have to waste time teaching the robot how to make sound waves, they could use 100% of their brainpower to teach the robot what to say. They didn't need a translator (a phonemizer) to turn letters into sounds, and they didn't need to break words into tiny sound tokens. They just fed the robot raw Turkish text (92 symbols, including special letters like ç, ğ, ı, ö, ş, ü) and let it figure out the pronunciation on its own. It's like teaching a child to speak by talking to them, rather than giving them a textbook on phonetics.

2. The "All-at-Once" Magic (Non-Autoregressive)

Most speech robots are like a slow typist: they type one letter, then the next, then the next. If they make a mistake early on, the whole sentence gets garbled. This is called "autoregressive" generation.

FreyaTTS is different. It's like a painter who sees the whole canvas and paints the entire picture in one go. It uses a conditional flow-matching Diffusion Transformer to predict the entire sentence's sound pattern in parallel.

The Analogy: Imagine you are guessing a friend's voice. Instead of guessing one word at a time (which might lead you to guess the wrong word and ruin the sentence), FreyaTTS guesses the entire sentence's rhythm and tone at once.
The Result: It avoids the "left-to-right error accumulation." If you ask it to say a long list of numbers, it doesn't get confused halfway through. It predicts the whole duration and the whole sound pattern simultaneously.

3. The "Voice Lock" (Making it Consistent)

When they first trained this robot from scratch, it was a bit of a chameleon. Every time you asked it to say "Hello," it would sound like a different person. One time it might sound like a deep-voiced man, the next like a high-pitched child. This is because the robot was learning from a mix of many voices and didn't have a specific identity.

To fix this, the team did a two-step "voice lock":

Stage 1: They taught the robot to mimic one specific person (a single professional voice talent). This collapsed the robot's voice pitch variation from a wild 74.9 Hz (a huge range) down to a steady 5.0 Hz. Now, it sounds like the same person every time.
Stage 2: They realized the robot was bad at short phrases (like "Yes" or "No"). So, they gave it extra practice specifically on short, one-word and two-word sentences.

The Results: Small but Mighty

The team tested their robot against other famous open-source speech models. Here is what they found:

Size: FreyaTTS is tiny (183.2M parameters) compared to giants like XTTS-v2 (470M) or F5-TTS (336M).
Accuracy: On a test of 495 Turkish sentences, FreyaTTS made fewer mistakes than the bigger models. It achieved a Word Error Rate (WER) of 8.0% and a Character Error Rate (CER) of 3.0%.
- Note: The bigger models got 11.1% and 24.3% respectively.
Speed: Because it doesn't have to wait for one word to finish before starting the next, it is incredibly fast. On a standard consumer graphics card, it runs at a real-time factor of 0.11. This means it can generate 10 seconds of audio in just 1.1 seconds.
Laptop Power: It's so efficient that it can run faster than real time on a laptop CPU (like an Apple M3), making it perfect for phones or small devices.

What They Explicitly Say It's NOT

The paper is very clear about what FreyaTTS is not:

It is not a "zero-shot" cloner that can mimic any voice you give it a clip of. It is a single-voice model. You get the one voice it was trained on, and that's it.
It is not built on a massive multilingual foundation that tries to speak 50 languages at once. It is a Turkish-first model, specialized for one language.
It does not use a "phonemizer" (a rule-based system to convert letters to sounds). It learns the sounds directly from the audio.
It is not perfect at reading numbers in their written form (like "1999"). The paper notes that you still need to expand numbers into their spoken form (like "one thousand nine hundred ninety-nine") before feeding them to the model, because the robot sometimes struggles with the duration of long digit strings.

How Sure Are They?

The team is very confident in these numbers because they didn't just guess; they measured everything.

They built a specific benchmark called Freya-TR-Eval with 495 sentences.
They ran the tests 10,000 times using a "bootstrap" method to make sure the results weren't just luck.
They compared their model against the exact same sentences using the exact same scoring rules (downsampling everything to 8 kHz so the playing field was level).
They even measured the "voice lock" stability, showing the pitch variation dropped from 74.9 Hz to 5.0 Hz after their training steps.

In short, FreyaTTS proves that you don't need a billion-dollar, multi-language monster to make great Turkish speech. You can build a small, specialized, "all-at-once" engine that runs on a laptop, sounds consistent, and speaks Turkish better than the giants.

Technical Summary: FreyaTTS

Problem Statement

The field of Text-to-Speech (TTS) has advanced significantly, yet it remains heavily skewed toward high-resource languages like English and Chinese. Mid-resource languages, specifically Turkish, suffer from a dual gap:

Linguistic Neglect: Large multilingual models treat Turkish as a marginal language among dozens, failing to account for its specific linguistic complexities, such as agglutinative morphology, vowel harmony, and the spoken-form expansion of numbers and acronyms. Existing baselines often rely on brittle, hand-built grapheme-to-phoneme (G2P) pipelines that struggle with these features.
Computational Inefficiency: State-of-the-art open systems often require multi-billion-parameter backbones and massive training corpora (hundreds of thousands to millions of hours), rendering them unsuitable for single-GPU serving or on-device deployment.

Furthermore, dominant paradigms face specific limitations:

Discrete-token systems (e.g., VALL-E, CosyVoice) rely on quantization that discards fine acoustic detail and often require multi-stage pipelines.
Autoregressive continuous-latent models (e.g., VoxCPM2) suffer from left-to-right error accumulation, leading to garbled numbers and words over long horizons.

Methodology

FreyaTTS introduces a compact, tokenizer-free, non-autoregressive (NAR) architecture designed specifically for Turkish. The system is built upon three core pillars:

1. Frozen Continuous-Latent Space

FreyaTTS utilizes the AudioVAE2 codec (from VoxCPM2) as a fixed, frozen component.

Architecture: The encoder maps 16 kHz waveforms to a 64-dimensional continuous latent space at 25 Hz. The decoder reconstructs audio at 48 kHz.
Strategy: By keeping the codec frozen, the model avoids learning waveform reconstruction. Instead, the entire 183.2M-parameter capacity is dedicated to learning the text-to-latent mapping. This allows the model to inherit 48 kHz reconstruction "for free" while operating in a continuous space that preserves fine acoustic details without discrete quantization.

2. Tokenizer-Free Non-Autoregressive Generation

Unlike autoregressive models that generate tokens sequentially, FreyaTTS employs a Conditional Flow-Matching Diffusion Transformer (DiT).

Input: The model accepts raw Turkish text via a 92-symbol character vocabulary (including specific Turkish characters like ç, ğ, ı, ö, ş, ü). It requires no phonemizer, G2P frontend, or discrete speech tokenizer.
Mechanism: The model learns pronunciation, including the spoken forms of numbers and acronyms, directly from audio. It predicts the duration of the utterance and denoises the entire latent sequence in parallel over a fixed number of ODE steps (32 steps).
Advantage: This parallel generation eliminates left-to-right error accumulation, a critical failure mode for digit strings in autoregressive systems.

3. Two-Stage Post-Training Recipe

To transform the speaker-agnostic pre-trained prior into a reliable production model, the authors employ a two-stage fine-tuning process:

Stage 1: Voice Lock (SFT v1): The model is fine-tuned on a single consented speaker. This "writes" the voice identity into the weights, collapsing the cross-generation fundamental frequency ( $F_0$ ) standard deviation from 74.9 Hz to 5.0 Hz.
Stage 2: Short-Utterance Coverage (SFT v2): The model is further fine-tuned on mined one-word and two-word segments. This addresses structural failures where the model struggled with isolated tokens, bare numbers, or short acknowledgments, which are underrepresented in standard sentence-level corpora.

Key Contributions

First Open Tokenizer-Free NAR Turkish TTS: FreyaTTS is a 183.2M-parameter model trained from scratch as a Turkish-first system. It is the first openly released tokenizer-free NAR TTS model that learns agglutinative morphology and spoken forms directly from audio without G2P rules.
Production-Hardening Recipe: The paper introduces a specific pretrain-to-SFT workflow (Voice Lock + Short-Utterance Coverage) that stabilizes speaker identity and resolves isolated-token failures, converting a generative prior into a deployable single-voice product.
Freya-TR-Eval Benchmark: The authors release a domain-neutral, 495-sentence Turkish benchmark with seeded build scripts and evaluation harnesses, enabling reproducible WER/CER and MOS evaluation without redistributing third-party audio.

Experimental Results

Evaluated on the Freya-TR-Eval benchmark using a band-matched Whisper-large-v3 protocol (downsampled to 8 kHz for fair comparison):

Accuracy: FreyaTTS achieves a Word Error Rate (WER) of 8.0% and a Character Error Rate (CER) of 3.0%.
Comparison: It outperforms significantly larger open-source systems, including XTTS-v2 (470M parameters, WER 11.1%) and F5-TTS (336M parameters, WER 24.3%). It also surpasses smaller VITS-based baselines like Piper (16M) and MMS-TTS (36M) in naturalness, despite those models having lower WER due to highly regular, phonemizer-driven pronunciation.
Naturalness: In a listening study with 24 native Turkish raters, FreyaTTS achieved a Mean Opinion Score (MOS) of 3.68, ranking second only to XTTS-v2 (3.82) and outperforming all other same-class systems.
Speaker Consistency: The voice-lock stage reduced the $F_0$ standard deviation across generations from 74.9 Hz to 5.0 Hz, effectively eliminating gender flips and ensuring a consistent speaker identity without reference prompts.

Efficiency and Deployment

Real-Time Factor (RTF): On consumer GPUs (RTX 4090), FreyaTTS achieves an RTF of 0.11. On an H100, the mean RTF is approximately 0.14.
Edge Deployment: The model runs faster than real-time on a laptop CPU (Apple M3), with an end-to-end RTF of approximately 0.12 when compiled for Core ML.
Throughput: With static batching, the model achieves 65.2 audio-seconds per wall-second, significantly outperforming the 2B-parameter VoxCPM2 even when the latter uses a dedicated continuous-batching serving engine.

Significance and Claims

The paper positions FreyaTTS as a demonstration that high-quality, reliable TTS for mid-resource languages does not require multi-billion-parameter multilingual foundations. By focusing on a Turkish-first approach, utilizing a frozen high-fidelity codec, and employing a non-autoregressive flow-matching objective, the authors achieve state-of-the-art performance for its parameter class.

The authors claim that FreyaTTS successfully bridges the gap between research-grade generative priors and deployable single-voice products. The model proves that a compact, tokenizer-free system can handle the linguistic complexities of Turkish (agglutination, vowel harmony) and the reliability requirements of conversational speech (digit accuracy, speaker consistency) while remaining efficient enough for edge deployment. The release of the model, code, and benchmark aims to provide a solid foundation for future Turkish and low-resource speech research.