💬 NLP

Calliope: A TTS-based Narrated E-book Creator Ensuring Exact Synchronization, Privacy, and Layout Fidelity

Calliope is an open-source framework that converts text e-books into EPUB 3 narrated e-books by leveraging state-of-the-art TTS to ensure exact audio-text synchronization, layout fidelity, and complete offline privacy.

Original authors: Hugo L. Hammer, Vajira Thambawita, Pål Halvorsen

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Hugo L. Hammer, Vajira Thambawita, Pål Halvorsen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a digital book on your tablet. Now, imagine if that book could also "read itself" to you—not just like a robotic voice-over, but like a professional audiobook where the words light up on the screen exactly as they are spoken. This is called a narrated e-book, and it’s a game-changer for kids learning to read, people with dyslexia, or anyone who wants to "read" while multitasking.

Currently, if you want this feature, you usually have to pay for expensive monthly subscriptions to big companies. These companies take your books, upload them to their "cloud" (their private computers), and process them there. This costs money, raises privacy concerns, and can even cause legal headaches regarding copyright.

Enter "Calliope": The DIY Digital Storyteller.

A group of researchers has created an open-source tool called Calliope. Think of Calliope not as a store where you buy books, but as a high-tech, private printing press that you own and keep in your own house.

Here is how it works, using a few simple analogies:

1. The "Surgical" Approach (Preserving the Beauty)

Most existing tools are like taking a beautiful, hand-painted tapestry, ripping it apart, and trying to weave it back together into a new blanket. You end up with a blanket, but the original art is gone.

Calliope is different. It uses "surgical modifications." Imagine a master restorer working on an ancient painting. Instead of repainting the whole thing, they carefully slide tiny, invisible markers behind the canvas to show where the story is happening. Because of this, your e-book keeps its original beautiful fonts, colors, and layouts. It looks exactly like the publisher intended.

2. The "Perfect Beat" (Exact Synchronization)

Have you ever watched a dubbed movie where the actor's lips move, but the sound comes a split second later? It’s distracting and ruins the magic. This is a common problem in digital books called "drift."

Other tools try to "guess" when a word is spoken by listening to the audio and matching it to the text (this is called forced alignment). It’s like trying to clap your hands perfectly in time with a drummer you can't quite see. You’ll eventually fall out of sync.

Calliope doesn't guess. It uses the "DNA" of the voice. Because Calliope is the one actually "speaking" the words using AI, it knows exactly when every single syllable starts and ends. It’s like a conductor who is also the musician—the timing is mathematically perfect. There is zero "lag."

3. The "Private Vault" (Privacy and Cost)

Using commercial services is like sending your private diary to a big corporation to have it read aloud; you have to trust them with your data.

Calliope works entirely offline. It’s like having a personal librarian living inside your computer. You don't need an internet connection, you don't have to pay a monthly fee, and your books never leave your device. Your data stays in your "vault."

Summary

In short, Calliope is a free, private, and incredibly precise tool that turns any standard e-book into a high-quality, synchronized "read-aloud" experience. It keeps the book beautiful, keeps the timing perfect, and keeps your reading habits private.

Technical Summary: Calliope – A TTS-based Narrated E-book Creator

Problem Statement

While "narrated e-books" (or immersive reading) significantly aid early literacy and individuals with reading challenges like dyslexia, the current landscape lacks a high-quality, open-source solution for creating them. Existing commercial services (e.g., ElevenReader) rely on cloud-based APIs, which introduce recurring costs, privacy risks, and potential copyright compliance issues when uploading protected content. Furthermore, existing open-source alternatives often suffer from two major flaws:

Synchronization Drift: Methods using "forced alignment" (mapping pre-existing audio to text) often result in a lag or lead between the spoken word and the visual highlight.
Layout Degradation: Many tools use a "reconstruction" strategy that strips the original CSS, destroying the publisher's intended typography, styling, and media layout.

Methodology

The authors propose Calliope, an open-source framework that converts standard EPUB files into EPUB 3 Media Overlay format using state-of-the-art (SOTA) neural Text-to-Speech (TTS) models (XTTS-v2 and Chatterbox). The pipeline is divided into three distinct phases:

Text Extraction and Layout Preservation: Instead of rebuilding the book, Calliope performs "surgical modifications." It traverses the Document Object Model (DOM) of the XHTML chapters, segments text into sentences, and injects unique <span> identifiers into the existing structure. This ensures the original CSS and layout remain intact.
TTS Synthesis and Deterministic Synchronization: To solve the synchronization problem, Calliope does not use forced alignment. Instead, it captures audio timestamps directly during the TTS generation process.
- Recursive Segmentation: To handle the context-window limits of Transformer-based models, the system uses a two-tier strategy: a pre-emptive heuristic (splitting long sentences) and a reactive error handler (splitting segments that cause token-overflow).
- Gapless Continuity: To prevent visual "flickering" during pauses, the system applies a linear fade-out to audio segments and extends the visual highlight duration to cover the silence between sentences.
Packaging: The framework reconstructs the EPUB container, injecting SMIL (Synchronized Multimedia Integration Language) files to map the text IDs to the precise audio timestamps and updating the OPF (Package Document) metadata.

Key Contributions

First Open-Source Solution: Provides a complete, offline pipeline for creating EPUB 3 Media Overlays.
Exact Synchronization: Moves away from forced alignment in favor of deterministic timestamping derived directly from the TTS engine.
High Fidelity: Preserves the publisher's original typography and styling through surgical DOM manipulation.
Privacy and Cost-Efficiency: Operates entirely offline on consumer-grade hardware, eliminating API costs and protecting user/copyrighted data.
Robust Text Handling: Introduces a recursive segmentation algorithm to mitigate TTS model constraints (token limits and phonetic density).

Results and Evaluation

The authors compared Calliope against two forced alignment methods: Storyteller (transcription-based) and Syncabook (DTW-based).

Synchronization Accuracy: Experiments on The Gift of the Magi showed that Storyteller had massive drifts (up to 30 seconds), and Syncabook—while more consistent—still had over 30% of sentences falling outside the acceptable synchronization window (defined as $[-50, 150]$ ms).
Calliope Performance: Calliope achieved zero drift, providing perfect alignment between the audio and the text highlight.

Significance

Calliope represents a significant advancement in digital accessibility. By providing a high-fidelity, privacy-preserving, and perfectly synchronized method for creating narrated e-books, it empowers developers and publishers to create immersive reading experiences without the technical or financial burdens of cloud-based or reconstruction-heavy workflows. The project is released under the MIT License, fostering further innovation in accessible digital publishing.