⚡ electrical engineering

MerkleSpeech: Public-Key Verifiable, Chunk-Localised Speech Provenance via Perceptual Fingerprints and Merkle Commitments

MerkleSpeech is a dual-layered speech provenance system that combines robust neural watermarking with Merkle-tree-based cryptographic commitments to provide public-key verifiable, chunk-level authentication that remains resilient to common audio transformations and splicing.

Original authors: Tatsunori Ono

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tatsunori Ono

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video of a famous politician giving a speech. Suddenly, you notice a sentence that sounds suspicious. You wonder: “Did they actually say that, or was this clip edited to change their meaning?”

Currently, detecting these "deepfakes" or edits is like trying to spot a counterfeit bill by looking at it with the naked eye—it’s hard, and even experts can be fooled.

The paper "MerkleSpeech" proposes a high-tech "digital DNA" system for speech that makes it nearly impossible to lie about what was said without getting caught.

Here is how it works, explained through three simple analogies.

1. The "Digital Wax Seal" (The Two Layers of Proof)

Think of a traditional letter. Usually, you just trust the signature at the bottom. But MerkleSpeech uses two different levels of security, much like a high-security package:

Layer 1: The Brand Label (WM-only). Imagine a box of cereal. Even if the box gets a little dented or the colors fade in the sun (like audio getting compressed or noisy), you can still clearly see the "Kellogg’s" logo. This tells you, "This definitely came from the Kellogg's factory." In the paper, this is the watermark. It tells you who made the audio, even if the audio quality isn't perfect.
Layer 2: The Tamper-Evident Seal (MSv1). Now imagine that same cereal box has a special, fragile wax seal over the flap. If someone tries to open the box to swap the cereal for something else, the seal must break. This is the cryptographic layer. It doesn't just tell you who made it; it proves that not a single grain of cereal has been touched since it left the factory.

2. The "Library of Fingerprints" (The Merkle Tree)

How do you prove that a specific 2-second clip is part of a 1-hour speech without checking the whole hour every time?

Imagine a massive library where every single page of a book has a unique fingerprint. Instead of carrying the whole book around, the author creates a "Master Fingerprint" (called a Merkle Root) that represents every single page combined.

If you want to prove that "Page 42" is authentic, you don't need the whole book. You just need a tiny "receipt" (an inclusion proof) that mathematically links Page 42 to that one Master Fingerprint. If someone tries to swap Page 42 with a fake one, the math won't add up to the Master Fingerprint, and the alarm goes off.

3. The "Splice-Aware Timeline" (Detecting the Edit)

Most current systems are "all or nothing"—they tell you if a whole file is fake. But real liars don't fake a whole speech; they just "splice" in one bad sentence.

MerkleSpeech works like a medical heart monitor. Instead of just saying "the patient is dead," it shows a continuous line of pulses.

If the line is green, the speech is original.
If the line turns yellow, it means: "We know who said this, but the audio has been slightly distorted (maybe by bad Wi-Fi or noise)."
If the line turns red, it means: "Warning! This specific 2-second chunk does not match the original fingerprint. Someone has tampered with it!"

This allows a viewer to see exactly where a "truthful" speech was interrupted by a "fake" edit.

Summary: Why does this matter?

In a world where AI can mimic anyone's voice, we can't just rely on our ears. We need a system where:

The Creator signs their work with a digital key.
The Audio carries a tiny, hidden "ID card" (the watermark).
The Math (the Merkle Tree) proves that every tiny chunk of sound is exactly what the creator intended.

MerkleSpeech provides a way to move from "I think this is real" to "I can mathematically prove this is real."

Technical Summary: MerkleSpeech

MerkleSpeech: Public-Key Verifiable, Chunk-Localised Speech Provenance via Perceptual Fingerprints and Merkle Commitments

1. The Problem: The Gap in Speech Provenance

Current speech provenance technologies face a fundamental trade-off between robustness and cryptographic integrity.

Watermarking limitations: While neural watermarking can survive distribution transforms (like resampling or noise), it typically lacks a third-party verifiable cryptographic proof that ties a specific audio segment to a signed original.
Standardization limitations (e.g., C2PA): Existing standards often use file-level or container-level hashes. These are "brittle"; benign processing like transcoding, re-encoding, or resampling breaks the hash, rendering the provenance invalid.
The "Splice" Problem: Real-world audio is often edited (cropped, spliced, or concatenated). Most systems struggle to provide a "splice-aware" timeline that identifies exactly which parts of an audio file are authentic and which have been altered.

2. Methodology: A Two-Tiered Architecture

MerkleSpeech proposes a system that separates attribution (who sent this?) from integrity (has this been changed?). It achieves this by combining perceptual fingerprinting with Merkle tree commitments.

The Core Components:

Chunking: Audio is divided into discrete time segments (chunks), e.g., 2.0 seconds.
Perceptual Fingerprinting: For each chunk, a deterministic fingerprint ( $b_i$ ) is computed. The paper evaluates a strict MFCC-based spectral fingerprint.
Merkle Commitment: Fingerprints are hashed into a Merkle tree. The root of this tree ( $R$ ) is signed by the issuer’s private key ( $\sigma$ ). This creates a cryptographic anchor for all chunks in the asset.
In-Band Watermark Payload: Instead of embedding large proofs, a compact payload is embedded in the audio via a watermark (using QIM-STFT). This payload contains a Content Identifier (CID) and a chunk index, acting as a "pointer" to a repository.
The Repository: A central service that stores the signed manifest (containing the root and signature) and the Merkle inclusion proofs ( $\pi_i$ ).

The Two Tiers of Verification:

WM-only (Attribution Tier): The verifier extracts the watermark, finds the manifest via the CID, and checks the issuer's signature. This answers: "Was this chunk issued by a known party?" It is designed to be robust against noise and resampling.
MSv1 (Integrity Tier): The verifier recomputes the fingerprint of the suspect chunk and checks its Merkle inclusion against the signed root. This answers: "Does this chunk match the original exactly?" This tier is highly sensitive to any spectral modification.

3. Key Contributions

Public-Key Verifiability: Unlike many watermarking schemes that require secret keys for verification, MerkleSpeech allows any third party to verify provenance using only public keys and the repository.
Splice-Awareness: By providing verification at the chunk level, the system generates a timeline that can pinpoint exactly where an edit occurred.
Pluggable Design: The system allows for different fingerprinting methods (e.g., strict MFCC vs. transform-tolerant SSL embeddings) to balance the trade-off between robustness and integrity.
Streaming-Capable Protocol: The design supports real-time verification by allowing chunks to be processed and verified as they arrive.

4. Experimental Results

The system was evaluated using the LibriSpeech dataset under various stress conditions:

Accuracy: On clean audio, the system achieved a 99.9% decode rate and message match, with zero false positives observed in 1.5 million negative samples.
Robustness (WM-only): The attribution layer proved highly resilient, maintaining 100% verification under clipping, resampling (8/12 kHz), and moderate noise (30 dB).
Sensitivity (MSv1): As intended, the integrity layer was extremely sensitive. It failed under almost all transforms (noise, resampling, etc.) because those transforms altered the spectral fingerprint, correctly flagging the audio as "modified."
Splice Detection: The system achieved perfect IoU and F1 scores for boundary-aligned splices, demonstrating its ability to accurately map the provenance of edited audio.

5. Significance

MerkleSpeech provides a blueprint for a scalable, cryptographically defensible provenance framework. Its primary significance lies in its diagnostic capability: by offering two tiers of verification, it doesn't just say "this is fake"; it can distinguish between "this is an authentic clip that has been compressed/resampled" (passes WM-only, fails MSv1) and "this is a completely different, unauthenticated clip" (fails both). This distinction is critical for forensic applications, media authentication, and combating sophisticated voice-based misinformation.