💻 computer science

MultiChain Blockchain Data Provenance for Deterministic Stream Processing with Kafka Streams: A Weather Data Case Study

This paper proposes a MultiChain-backed provenance architecture for Kafka Streams that ensures deterministic, auditable, and reproducible real-time weather data processing by anchoring cryptographic Merkle roots of windowed data to the blockchain while keeping payloads off-chain.

Original authors: Niaz Mohammad Ramaki, Florian Schintke

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Niaz Mohammad Ramaki, Florian Schintke

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live news broadcast of the weather in Berlin. Every few minutes, the news anchor (the computer system) announces the average temperature for the last hour.

The Problem: The "Magic Trick" of Live TV
Usually, live data processing is a bit like a magician's trick. The computer takes in thousands of temperature readings, shuffles them around, and spits out an average. But because the computer is so fast and the internet is sometimes jittery, the order in which the data arrives can change.

If you asked the computer to do the exact same calculation twice, it might give you two slightly different answers.

Run 1: The computer sees three temperature readings and says, "The average is 21.0°C."
Run 2: Because of a tiny network delay, a fourth reading arrives a split second later. The computer puts it in a different "bucket" (time window) and says, "The average is 21.75°C."

This makes it impossible for an auditor to say, "I know for a fact that the computer calculated this correctly," because the result changes every time you replay the tape. It's like trying to verify a math problem where the numbers keep rearranging themselves.

The Solution: The "Sealed Envelope" System
The authors of this paper built a system to fix this using two main tools: Kafka Streams (the computer doing the math) and MultiChain (a special, unchangeable digital ledger, like a blockchain).

Here is how their system works, step-by-step, using a simple analogy:

1. The "Fixed Time Buckets" (Deterministic Windowing)

Instead of letting the computer decide when to stop counting and start a new average based on when it happens to receive data, the system uses fixed time buckets.

Analogy: Imagine a factory conveyor belt that stops every exactly 2 hours, no matter what.
How it works: Whether the data arrives at 1:59 PM or 2:01 PM, the system forces it into the "1:00 PM to 3:00 PM" bucket. This ensures that every time you run the calculation, the exact same set of data goes into the exact same bucket.

2. The "Standardized Recipe" (Canonical JSON)

Even if the data is in the same bucket, computers can sometimes write things differently (e.g., "10:00:00" vs "10:00").

Analogy: Imagine a chef who insists that every ingredient must be written down in the exact same handwriting and order before being put in a pot.
How it works: The system takes every temperature record and rewrites it into a strict, standard format. This guarantees that the data looks identical every time it is processed.

3. The "Digital Sealing Wax" (Merkle Trees)

Now that the data is in a fixed bucket and written in a standard format, the system needs to prove that the data hasn't been tampered with.

Analogy: Imagine you have a pile of receipts. Instead of locking the whole pile in a safe, you hash them (turn them into a unique digital fingerprint) and arrange them in a pyramid. You then take the very top number of the pyramid—the Merkle Root—and seal it in a wax envelope.
How it works: The system creates a "Merkle Root" for all the temperatures in that 2-hour bucket. This root is a single, tiny code that represents the entire group of data. If even one temperature changes, the code changes completely.

4. The "Unchangeable Logbook" (Blockchain Anchoring)

This is where the magic happens. The system does not put the heavy, messy temperature data on the blockchain (that would be too slow and expensive).

Analogy: You keep the heavy receipts in a drawer (off-chain storage), but you write the Seal Number (the Merkle Root) and the Page Number in a public, unchangeable diary (the blockchain).
How it works: The system writes the Merkle Root and the "page numbers" (Kafka offsets) to the MultiChain blockchain. The blockchain acts as a permanent, unalterable witness.

5. The "Auditor's Check" (Verification)

Now, imagine an auditor wants to check if the weather report was correct.

They look at the Diary (Blockchain) to see what the "Seal Number" was supposed to be for that time slot.
They go to the Drawer (Off-chain storage) and pull out the actual receipts (the temperature data).
They re-run the "Standardized Recipe" and the "Pyramid Seal" calculation.
The Result: If the new Seal Number matches the one in the Diary, the data is proven to be 100% authentic and complete. If it doesn't match, someone tampered with the receipts.

The Results
The authors tested this with real weather stations in Berlin. They found:

Reproducibility: They could run the calculation on two different computers, and it produced the exact same result every time.
Speed: Checking the math (verification) was very fast, even with lots of data.
Efficiency: The blockchain didn't get clogged because it only stored the tiny "Seal Numbers," not the heavy data itself.

In Summary
This paper presents a way to make live data streams "trustworthy." By locking the data into fixed time buckets, standardizing the format, and sealing the results in a digital envelope recorded on a blockchain, they allow anyone to independently verify that a live calculation (like a weather average) is mathematically correct and hasn't been faked or altered.

Technical Summary: Multichain Blockchain Data Provenance for Deterministic Stream Processing with Kafka Streams

1. Problem Statement

Real-time data stream processing pipelines, such as those built on Kafka Streams or Apache Flink, face critical challenges regarding auditability and reproducibility. These systems are inherently non-deterministic due to dependencies on runtime scheduling, window triggers, network jitters, and record arrival orders. Consequently, identical input streams processed at different times or on different nodes may yield different outputs.

Current blockchain-based provenance solutions often rely on hash chains to store data fingerprints. However, the authors identify two major drawbacks in this approach for stream processing:

Storage Overhead: As the chain grows, storage requirements increase significantly.
Computation Cost: Verification requires repeated hashing of long chains, which is inefficient for large datasets.
Furthermore, hash chains bind provenance to ingestion order, making them unsuitable for deterministic replay under re-keyed or time-based windows where record membership changes due to external factors (e.g., rebalancing).

2. Methodology

The paper proposes a blockchain-backed provenance architecture that decouples heavy data payloads from the blockchain, storing only cryptographic anchors and metadata. The system is implemented using Kafka Streams for processing and MultiChain (a permissioned blockchain) for anchoring.

Core Components and Workflow

Ingestion & Processing:
- Real-time weather data from two stations (Berlin-Brandenburg and Berlin-Tempelhof) is ingested via Kafka.
- The system uses epoch-aligned tumbling windows. Unlike processing-time windows, these are fixed absolute intervals (e.g., 2-hour blocks starting at specific epoch times) to ensure global determinism regardless of ingestion delays.
- Within each window, records are aggregated (min, max, average temperatures).
Deterministic Canonicalization:
- To eliminate non-determinism caused by record ordering, the system applies Canonical JSON to all records within a window.
- This process involves removing inconsistent fields, normalizing timestamps to UTC (ISO-8601), and lexicographically sorting object keys.
- Records are then deduplicated.
Merkle Tree Construction:
- Instead of a linear hash chain, the system constructs a Merkle Tree from the canonicalized records.
- Leaf nodes are SHA-256 hashes of the canonical records.
- Internal nodes are hashes of concatenated child nodes.
- The final Merkle Root serves as a compact, cryptographic commitment to the entire window's payload.
Blockchain Anchoring (Checkpoints):
- The full payload remains off-chain (stored in a distributed file system as NDJSON).
- A checkpoint containing the Merkle Root, Kafka offset boundaries (offsetStart, offsetEnd), record count, payload path, and payload hash is published to a MultiChain stream.
- MultiChain is chosen for its append-only, key-value stream features and permissioned nature, avoiding the complexity of smart contracts in favor of external application logic.
Verification:
- An independent auditor can verify a window's integrity by:
  1. Retrieving the checkpoint metadata (Merkle Root $R$ and record count $N$ ) from the blockchain.
  2. Loading the off-chain payload.
  3. Re-canonicalizing the records and recomputing the Merkle Root ( $\hat{R}$ ) and record count ( $\hat{N}$ ).
  4. Comparing $\hat{R}$ with $R$ and $\hat{N}$ with $N$ . Verification passes only if both match.

3. Key Contributions

Deterministic Window Provenance: The introduction of a method to create reproducible stream windows using epoch-aligned boundaries and canonical serialization, solving the non-determinism inherent in standard stream processing.
Merkle Tree over Hash Chains: A shift from linear hash chains to Merkle trees for window provenance, reducing verification complexity from linear to logarithmic relative to the number of records.
Off-Chain/On-Chain Split: A scalable architecture where large, mutable payloads are stored off-chain, while immutable cryptographic anchors (Merkle roots) and metadata are stored on-chain, preventing blockchain bloat.
Independent Auditability: A framework allowing third-party auditors to verify the completeness and correctness of derived analytics (e.g., temperature averages) without needing access to the internal state of the streaming engine.

4. Evaluation and Results

The system was evaluated using real-time weather data from Berlin.

Reproducibility: Independent runs on different hardware (MacOS M2 and M4) with Java 21 produced byte-identical results for canonical JSON, payload SHA-256, and Merkle roots, confirming the elimination of environment-induced non-determinism.
Verification Cost:
- End-to-end verification time ( $T(n)$ ) scales linearly with the number of records ( $n$ ), following the model $T(n) = \alpha n + \beta$ .
- The system demonstrated high throughput (records per second) during verification, with no collapse in performance as window sizes increased.
- Canonicalization and hashing costs showed linear scaling, and Merkle tree construction remained near-linear.
Blockchain Performance:
- Real-world checkpointing resulted in low Transactions Per Second (TPS) (0–0.015 items/s) due to the infrequent nature of window closures compared to raw event rates.
- Benchmarking with parallel publishers showed that MultiChain can handle higher TPS (up to ~2.1 tx/s for small payloads) but hits a saturation point (around 8 workers) due to its round-robin consensus algorithm.

5. Significance and Claims

The paper claims that its design enables independent auditors to verify three critical aspects of stream processing:

The completeness of window payloads (ensuring no records were dropped or added).
The canonical serialization of data (ensuring deterministic processing).
The correctness of derived analytics (e.g., verifying that a reported average temperature is mathematically consistent with the underlying records).

The authors position this work as a solution to the "auditability gap" in modern stream processing. By combining deterministic windowing with cryptographic anchoring, the system allows for reproducible reprocessing of data streams, a capability currently lacking in standard streaming platforms. The paper modestly notes that while it solves integrity and auditability, it does not yet address the semantic meaning of the data (requiring future work with RDF) or privacy concerns (suggesting zero-knowledge proofs as a future direction). It concludes that blockchain can be successfully integrated with streaming platforms to provide a trustworthy, immutable reference layer for data provenance.