⚡ electrical engineering

BigDipper: Sharded Censorship Resistant Data Availability for Leader-Based BFT

BigDipper is a sharded data availability framework for leader-based BFT protocols that decouples censorship resistance from the consensus layer by introducing a DA-CR mechanism, allowing validators to verify mini-block availability without full reconstruction while enabling higher-layer applications to customize their own protection strategies.

Original authors: Bowen Xue, Samuel Laferriere, Soubhik Deb, Sreeram Kannan

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bowen Xue, Samuel Laferriere, Soubhik Deb, Sreeram Kannan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Boss" Who Can Hide Your Letter

Imagine a blockchain network is like a massive, high-speed postal service. In many of these systems, there is a Leader (like a Postmaster) who is responsible for sorting the mail and putting it onto the delivery trucks (blocks) that go out to the world.

This system is fast and efficient. However, it has a flaw: the Postmaster has total control over what gets on the truck right now. If the Postmaster is malicious, they can keep the trucks running on schedule but secretly leave your letter off the truck. They don't have to stop the whole system; they just have to delay your specific letter for a few minutes.

In the real world, this is a disaster for things like:

Auction bids: If your bid is delayed by one truck, you lose the auction.
Liquidations: If a bank can't liquidate a bad loan quickly, they lose money.
Bridge messages: If you are moving crypto between chains, a delay could mean lost funds.

Existing solutions try to fix this by forcing the Postmaster to follow strict rules for everyone. But this is like making everyone pay for a VIP armored truck just because one person is sending a time-sensitive letter. It's expensive and slows everything down.

The Solution: BigDipper

BigDipper is a new system designed to fix this without slowing down the whole network. It follows a simple rule: "Don't hardcode the rules; give the tools and let the user decide."

Think of BigDipper as a Smart Sorting Room that sits between the people sending letters and the Postmaster.

1. The "Mini-Blocks" (The Shards)

Instead of the Postmaster collecting all the mail themselves, BigDipper lets the Replicas (the local post offices) collect the mail first.

Each local post office gathers the letters they receive and puts them into a small, sealed envelope called a Mini-Block.
These envelopes are then sent to the Postmaster.

2. The "Magic Puzzle" (Erasure Coding)

Here is the clever part. The Postmaster takes all these Mini-Blocks and shreds them into puzzle pieces using a special mathematical trick called Erasure Coding.

Imagine taking 100 letters, shredding them, and creating 300 puzzle pieces.
You can reconstruct the original 100 letters from any 100 of those 300 pieces.
The Postmaster sends these puzzle pieces to the Validators (the auditors who check if the system is honest).

Why is this cool?
The auditors don't need to see the whole truckload of mail to know it's there. They just need to check a few puzzle pieces and a "magic receipt" (a cryptographic commitment) to prove the whole truck is valid. This keeps the system fast and scalable.

3. The "Censorship-Resistant" Rule (DA-CR)

This is the core innovation. BigDipper introduces a rule called DA-CR (Data Availability with Censorship Resistance).

The Rule: The Postmaster must include puzzle pieces from at least a certain number of local post offices. They cannot just pick and choose who gets on the truck.
The Catch: The Postmaster can still try to hide a specific letter, but they can only do it if they manage to exclude the entire local post office that received it.

4. The "Fanout" Strategy (Navigator)

This is where the user (the client) comes in. BigDipper gives you a choice: How much protection do you want?

The Low-Cost Option: You send your letter to just one local post office. It's cheap and fast, but if that specific post office is ignored by the Postmaster, your letter gets censored.
The High-Security Option: You send your letter to many local post offices (high "fanout"). Even if the Postmaster ignores 50% of the offices, your letter is still in the hands of the others. It costs a bit more to send it to more places, but your chance of getting censored drops to near zero.

The Analogy:
If you are sending a regular birthday card, you might just drop it in one mailbox. If you are sending a winning lottery ticket, you might send copies to 10 different mailboxes. BigDipper lets you choose how many mailboxes to use based on how much the letter is worth to you.

The Result: A Modular System

BigDipper separates the Consensus (agreeing on the order of events) from the Execution (actually processing the data).

The Validators (Auditors) only check the "Magic Receipts" and a few puzzle pieces. They don't need to download the whole truck. This makes the system incredibly fast.
The Execution Layer (The people who actually run the apps) can decide how to retrieve the full data later. They can use powerful computers, specialized teams, or other methods to get the full picture.

Summary of Claims

The paper claims that by using this "Sharded" (split-up) approach with Erasure Coding and Linear Commitments:

Censorship is Tunable: Users can choose their own level of protection. If you need to be uncensorable, you pay the cost of sending to more nodes. If you don't, you save money.
Scalability: The system doesn't slow down as it grows because validators don't have to download the entire block; they only check their specific "shards" (puzzle pieces).
Safety: Even with a malicious Postmaster, the system guarantees that if enough honest nodes have your data, the system cannot be tricked into thinking the data is missing or changed.
Integration: They successfully built this into HotStuff-2 (a popular blockchain protocol) without breaking its safety rules, proving it works in a real-world setting.

In short, BigDipper stops the "Postmaster" from being a single point of failure by spreading the mail across many hands, using math to prove the mail is there, and letting the sender decide how many hands they want to trust.

Technical Summary: BigDipper

Problem Statement

Leader-based Byzantine Fault Tolerant (BFT) protocols are dominant in production blockchain systems due to their low latency and simple communication structures. However, they grant the leader short-term control over transaction inclusion. A malicious leader can keep the protocol live (ensuring liveness) while selectively delaying or excluding time-sensitive transactions, such as auction bids, oracle updates, liquidations, and bridge messages. This phenomenon, termed short-term censorship, is not captured by standard BFT liveness guarantees.

Existing solutions often hardcode strong anti-censorship, hiding, or ordering mechanisms into the consensus path. While effective for specific use cases, these approaches force all transactions to pay the same high cost for protection, regardless of whether a specific transaction requires such robustness. This paper argues for a more modular design that adheres to the end-to-end principle: the consensus layer should expose minimal primitives for safe ordering and data availability, leaving the choice of censorship resistance strategies and costs to higher-layer applications.

Methodology

BigDipper introduces a Censorship-Resistant Data Availability (DA-CR) layer positioned between transaction submission and the leader-based BFT consensus. The system is designed around three core components:

DA-CR Abstraction: A layer where replicas contribute mini-blocks, and a mechanism certifies which data is available and must be captured by the leader. This layer is characterized by parameters $(t, \eta)$ , where $t$ represents the guaranteed inclusion of honest mini-blocks, and $\eta$ represents the leader's residual ability to capture space (exclude data).
Sharded Consensus Critical Path: Unlike traditional designs where validators must download and execute the full block payload, BigDipper keeps data sharded on the critical path. Validators vote based on compact commitments, availability evidence, and DA-CR inclusion rules rather than reconstructing the full payload.
Client-Tunable Submission (Navigator): A transaction dissemination protocol where clients choose a "fanout" parameter $x$ , sending transactions to $x$ distinct replicas. This allows clients to trade off communication cost against the probability of prompt inclusion.

Core Constructions

The paper presents concrete DA-CR constructions based on Reed-Solomon (RS) erasure coding and linear polynomial commitments (KZG):

Card-7: An accountable protocol where the leader must provide attestations for included mini-blocks. It guarantees $t = n - 2f$ honest mini-blocks are included and $\eta = 0$ (no space capturing). It requires $n \ge 3f + 1$ .
Card-1/4: A non-accountable variant with lower overhead. It removes the requirement for public attestations, allowing the leader to hide which replicas were included. It achieves $t = n - 3f$ and $\eta = n - f - 1$ , requiring a stronger assumption of $n \ge 4f + 1$ .
Card-Lite: An optimization of Card-7 using a doubly homomorphic commitment scheme. It reduces the per-replica overhead to $O(\log n)$ by sampling commitments rather than transmitting the full list, while maintaining $t \approx n - 2f$ .

Integration with HotStuff-2

BigDipper is instantiated within HotStuff-2. The integration is surgical, introducing a Scooping rule as a validity predicate. A leader proposal is valid only if it incorporates a completed DA-CR output (a certificate proving data availability and inclusion of required mini-blocks). This does not alter the safety core of HotStuff-2 but prevents leaders from making progress without capturing the required replica-contributed data.

Key Contributions

Modular Abstraction for Short-Term Censorship Resistance: The introduction of DA-CR, which decouples the mechanism for data availability from the specific censorship resistance policies. It formalizes the trade-offs between guaranteed inclusion ( $t$ ) and leader influence ( $\eta$ ).
Sharded, Scalable Data Path: A design where ordinary validators do not need to download the entire block payload to vote. They verify commitments and metadata, allowing aggregate proposal bandwidth to scale with replica-side data contribution rather than being limited by a single leader's input bandwidth.
Client-Tunable Censorship Resistance: The Navigator protocol allows clients to amplify inclusion probability by increasing their fanout. The paper analyzes the cost/security tradeoff, proving via a union-bound argument that broad censorship becomes exponentially unlikely as the number of submitted transactions and replicas increases.
Concrete Integration and Analysis: A full instantiation of BigDipper inside HotStuff-2, demonstrating that DA-CR can be precisely connected to existing leader-based BFT protocols without compromising safety or liveness.

Results and Analysis

Censorship Resistance: The analysis shows that while a malicious leader can censor transactions sent to a small number of replicas, censoring a large fraction of transactions becomes exponentially difficult as the system scales. Specifically, if $T$ clients each send $x$ copies, the probability of an adversary censoring an $\alpha$ fraction of transactions decreases exponentially with $T$ .
Scalability: By separating consensus from full-data execution, BigDipper allows the consensus layer to scale linearly with the aggregate bandwidth of replicas. The system capacity is not bottlenecked by the need for every validator to process the full payload.
Performance: The Card-7 protocol fits within the existing message schedule of HotStuff-2 (Propose, Vote, Prepare, Vote2) without adding new phases, piggybacking DA-CR messages (Collection, Dispersal, Approval, Agreement) onto standard BFT messages.

Significance

The paper claims that BigDipper provides a modular framework for short-term censorship resistance that avoids the "one-size-fits-all" tax of existing protocols. By following the end-to-end argument, it allows applications to choose their own submission strategies (e.g., how many replicas to contact, whether to use erasure coding) based on their specific economic needs for latency and inclusion guarantees.

The significance lies in shifting the burden of censorship resistance from a fixed protocol cost to a tunable client-side parameter, while ensuring that the consensus layer remains efficient and sharded. It demonstrates that leader-based BFT can be made censorship-resistant without sacrificing the low-latency benefits of the leader-based model, provided that data availability is certified through a sharded, commitment-based mechanism rather than full replication.