🤖 machine learning

Lakestream: A Consistent and Brokerless Data Plane for Large Foundation Model Training

Lakestream is a brokerless, object-store-native data plane for large foundation model training that introduces Transactional Global Batches and a Decentralized Adaptive Commit algorithm to provide ACID-like consistency, failure isolation, and high-throughput ingestion with lower latency than existing message queue or colocated solutions.

Original authors: Ting Sun, Junjie Zhang, Xiao Yan, Songxin Zhang, Zhuoyang Song, Jingyi Xi, Zunyao Mao, Bingyi Jing, Jiaxing Zhang, Zejian Xie

Published 2026-05-12

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ting Sun, Junjie Zhang, Xiao Yan, Songxin Zhang, Zhuoyang Song, Jingyi Xi, Zunyao Mao, Bingyi Jing, Jiaxing Zhang, Zejian Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant, super-smart robot (a Large Foundation Model) how to understand the world. To do this, you need to feed it massive amounts of data, like videos, images, and text, one "batch" at a time.

In the past, feeding this robot was like a simple conveyor belt in a factory. The data was already pre-packed into boxes, and the belt just moved them to the robot at a steady pace. But modern AI is different. The data is messy, huge, and changes size depending on what's inside it. Sometimes a single video file needs to be unpacked and stretched into something 1,000 times bigger before the robot can use it.

The old conveyor belts (existing systems) couldn't handle this. They either got clogged up, or if one worker dropped a box, the whole factory stopped.

Enter Lakestream: The "Smart, Brokerless" Data Plane.

The authors of this paper built a new system called Lakestream. Think of it as a revolutionary way to organize the data delivery for these giant AI models. Here is how it works, using simple analogies:

1. The Problem with the Old Ways

The "Colocated" Problem (The Kitchen in the Gym): Imagine the robot (the trainer) is lifting weights, and the person preparing its food (the data loader) is standing right next to it in the same small room. If the food prep takes too long, the robot has to stop lifting. If the food prep person trips and falls, the robot stops forever. They are too tangled together.
The "Message Queue" Problem (The Confused Courier): Imagine using a central post office (like Kafka) to deliver data. The post office treats every piece of paper as a separate letter. But the robot needs a complete meal (a batch) all at once. If the post office delivers the "chicken" part of the meal to one robot arm and the "rice" part to another arm at different times, the robot gets confused and learns the wrong things. Also, the post office is a single point of failure; if the postmaster gets sick, no one gets fed.

2. The Lakestream Solution: A Shared Digital Warehouse

Lakestream gets rid of the central post office and the cramped kitchen. Instead, it uses a massive, shared Object Storage (like a giant, infinite digital warehouse, e.g., Amazon S3) and a Versioned Manifest (a master ledger).

Here are the three magic tricks Lakestream uses:

A. The "Transactional Global Batch" (The Perfect Meal)

In the old days, data was just a stream of individual records. Lakestream treats a whole "batch" of data as a single, unbreakable unit called a Transactional Global Batch (TGB).

The Analogy: Imagine a chef preparing a full banquet. The food is cooked and placed on the table, but it's covered by a lid. The lid is locked. No one can see the food yet.
The Magic: When the chef is sure the whole banquet is ready, they flip a switch on the master ledger (the Manifest). Suddenly, the lid lifts, and every robot arm sees the complete, perfect meal at the exact same moment. If the chef drops a plate before flipping the switch, the lid stays down, and no one sees the mess. This ensures everyone is always working with the same data.

B. The "Decentralized Adaptive Commit" (The Smart Traffic Light)

When many chefs (producers) are trying to update the master ledger at the same time, they might try to write on the same page, causing a conflict.

The Old Way: They would just keep banging on the door until someone let them in, wasting time.
The Lakestream Way (DAC): The chefs have a smart, self-taught rhythm. They watch how busy the ledger is. If the ledger is getting huge and updates are slow, they automatically wait a tiny bit longer before trying to write again. If it's quiet, they write faster. They do this without talking to each other, just by looking at the ledger. This keeps the traffic flowing smoothly even when hundreds of chefs are working at once.

C. The "Checkpoint-Aligned Lifecycle" (The Time-Travel Safe)

AI models often need to "save their progress" (checkpoint) and sometimes go back to a previous save to try a different path.

The Problem: In old systems, once data is eaten, it's gone. If you need to go back, you can't.
The Lakestream Way: The system keeps a history of every meal served, tied to the specific "save point" of the robot. When the robot saves its progress, it also writes a "watermark" (a safety line) in the ledger. The system knows to keep all the data before that line safe and sound. It only deletes the old data once it knows no robot save point needs it anymore. This means you can rewind time perfectly without losing your data, and you don't have to pay to store data nobody needs.

3. The Results

The paper tested this system on 64 powerful GPUs with huge video and image datasets.

Speed: It was 2.7 to 7.7 times faster than the best existing "colocated" systems (where data prep and training happen together).
Reliability: It handled failures much better. If a data prep worker crashed, the robot kept training without stopping.
Efficiency: It was much faster than using a central message queue (like Kafka), which struggled to keep up with the size and complexity of the data.

Summary

Lakestream is like replacing a chaotic, single-lane road with a smart, decentralized highway system. It uses a shared warehouse and a master ledger to ensure that:

Everyone gets the exact same "batch" of data at the same time.
The system keeps moving fast even when the traffic gets heavy.
You can rewind the training process to any point in history without losing your data.

It achieves all this without needing a central manager (broker) to tell everyone what to do, making it faster, cheaper, and more reliable for training the world's biggest AI models.

Technical Summary: Lakestream

Problem Statement

Modern Large Foundation Model (LFM) training has fundamentally altered the requirements for data pipelines. Unlike traditional deep learning workloads that rely on static, pre-processed datasets, LFM training involves dynamic, runtime-coupled data generation. This shift introduces three critical challenges that existing systems fail to address:

Unpredictable Data Volume: In multimodal training (e.g., video, interleaved image-text) and reinforcement learning, raw inputs undergo heavy runtime preprocessing (decoding, rescaling, token packing). This results in data expansion factors ranging from 62× to over 9,000×, making static resource allocation and pre-partitioning infeasible.
Dynamic Batch Membership: In workloads using online token packing or model-dependent sample selection, the exact composition of a training batch is determined only after preprocessing completes. Static partitioning schemes cannot capture these boundaries, requiring a mechanism to expose complete batches atomically to all training ranks.
Checkpoint Alignment: LFM pipelines often interleave pre-training, fine-tuning, and reinforcement learning, necessitating the ability to roll back to specific checkpoints and replay exact batch sequences. Ephemeral delivery systems cannot guarantee the persistence and versioning required for deterministic recovery.

Existing solutions fall short:

Colocated Dataloaders: Offer semantic alignment but lack failure isolation. A preprocessing crash stalls the entire job, and shared CPU/memory resources create throughput ceilings.
Disaggregated Dataloaders (Centralized): Decouple preprocessing but rely on ephemeral RPC delivery. They lack persistent history, batch-level atomicity, and mechanisms for replaying specific batches after failures.
Message Queues (e.g., Kafka): Operate on a record/offset abstraction rather than batch semantics. They either force out-of-order shard delivery (corrupting gradient aggregation) or require downloading full batches for every rank (causing massive read amplification).

Methodology

Lakestream is a brokerless, object-store-native training data plane designed to solve these issues by adapting lakehouse architecture principles to the training loop. Its core design philosophy is that the object store should be the sole coordination medium, eliminating brokers, queues, or central services.

Core Abstraction: Transactional Global Batch (TGB)

The system introduces the Transactional Global Batch (TGB), a first-class persistent entity that treats each training batch as an atomic unit with durability and globally consistent ordering.

Materialization: TGBs are written as immutable objects to the object store.
Manifest-Gated Visibility: Writing data objects does not make them visible. A TGB becomes visible to consumers only when a committed, versioned manifest references it. This ensures that all data-parallel ranks observe a complete, consistent batch simultaneously.
Layout: TGBs are organized into data slices corresponding to the device mesh (Data Parallelism and Context Parallelism). Each rank reads only its specific slice via targeted range reads, eliminating read amplification.

Control Plane: Decentralized Coordination

Lakestream achieves coordination without a central server through two key mechanisms:

Optimistic Concurrency Control (OCC) via Manifests: Producers commit TGBs by attempting a conditional write (e.g., If-None-Match) of a new manifest version. If the version name is already claimed, the write fails, and the producer rebases its local state onto the latest committed manifest before retrying. This linearizes the global batch sequence without inter-producer communication.
Decentralized Adaptive Commit (DAC): To sustain high throughput as the manifest grows, Lakestream employs the DAC algorithm. Instead of fixed intervals, producers dynamically adjust their commit cadence based on real-time estimates of manifest I/O time and conflict rates. DAC maintains an explicit "overhead budget" and "conflict budget," calculating an optimal waiting gap between commit attempts to maximize freshness while minimizing wasted retries.

Lifecycle Management

Exactly-Once Semantics: Producer state (stream offsets) is inlined directly into the manifest's per-producer state map. Upon restart, a producer can recover its exact position without re-processing data.
Checkpoint-Aligned Reclamation: Data retention is tied directly to distributed checkpoint progress. A global watermark ( $W_{global}$ ) is derived from the minimum watermark of all consumers. Any TGBs or manifest versions below this watermark are eligible for deletion, ensuring storage costs remain bounded while guaranteeing that data required for any live checkpoint is never prematurely reclaimed.

Key Contributions

Brokerless, Object-Store-Native Architecture: The system establishes the object store as the sole coordination medium. No server-side processes exist in the critical path; a new training run requires only a fresh namespace prefix.
Transactional Global Batch (TGB): A novel abstraction that unifies data visibility with training step semantics, providing atomic all-rank visibility, global ordering, and exactly-once delivery entirely through object store primitives.
Decentralized Adaptive Commit (DAC): An algorithm that sustains stable ingestion throughput as the manifest grows, adapting commit intervals dynamically without inter-producer communication.
Checkpoint-Aligned Lifecycle: A mechanism that ties data retention to distributed checkpoint progress, enabling safe rollbacks and bounded storage overhead without a persistent coordination service.

Evaluation Results

The authors evaluated Lakestream on large-scale multimodal pre-training and Supervised Fine-Tuning (SFT) workloads using up to 64 GPUs (NVIDIA H200).

Throughput vs. Colocated Pipelines: Lakestream outperformed expert-optimized colocated pipelines by 2.68× to 7.73× in end-to-end throughput. This gain is attributed to the elimination of structural resource contention (CPU/memory bandwidth sharing) between preprocessing and training.
Throughput vs. Kafka: Lakestream significantly outperformed Apache Kafka in producer ingestion throughput, particularly as the number of producers scaled. Kafka failed to sustain strict-TGB semantics for large payloads (e.g., video) due to message size limits and broker backpressure.
Latency: Lakestream achieved lower consumer read latency than Kafka. For example, in video SFT workloads, Lakestream reduced P95 latency from ~42.7s (Local) and ~9.5s (Kafka) to 367ms.
Scalability: The DAC algorithm successfully maintained high commit success rates (96.3%) and throughput (431.9 MB/s) over 5-hour runs with 32 producers, whereas fixed-interval and heuristic baselines (AIMD, INCR) degraded significantly as the manifest grew.
Read Amplification: Lakestream achieved near 1× read amplification regardless of world size, as each rank reads only its specific data slice. In contrast, Kafka and dense-read baselines exhibited amplification scaling linearly with the number of ranks (e.g., 128×).
Storage Efficiency: Checkpoint-driven reclamation reduced peak storage capacity by 72.0% (from 34.85 GiB to 9.76 GiB) compared to runs without physical deletion, confirming that watermarks effectively bound storage costs.

Significance

The paper claims that Lakestream demonstrates that object storage, when combined with versioned metadata and decentralized coordination, is a sufficient substrate for a training-aware data plane. By treating the Global Batch as a persistent, transactional entity rather than an ephemeral delivery artifact, Lakestream addresses the specific semantic requirements of LFM training—dynamic batch membership, checkpoint alignment, and failure isolation—that existing record-based streaming systems cannot satisfy. The system proves that a brokerless architecture can sustain high-throughput, low-latency data delivery for large-scale foundation model training without the overhead and failure domains of centralized services.