🤖 AI

Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

This paper introduces Workload-Aware DiLoCo (WA-DiLoCo), a calibrated scheduling framework that demonstrates how incorporating burst forecasting and rigorous matched-random baselines can significantly reduce SLO violations in shared AI infrastructure compared to existing forecast-free policies.

Original authors: Maxwell Twelftree, David Lemphers, An-chi He, Yue Yang

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maxwell Twelftree, David Lemphers, An-chi He, Yue Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a busy restaurant kitchen (the AI Fleet) where two very different activities are happening at the same time:

The Prep Team (Training): A group of chefs is working on a massive, complex recipe. They need to taste their dish, adjust the spices, and then shout their changes to the head chef so everyone can update their recipe cards. This shouting happens in "sync" moments.
The Waiters (Serving): At the same time, waiters are rushing to the kitchen to pick up finished meals for customers. These customers are very impatient; if their food takes too long, they get angry (this is an SLO violation).

The Problem: The "Shouting" Interrupts the "Ordering"

In the old way of doing things, the chefs would shout their updates to the head chef on a strict timer (e.g., every 5 minutes), no matter what was happening in the kitchen.

The paper introduces a new method called DiLoCo. Instead of shouting every 5 minutes, the chefs work quietly on their own for a while, then shout all at once. This saves time.

But here is the catch: When the chefs finally shout (the "Outer Merge"), it takes about 8 seconds. During those 8 seconds, the kitchen is chaotic. The waiters can't get their food, the phones ring off the hook, and customers get angry.

The big question the paper asks is: When is it safe to shout?

If you shout while a rush of orders is coming in, you ruin the service.
If you shout when the kitchen is quiet, nobody notices.

The Mistake in Previous Research

Previous studies tried to figure out the best time to shout. They compared their "smart" schedules against a "dumb" schedule that shouted at fixed times. They claimed, "Our smart schedule is 20% better!"

The authors say: "Wait a minute. That's not a fair test."

Imagine you have a budget of three shouts for the whole day.

The Dumb Schedule: Shouts at 9:00, 1:00, and 5:00. (It might hit a rush hour).
The "Smart" Schedule: Tries to avoid rush hours.
The "Matched Random" (The Paper's New Control): This is the paper's secret weapon. It takes the exact same budget of three shouts but places them at random times.

The paper argues that if your "smart" schedule can't beat a random schedule that has the same number of shouts, then your "smartness" isn't actually doing anything. You might just be lucky, or the random schedule might have accidentally avoided the rush hours too.

The Solution: "Calibrated" Scheduling

The authors built a system called WA-DiLoCo (Workload-Aware DiLoCo). Think of it as a Kitchen Manager who looks at two things before deciding when to shout:

How much progress the chefs have made (Do they have enough new spices to share?).
How busy the waiters are (Is the kitchen currently in a rush?).

The Manager uses a score. If the kitchen is busy, the score goes down, and the Manager says, "Wait, don't shout yet." If the kitchen is quiet, the score goes up, and the Manager says, "Go ahead, shout now!"

The "Calibration" Protocol (The Reality Check)

The paper introduces a strict set of rules (a protocol) to prove that this Manager actually works. They don't just say "it works." They prove it in three steps:

The Stress Test: They simulate a kitchen with fake, predictable chaos. The Manager does well here.
The Real Kitchen Replay: They take the Manager's schedule and replay it against real customer data from a real AI system (vLLM).
- Result: In a steady, busy kitchen, the Manager does better than the fixed timer, but random timing does just as well. The Manager didn't prove it was special yet.
- Result: In a bursty kitchen (where orders come in sudden, unpredictable waves), the Manager shines. By looking at the pattern of the waves, the Manager can hide the "shout" in the quiet gaps between waves.
The Forecast: The Manager gets a crystal ball (an EWMA forecast) that predicts the next wave of orders. With this crystal ball, the Manager can dodge the chaos even better.

The Results (In Plain English)

Without the crystal ball: The Manager is good, but sometimes a random schedule gets lucky and does the same job.
With the crystal ball: The Manager beats the random schedule significantly.
- In their tests, the "angry customer" rate (SLO violations) dropped from 6.54% to 5.09%.
- This means fewer customers got angry because the kitchen didn't get interrupted during the busiest moments.

The Big Lesson

The paper's main takeaway isn't just "we built a better scheduler." It's about how we prove it works.

Before you claim your new AI system is faster or better for customers, you must:

Compare it to a random schedule with the same resources (not just a fixed timer).
Test it with real customer data, not just fake simulations.
Show that your system actually avoids the "busy windows" better than luck would.

If you can't beat the random schedule in a real-world test, you haven't actually solved the problem; you've just been lucky. The paper proves that with the right "calibration" and a little bit of prediction, you can make the kitchen run smoother without slowing down the chefs.

Technical Summary: Not Every Sync Is Safe: Calibrated DiLoCo Scheduling for Shared AI Infrastructure

Problem Statement

In fragmented industrial AI fleets where training shares hardware with latency-sensitive serving (e.g., vLLM sidecars), conventional DiLoCo-style training faces a critical production challenge: when is it safe to synchronize?

While DiLoCo reduces communication by allowing learner islands to perform local optimization steps before occasional outer synchronization, existing scheduling studies often fail to isolate the value of timing from the value of budget. Most evaluations compare workload-aware policies against fixed-period baselines (e.g., every 256 or 512 steps). However, a fixed-period baseline may simply synchronize less frequently or at different times by chance, rather than demonstrating true workload awareness.

The core problem is that an outer merge consumes host, network, and checkpoint resources for a fixed duration (e.g., 8 seconds). If this window overlaps with latency-sensitive requests, it causes Service Level Objective (SLO) violations. The question is not just how often to sync, but which specific windows to choose to avoid high-cost periods, and whether a policy can reliably beat a random placement of the same synchronization budget.

Methodology

1. The Calibration Protocol

The authors propose a rigorous evaluation protocol to determine if a synchronization policy earns a claim of improving serving SLOs. The protocol introduces a Matched Random Deferral control:

Concept: A non-deployable control that inherits the exact synchronization budget (number of merges) of the evaluated policy but randomizes the placement of those merges within the feasible time corridor.
Purpose: This separates synchronization frequency from window placement. If a workload-aware policy cannot beat a random placement of the same budget, it has not demonstrated "placement gain."
Gate: A claim of SLO improvement is only valid if the policy beats the matched-random envelope in a calibrated regime.

2. Workload-Aware DiLoCo (WA-DiLoCo)

The paper instantiates this protocol with WA-DiLoCo, a lightweight, score-based controller:

Inputs: Learner progress (loss reduction, token contribution, staleness) and fleet pressure (network, serving, checkpoint).
Scoring: A linear score $s_i$ is computed for each learner, weighing progress positively and pressure negatively. Serving pressure carries the highest weight in the target setting.
Decision Logic: The coordinator synchronizes if the gap since the last merge exceeds $H_{min}$ (128 steps) and the average score exceeds a threshold, or if the gap reaches $H_{max}$ (512 steps). This ensures merges stay within a bounded corridor.

3. Calibrated Overlap and Forecasting

For bursty workloads, the authors identify that direct overlap with sidecar requests is a more precise cost signal than general pressure metrics.

Offline Calibrated-WA: An oracle that observes the full burst trace and reselects the lowest-overlap feasible windows. This establishes the theoretical upper bound of what is possible.
Online Forecasting: To make this deployable, the authors augment the controller with a one-step Exponential Weighted Moving Average (EWMA) forecast of request rates. This allows the controller to score candidate windows based on expected overlap before the merge is launched.

Key Contributions

Calibrated Evaluation Protocol: The paper identifies a gap in existing literature where fixed-period baselines conflate budget and placement. The protocol mandates real-sidecar effect-size transfer, a no-sync load match, and a matched-random envelope before claiming SLO improvements.
Matched Random Deferral Control: Introduces a non-deployable control that inherits the evaluated scheduler's merge budget but randomizes placement. This serves as a strict baseline; a scheduler that cannot beat this random placement has only controlled the budget, not the workload.
Bounded Calibrated Scheduling Formulation: The authors formalize synchronization as feasible placement within a min/max-H corridor. They prove that under a calibrated overlap model, offline calibrated scheduling is near-optimal within the matched-budget class, while the online variant degrades gracefully with forecast error.
Extensive Real-Sidecar Regime Study: The study evaluates policies across ordinary, pressure-heavy, steady sync-heavy, bursty, and non-periodic serving regimes using real vLLM sidecar replays.

Results

The experiments reveal that the efficacy of scheduling depends heavily on the workload regime:

Ordinary/Pressure-Heavy Regimes: In ordinary sidecar traces, synthetic pressure effects did not transfer to real latency (p95 ratios $\approx$ 1.0). In pressure-heavy regimes, all policies were saturated, and no policy dominated.
Steady Sync-Heavy Regimes: WA-DiLoCo outperformed fixed-interval and pressure-gate baselines. However, it tied the matched-random deferral control statistically ( $p=0.56$ ), indicating that for steady loads, simple random placement is often sufficient, and no strong dominance claim can be made.
Bursty Regimes: This is where calibration yields value.
- Offline Calibrated-WA: Reduced sync-excess SLO violations to 4.45%, significantly beating the matched-random envelope (6.26%) and WA-DiLoCo (6.92%).
- Online Forecaster: The EWMA-augmented online controller reduced SLO violations from 6.54% (matched random) to 5.09% in real vLLM sidecar replays ( $p=0.021$ ). This represents a 1.46 percentage point placement gain, reducing SLO violations by roughly 60 requests per sidecar per ten minutes at the tested throughput.

Significance and Claims

The paper's primary significance is methodological rather than purely algorithmic. It argues that not every sync is safe, and claiming SLO improvements requires a higher bar of evidence.

The Protocol First: The authors claim that the "deployable lesson" is the protocol itself: researchers must report real-sidecar effect-size transfer, a no-sync load match, and a matched-random envelope before claiming serving-SLO improvements.
Modest Claims on Deployability: The paper is modest about the current state of the online controller. While the offline oracle proves exploitable structure exists in bursty traces, the online controller (without forecasting) did not significantly beat matched random. The successful deployment claim rests specifically on the forecast-augmented variant in a specific non-periodic trace family.
Limitations: The authors explicitly state that these are stress-tested replay studies, not live production deployments. The 8-second merge duration is configuration-specific, and the online controller remains an "incomplete implementation" of the calibrated scheduling idea, requiring further tuning of weights and guardrails for fairness and starvation in future deployments.

In conclusion, the paper demonstrates that while DiLoCo reduces communication, the scheduling of the remaining synchronization events is a critical control problem in shared fleets. By introducing a matched-random baseline and a calibration protocol, the authors show that workload-aware scheduling only yields measurable SLO gains when the workload has exploitable structure (like bursts) and when the controller can accurately forecast or observe that structure.