🤖 AI

Does Distributed Training Undermine Compute Governance?

This paper argues that advances in distributed training algorithms could allow developers to evade compute governance regulations by using decentralized hardware agglomerations, necessitating new countermeasures such as chip tracking, forensic accounting, and enhanced monitoring thresholds to detect and prevent illicit operations.

Original authors: Robi Rahman

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Robi Rahman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world's most powerful AI models are like massive, super-advanced factories. Currently, governments are trying to regulate these factories by setting up checkpoints. The rule is simple: "If you build a factory bigger than a certain size (measured by how much computing power it has), you must register it, let us inspect it, and prove you are safe."

The logic behind this is that big factories are hard to hide. They use huge amounts of electricity, take up a lot of space, and generate a lot of heat. If you try to build a secret super-factory, satellites and power companies will likely spot it.

The Problem: The "Lego" Loophole
This paper argues that a new technology called Distributed Training changes the game. Instead of building one giant factory, a developer could break the work up into thousands of tiny, separate workshops scattered all over the world.

Think of it like this:

The Old Way: Building one massive, 50-story skyscraper. It's impossible to hide; everyone sees the construction crane and the power lines.
The New Way: Instead of one skyscraper, you build 1,000 small, single-story sheds in different neighborhoods. Each shed is small enough that it doesn't trigger the "big building" alarm. But, if you connect all these sheds with a digital network, they can work together as if they were one giant skyscraper.

The paper claims that recent advances in software (specifically algorithms like DiLoCo) have made it possible to connect these tiny sheds over regular internet connections. Even though the internet is much slower than the super-fast cables inside a real data center, the new software is smart enough to compress the data being sent between sheds so much that the training still works.

The Core Finding: Can They Evade the Rules?
The author ran a simulation to see if a developer could build a "frontier" AI (one as smart as the best models currently in existence) using only these small, unregistered sheds.

The results are alarming for regulators:

Yes, it's possible. A developer could theoretically train a model as powerful as the current state-of-the-art (like Llama 3.1 or GPT-4) using only hardware clusters that are below the legal reporting threshold.
It's expensive, but doable. To pull this off, they would need to spend millions (or even billions) of dollars on hardware. However, if a developer is determined to bypass the rules, they might be willing to pay that price.
The "Speed" Trap. The paper assumes the developer has about two years to finish the project. If they try to train faster, they need more hardware and more people, which makes them easier to catch. If they train slower, they risk being overtaken by new technology. So, they aim for a "sweet spot" of about two years, which is long enough to build a powerful model but short enough to stay ahead of progress.

Why is this a problem?
If a developer can hide their operation in thousands of tiny, unregistered sheds, the government's current rules become useless. The government won't know the factory exists because no single shed is big enough to trigger an alarm. This means dangerous AI capabilities could be developed in the shadows without any safety checks.

The Proposed Solutions: How to Close the Loophole
The paper suggests that we can't just rely on "size" limits anymore. We need new ways to catch these "Lego" operations:

Track the Bricks (Chip Tracking): Instead of just counting how big the factory is, track every single "brick" (computer chip) from the moment it's made. If a chip is used in a secret shed, the registry should flag it.
The "Snitch" System (Whistleblowing): Running 1,000 sheds requires a huge team of people to install and maintain them. The paper suggests that the more people you need, the higher the chance someone will talk. Offering rewards for insiders who report secret operations could be very effective.
Check the Storage Shelves (Memory Limits): Currently, rules only look at how fast the chips are. But some chips have huge storage (memory) relative to their speed. The paper suggests setting a rule: "If your shed has more storage than 16 standard chips, you must register it," even if the speed is low. This forces the developer to use more sheds to get the same job done, making the operation bigger, more expensive, and easier to catch.
Surprise Inspections: Just like nuclear inspectors, governments should have the right to show up unannounced at suspected locations to check if they are part of a larger network.

The Bottom Line
The paper concludes that while distributed training makes it harder to regulate AI, it doesn't make it impossible. The "loophole" exists, but it's not a magic trick. It requires a lot of money, a lot of people, and a lot of coordination. By combining chip tracking, memory limits, and human intelligence (whistleblowers), regulators can still shine a light on these hidden operations and keep AI development safe.

Technical Summary: Does Distributed Training Undermine Compute Governance?

Problem Statement

Current and proposed compute governance frameworks (e.g., US Executive Order 14110, EU AI Act, California SB 53, and the Scher et al. 2025 proposal) rely on the assumption that training frontier AI models requires large, centralized datacenter clusters. These regulations typically mandate reporting and monitoring for clusters exceeding specific compute thresholds (e.g., $10^{24}$ to $10^{26}$ FLOPs) or hardware configurations (e.g., clusters larger than 16 H100-equivalents).

The core problem identified is that recent advances in distributed training algorithms, specifically the DiLoCo (Distributed Low-Communication) family, may allow developers to evade these regulations. By splitting a frontier-scale training run across thousands of smaller, geographically dispersed nodes—each falling below regulatory reporting thresholds—developers could theoretically conduct large-scale training without triggering detection mechanisms based on power consumption, physical footprint, or cluster size. This paper investigates whether such "illicit distributed training" is feasible under strict constraints (consumer-grade internet, sub-threshold hardware) and evaluates the efficacy of current and proposed countermeasures.

Methodology

The authors developed a comprehensive efficiency model and interactive simulator to evaluate the feasibility of distributed training evasion. The methodology involves:

Threat Model: The simulation assumes a well-resourced developer attempting to train models comparable to Llama 3.1-405B or DeepSeek-V3 under restrictive conditions:
- Hardware: Nodes limited to $\le$ 16 H100-equivalents (the threshold proposed by Scher et al., 2025).
- Network: Consumer-grade internet with 100 Mbps bandwidth and 100 ms latency (orders of magnitude slower than datacenter interconnects).
- Time: A maximum training duration of two years (740 days), derived from growth rate assumptions in a restrictive governance scenario.
- Detection: The governing body actively monitors power, thermal, satellite imagery, and network traffic.
Efficiency Metrics: The simulator calculates two primary metrics:
- Local-Equivalent Compute ( $C_{local}$ ): The nominal compute throughput adjusted for distributed inefficiencies ( $\eta$ ).
- Quality-Adjusted Compute ( $C_{quality}$ ): $C_{local}$ further adjusted by a Chinchilla-optimal penalty ( $\chi$ ) to account for suboptimal model sizing due to memory constraints and overtraining.
Inefficiency Factors ( $\eta$ ): The model accounts for four specific penalties inherent to distributed training:
- Sync Interval Penalty ( $\eta_H$ ): Loss from replicas performing local steps ( $H$ ) before synchronizing, causing gradient drift.
- Compression Quality ( $\eta_{comp}$ ): Loss from compressing gradients to save bandwidth (modeled at 150 $\times$ compression based on Covenant-72B results).
- Replica Divergence ( $\eta_{rep}$ ): Loss from averaging gradients across replicas trained on different data subsets.
- Activation Quality Penalty ( $\eta_{act}$ ): Loss from compressing activations when using pipeline parallelism across nodes.
Hardware Selection: The model evaluates various sub-threshold configurations, including NVIDIA A100 80GB, GH200, and various TPU/Ascend configurations, favoring hardware with high memory-to-compute ratios to minimize communication overhead.

Key Contributions and Results

1. Feasibility of Evasion

The simulation demonstrates that distributed training can achieve frontier-scale compute ( $>10^{24}$ FLOPs) using hardware configurations that remain below proposed regulatory thresholds.

Cost of Evasion: The paper estimates that a developer could evade:
- The Scher et al. (2025) threshold ( $10^{24}$ FLOPs) for approximately $1.6 million.
- The EU AI Act threshold ( $10^{25}$ FLOPs) for approximately $31 million.
- The California SB 53 threshold ( $10^{26}$ FLOPs) for approximately $3.8 billion.
Node Counts: Achieving the compute level of GPT-4 ( $10^{25}$ FLOPs) requires roughly 101 sub-threshold nodes, while Llama 3.1-405B ( $10^{25}$ FLOPs) requires 625 nodes.
Network Constraints: Even with low bandwidth (100 Mbps), distributed training remains feasible. While costs increase roughly 3 $\times$ compared to centralized training, this is not a prohibitive barrier for determined actors.

2. Hardware and Architecture Optimization

The study identifies that memory-to-compute ratio is a critical factor for evasion. Nodes with high memory (e.g., 50 $\times$ A100 80GB) allow for larger models per node, reducing the need for pipeline parallelism and the associated activation compression penalties. The simulator shows that using Pipeline-Parallel (PP) DiLoCo becomes necessary at very large scales to fit models into memory, though this introduces efficiency penalties.

3. Evaluation of Countermeasures

The paper assesses various countermeasures to detect or prevent evasion:

Ineffective Measures:
- Bandwidth Caps: Infeasible to enforce globally; the cost of high-speed internet is negligible compared to hardware costs.
- Internet Traffic Monitoring: DiLoCo traffic is low-volume (<40 Mbps) and easily disguised via jitter, variable-rate streaming, and VPNs.
Effective Measures:
- Chip Tracking & Registries: Mandatory reporting of chip possession and transfer. Since distributed training requires more total hardware than centralized training to achieve the same compute, a registry tracking a high fraction of chips makes evasion difficult.
- Whistleblower Programs: Distributed operations require significantly larger workforces for procurement, installation, and maintenance across hundreds of locations, increasing the "attack surface" for insider reporting.
- Memory Thresholds: The paper proposes amending cluster definitions to include accelerator memory thresholds (e.g., 1,280 GB HBM) in addition to compute thresholds. This forces evaders to either use smaller models (causing severe overtraining) or shard models across more nodes, increasing costs and operational complexity by ~50%.
- Challenge Inspections & Intelligence: Conventional financial and physical inspections remain viable, particularly when combined with the above technical measures.

Significance and Claims

The paper claims that distributed training fundamentally undermines current compute governance proposals that rely solely on cluster size or compute throughput thresholds. The authors argue that without specific countermeasures, regulations will create an "enforcement gap" where frontier models can be developed illicitly.

The significance of the work lies in:

Identifying the Vulnerability: Demonstrating that algorithmic advances (DiLoCo) have outpaced the static assumptions of current governance frameworks.
Quantifying the Risk: Providing concrete cost estimates and hardware configurations required to bypass specific, real-world regulations.
Proposing Robust Solutions: Arguing that a combination of chip registries, memory-based thresholds, and whistleblower programs is necessary to close the loophole. The authors emphasize that while no single measure is perfect, their combination raises the cost and complexity of evasion to a level that restores monitorability.

The paper concludes that while the increased costs of distributed evasion are currently "economically unpalatable," they may become acceptable if regulatory bottlenecks tighten. Therefore, governance standards must be designed to be robust against such evasion immediately, rather than waiting for the threat to materialize fully.