Technical Summary: Unextractable Protocol Models (UPMs)

1. Problem Statement

The paper addresses the economic and security challenges of decentralized training for large foundation models. While decentralized systems allow for the pooling of distributed compute resources to train massive models, they face a critical "appropriability problem": if the full model weights are accessible to participants, contributors cannot recoup their training costs because the model can be extracted and used outside the protocol.

Current decentralized approaches (e.g., Data Parallelism or standard Pipeline Parallelism) often require participants to hold full model replicas or allow the reconstruction of full weights over time. This leads to weight materialization, where an attacker (or coalition) can eventually access all model shards across different time steps to reconstruct a functional, extractable model. The paper argues that for decentralized training to be economically viable, the model must be unextractable—ensuring that no single participant ever possesses the full weight set, and that weights collected at different times cannot be stitched together to form a coherent model.

2. Methodology: Unextractable Protocol Models (UPMs)

The authors propose Unextractable Protocol Models (UPMs), a framework that leverages Pipeline Parallelism (PP) to enforce weight unextractability without relying on heavy cryptographic primitives.

Core Mechanism: Time-Varying Invertible Transforms

The central innovation is the periodic injection of random, invertible transforms at the boundaries between pipeline stages.

• Setup: The model is sharded into $S$ sequential pipeline stages. Participants hold and process only one stage at a time.
• Morphing: At specific time steps, the protocol introduces an identity function between adjacent stages $f_i$ and $f_{i+1}$, decomposed into a random transform $T$ and its inverse $T^{-1}$.
  • The transform $T$ is folded into the exit weights of stage $i$ ($V_i \leftarrow V_i T$).
  • The inverse $T^{-1}$ is folded into the entry weights of stage $i+1$ ($U_{i+1} \leftarrow T^{-1} U_{i+1}$).
• Ephemeral Nature: The transforms are discarded immediately after folding. They exist only within the weights.
• Incompatibility: While the end-to-end network function remains mathematically identical at any single time step $t$, weights collected at different time steps ($t$ and $t'$) become incompatible. Stitching $f_i(t)$ with $f_{i+1}(t')$ fails because the intermediate activation space has been transformed by $T(t)$ but the subsequent stage expects the space transformed by $T(t')$. Reconstructing the model requires guessing the "bridge matrix" of all intermediate transforms, which is computationally infeasible.

Handling Architectural Components

The paper details how to apply these transforms to various neural network components while preserving functionality:

• Valid Subfunctions: The method applies to subfunctions of the form $g(X) = \Phi(XU)V$. Transforms are applied to $U$ and $V$.
• Skip Connections: In Transformers, skip connections naturally enforce full incompatibility. The transform applied to the skip connection depends on both the incoming and outgoing boundary transforms, preventing an attacker from partitioning the stage into independent sets of weights.
• Normalization Layers (RMSNorm): Since normalization is non-linear, standard folding is difficult. The authors propose introducing an accumulation matrix $Q$ into the normalization layer and absorbing the scaling weights into the subsequent linear layer. This allows transforms to be folded into $Q$ and the linear weights while preserving the RMSNorm property due to the orthogonality of the initial $Q$.
• Transform Classes: To ensure numerical stability and resistance to brute-force attacks, the authors utilize:
  • Haar Orthogonal Matrices: Unit singular values, minimal numerical error, and uniform randomness.
  • Low-Condition Number Matrices: Matrices of the form $UDV^T$ with controlled singular values to introduce high-frequency perturbations while bounding floating-point error.

Training and Inference Adjustments

• Training: The framework requires specific handling of gradients. If weights transform as $W(T) = WT$, gradients transform as $G(T) = GT^{-T}$. The authors use the Muon optimizer, which relies only on the first moment of gradients, allowing the update steps to remain consistent with the untransformed model when using orthogonal transforms.
• Inference: To mitigate floating-point error accumulation from repeated transform folding, the system stores high-precision (FP64) weights on disk. At each morphing step, transforms are applied to the high-precision copy, cast to low precision (e.g., FP16/BF16), and loaded into GPU VRAM.

3. Key Contributions

1. Framework Introduction: The proposal of UPMs, a training and inference framework that enables collaborative model serving without allowing any participant to extract the full weight set.
2. Theoretical Analysis: A thorough analysis of which architectures (Transformers, MLPs, RNNs) support UPMs, the requirements for transforms (randomness, invertibility, condition number), and the mathematical proof of how these transforms render cross-time assemblies incoherent.
3. Attack Resistance:
   • Direct Extraction: Demonstrated that the probability of an attacker accessing all stages within a short time window is negligible, especially with "full incompatibility" (skip connections).
   • Stitching Attacks: Showed that solving for bridge matrices via intermediate activations is impractical due to the requirement of identical inputs across time steps, which is easily defended against via distribution checks.
   • Learning-Based Attacks: Empirically demonstrated that fine-tuning a "stitched" model (weights from different time steps) requires $\ge 60\%$ of the compute resources needed to train the model from scratch, preserving the economic viability of the protocol.
4. Empirical Validation: Extensive experiments on Qwen-2.5-0.5B and Llama-3.2-1B models.

4. Results

• Functional Equivalence: On Qwen-2.5-0.5B and Llama-3.2-1B, applying 10,000 morphing steps with orthogonal transforms resulted in negligible performance degradation ($\Delta PPL < 0.01$; Jensen–Shannon drift $< 4 \times 10^{-5}$) in FP32. With the high-precision workaround, stability was maintained across FP16 and BF16.
• Overhead:
  • Inference: Applying a transform every 30 seconds adds approximately 3% latency, 0.1% bandwidth, and 10% GPU memory overhead.
  • Training: Overhead is minimal, adding 1.6% time and <1% memory overhead, as the backward pass dominates costs.
• Attack Feasibility:
  • Stitching: Fine-tuning a stitched Llama 3.2-1B model required roughly 60% of the tokens needed to train from scratch to reach the same perplexity.
  • Excludability Ratio: Even with access to all incompatible weights, the excludability ratio (compute required to attack vs. train from scratch) remains high ($\ge 0.6$), making the attack economically unviable for most coalitions.

5. Significance and Claims

The paper claims that UPMs resolve the contradiction between collaborative training and weight secrecy. By binding the model's value to the protocol rather than static weights, UPMs make it practical to embed programmatic incentive mechanisms in community-driven decentralized training.

• Economic Viability: The framework ensures that the "excludability" of the model is maintained, allowing contributors to be compensated for their compute without the risk of the model being stolen and used freely.
• Open Source Compatibility: UPMs allow model architectures, hyperparameters, and datasets to remain open-source while keeping the weights unextractable.
• Trustless Decentralization: The system moves toward fully trustless, unextractable training, positioning UPMs as a technical pillar for open, decentralized AI.

The authors acknowledge limitations, noting that security relies on an honest majority of participants and that current evaluations are limited to single-machine simulations, excluding real-world side-channel attacks (timing, cache). They also note that the inability to inspect weights could hinder safety evaluations, suggesting that diverse stakeholder monitoring is necessary to mitigate this.