Technical Summary: LightSplit

Problem Statement

Split Learning (SL) enables collaborative training of Deep Neural Networks (DNNs) by partitioning a model between resource-constrained clients and a powerful server. While this preserves raw data locality, it introduces two critical challenges at the "cut layer" where intermediate activations (smashed data) are transmitted:

1. Communication Overhead: Transmitting high-dimensional activation tensors (e.g., 4,096 dimensions for ResNet-18 on CIFAR-10) creates a significant bandwidth bottleneck, scaling linearly with activation dimensionality and the number of clients.
2. Privacy Vulnerability: The smashed data contains rich, instance-specific information. Under a semi-honest (honest-but-curious) threat model, a server can exploit these representations to perform reconstruction attacks (e.g., SDAR, UnSplit, FORA), recovering client inputs with high fidelity.

Existing defenses typically address these issues in isolation. Privacy-focused methods (e.g., noise injection, perturbation) often fail to reduce communication payloads, while compression techniques (e.g., top-k sparsification, learned codecs) often preserve the very instance-specific signals exploited by reconstruction attacks or introduce prohibitive client-side computational overhead.

Methodology: LightSplit

The authors propose LightSplit, a framework that simultaneously reduces communication overhead and limits information exposure through a lightweight, fixed orthogonal random projection applied at the cut layer.

Core Mechanism

1. Fixed Orthogonal Projection:

   • Each client projects its $d$-dimensional smashed activation $z$ into a lower-dimensional subspace $k$ ($k \ll d$) using a fixed, random orthonormal matrix $R \in \mathbb{R}^{d \times k}$.
   • The projection is derived via the QR decomposition of a Gaussian random matrix, ensuring $R^\top R = I_k$.
   • This projection is non-trainable and fixed throughout training, requiring no additional learnable parameters on the client side.
   • Theoretical Basis: The method leverages the Johnson-Lindenstrauss (JL) lemma, which guarantees that pairwise Euclidean distances (and thus class separability) are approximately preserved in the lower-dimensional space.

2. Server-Side Lift-Back:
   To maintain compatibility with existing SL architectures, the server must reconstruct a $d$-dimensional input for its backbone. LightSplit offers two modes:

   • LIGHTSPLIT-F (Fixed): Uses a parameter-free linear lift-back ($\hat{z} = R\tilde{z}$), effectively projecting the received vector back onto the $k$-dimensional subspace. This is computationally minimal.
   • LIGHTSPLIT-L (Learned): Employs a small, trainable Multi-Layer Perceptron (MLP) to map the $k$-dimensional projection back to $d$ dimensions. This allows the server to adapt to the projected distribution, compensating for the information loss, at the cost of additional server-side parameters.

3. Within-Class Compaction (WCC) Regularizer:
   To address the limitation that random projection alone may retain residual instance-specific signals, LightSplit introduces a client-side regularization term.

   • Objective: Minimize the intra-class variance of the projected representations $\tilde{z}$.
   • Mechanism: The client computes the mean $\mu_c$ of projected samples for each class $c$ and minimizes the squared Euclidean distance between samples and their class centroid: $L_{WCC} = \sum_c \frac{1}{|S_c|} \sum_{i \in S_c} \|\tilde{z}_i - \mu_c\|_2^2$.
   • Effect: This encourages samples of the same class to form compact clusters, effectively discarding instance-specific variations that reconstruction attacks rely on, while preserving class-discriminative structure.

Key Contributions

1. Joint Privacy and Efficiency: LightSplit is the first method to simultaneously reduce per-sample payload (communication) and limit information exposure (privacy) without requiring additional client-side trainable components.
2. Irreversible Information Bottleneck: By discarding the orthogonal complement of the projection, the method creates an underdetermined inverse mapping for the server, making exact reconstruction of the original input mathematically impossible.
3. Lightweight Client Design: The client performs only a single matrix multiplication with a fixed matrix, preserving the low-compute premise of Split Learning for edge devices.
4. Class-Level Structure Preservation: The combination of JL projection and WCC regularization shifts the representation from an instance descriptor to a class prototype, enhancing robustness against reconstruction attacks.

Experimental Results

The authors evaluated LightSplit on CIFAR-10, CIFAR-100, Fashion-MNIST, and GTSRB under both IID and Non-IID settings, comparing against raw transmission and learned compression baselines (Learned 1x1).

• Privacy Performance:

  • LightSplit significantly degrades the performance of state-of-the-art reconstruction attacks (SDAR, FORA, UnSplit).
  • Against SDAR, SSIM scores dropped by up to 58.3% (at 32× compression) compared to raw transmission.
  • Against FORA, LPIPS scores increased by ~36%, indicating lower perceptual similarity to the original image.
  • Against UnSplit, LightSplit increased the Mean Squared Error (MSE) of reconstructions by up to 1.27× on MNIST, effectively preventing pixel-perfect recovery.
  • The method reduces the reconstruction capability of attacks by up to 7× compared to baselines.

• Utility and Efficiency:

  • Accuracy Retention: LightSplit retains >95% of the baseline accuracy in most scenarios. For example, on CIFAR-10 with 32× compression, accuracy drops only ~6% (vs. ~13% for some baselines).
  • Communication Reduction: The method achieves up to 32× reduction in transmitted dimensionality.
  • Stability: Training dynamics remain stable across varying compression ratios and client scales (up to 100 clients).
  • Backdoor Detection: The method does not interfere with server-side backdoor detection mechanisms; malicious clients remain identifiable via cosine-similarity signatures.

Significance and Claims

The paper claims that LightSplit addresses a critical gap in Split Learning where existing solutions force a trade-off between privacy and efficiency. By utilizing a fixed orthogonal projection and a variance-reducing regularizer, LightSplit:

• Mitigates Privacy Leakage: It limits the information available to a curious server to class-level prototypes rather than specific input instances, effectively raising the difficulty of reconstruction attacks.
• Reduces Overhead: It drastically lowers communication costs without the client-side training overhead associated with learned codecs.
• Maintains Compatibility: It integrates seamlessly into existing U-shaped Split Learning architectures without modifying the underlying network structure or requiring changes to the server's backbone (in the fixed mode).

The authors explicitly state that LightSplit does not provide formal privacy guarantees (e.g., Differential Privacy) but demonstrates robust empirical resistance against adaptive, state-of-the-art reconstruction attacks under a semi-honest threat model. They position the method as a practical, lightweight solution for deploying Split Learning in privacy-sensitive and bandwidth-constrained edge environments.