Technical Summary: Z-Plane Neural Networks

Problem Statement

The scalability of deep learning architectures is currently constrained by the inherent instability of Euclidean vector spaces. In standard Multi-Layer Perceptrons (MLPs), successive matrix multiplications often lead to exploding or vanishing gradients. To mitigate this, the field has relied on heuristic workarounds:

• Rectified Linear Units (ReLU): While effective at preventing negative value propagation, ReLU irreversibly discards negative information and causes "dead neurons."
• Layer Normalization (LayerNorm): This technique enforces a global scale to stabilize training but "brutally" destroys the pairwise orthogonality of feature representations.

The authors argue that these mechanisms fail to preserve critical directional information and the geometric integrity of feature spaces. Furthermore, existing approaches like Complex-Valued Neural Networks (CVNNs) and Unitary RNNs either fail to strictly bound energy in deep settings or impose strict orthogonality that limits representational capacity and increases computational overhead.

Methodology: The Z-Plane Architecture

Inspired by the frequency-modulation transmission of biological axons—where signal amplitude serves only as a transient trigger while information is encoded in frequency and phase—the authors propose the Z-Plane Neural Network. This architecture replaces scalar activations with a geometric approach based on 2D phasor bundles.

1. 2D Phasor Bundles

Instead of representing hidden states as scalar vectors $h \in \mathbb{R}^D$, the network maps them into 2D bundles $X \in \mathbb{R}^{B \times (D/2) \times 2}$. Each pair $(x, y)$ represents the real and imaginary components of a phasor.

• Linear Projection: During the linear layer (analogous to dendritic integration), these 2D bundles are mixed. Constructive interference increases vector magnitude ($\|X\|_2 > 1$), while destructive interference shrinks it ($\|X\|_2 < 1$). This fluctuation enables rich interactions and representational capacity.

2. Radial Bounding Activation

To introduce non-linearity and prevent energy explosion without discarding directional data, the authors introduce Radial Bounding as the activation function. For a 2D vector $v$, the function is defined as:
$$f(v) = \frac{v}{\max(1, \|v\|_2)}$$

• Identity Mapping: When $\|v\|_2 \leq 1$, the function acts as an identity ($f(v) = v$), preserving linear residuals and information.
• Geometric Refraction: When $\|v\|_2 > 1$, the magnitude is strictly bounded to 1, projecting the vector onto the unit hypersphere. This breaks the homogeneity of linear transformations, serving as a potent non-linearity.

3. Gradient Preservation and Stability

The authors provide mathematical proofs regarding the stability of this mechanism:

• Tangential Gradient Preservation: Unlike ReLU, which zeroes out gradients for negative values, Radial Bounding preserves the gradient of the phase. The Jacobian matrix for $\|v\|_2 > 1$ scales the tangential gradient by $1/R$ (where $R = \|v\|_2$), allowing phase rotation to be propagated backward safely.
• 1-Lipschitz Continuity: The bounded output intrinsically guarantees a 1-Lipschitz continuous system. This property eliminates the need for LayerNorm, as the system prevents gradient explosion by design.

Key Contributions

1. Z-Plane Neural Network: A novel architecture that maps hidden states to 2D phasor bundles on a hypersphere, shifting the paradigm from scalar amplitude to multi-dimensional phase interference.
2. Radial Bounding: A geometric activation function that limits energy magnitude while strictly preserving phase (direction). It acts as an isotropic multi-dimensional non-linearity.
3. Elimination of Standard Heuristics: The paper demonstrates that a single mechanism (Radial Bounding) can simultaneously provide non-linearity, bound system energy, and ensure stability, rendering ReLU and LayerNorm unnecessary.
4. Theoretical Guarantees: Proof that the activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients.

Experimental Results

To validate the stability of the proposed geometric activation, the authors trained an extremely deep 100-layer Z-Plane MLP on the MNIST dataset.

• Configuration: The model consisted of 100 consecutive Z-Plane layers with no ReLU, GELU, Sigmoid, LayerNorm, or BatchNorm.
• Training: Trained for 20 epochs using the AdamW optimizer (learning rate $5 \times 10^{-4}$).
• Performance:
  • The network converged smoothly, achieving a 98.34% training accuracy and 96.89% test accuracy.
  • In contrast, standard Euclidean MLPs of this depth without normalization suffer from catastrophic failure (gradient explosion leading to NaN loss).
• Conclusion: The results confirm that Z-Plane Radial Bounding acts as an intrinsic regularizer, enabling absolute numerical stability in deep architectures.

Significance and Claims

The paper claims a paradigm shift from scalar amplitude-based processing to multi-dimensional phase interference. The authors assert that:

• Mathematical Purity: The architecture provides a mathematically pure foundation for deep learning by removing the need for "Euclidean workarounds."
• Hardware Efficiency: By relying on geometric bounds rather than complex normalization layers, the approach offers a hardware-efficient foundation for next-generation networks.
• Sufficiency of Bounded Geometric Activation: The study proves that bounded geometric activation alone is sufficient to achieve stable deep learning, even at extreme depths (100 layers), without the auxiliary mechanisms currently standard in the field.