The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Technical Summary: The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

Problem Statement
Modern deep learning architectures, particularly foundational models, increasingly rely on shared representations where a single backbone extracts features for multiple task-specific heads. While empirical evidence suggests that joint multi-task learning outperforms separate single-task learning, the rigorous theoretical understanding of the parametric complexity of joint versus separate approximation remains limited. This gap is particularly acute when models must satisfy structural constraints, such as orthogonality, which are common in scientific machine learning (e.g., ensuring conservation laws or equivariance). The central question addressed is whether a joint approximation of a vector-valued function $F = (f_1, \dots, f_M)$ can achieve a strictly better approximation rate than $M$ separate approximations of its components, even when the components are constrained to be orthogonal. The authors investigate this in the uniform norm ($L^\infty$), a regime critical for scientific applications where error guarantees must hold at arbitrary spatial or temporal locations, independent of sampling measures.

Methodology
The authors employ an information-theoretic framework based on finite-bit discretization (codec theory) to quantify the description length required to approximate function classes. They construct a specific class of orthogonal, vector-valued functions designed to isolate the complexity of the shared feature from the task-specific readouts.

1. Function Class Construction:

   • Shared Feature: The core difficulty is embedded in a scalar Rademacher-Haar wavelet series $g$. The coefficients of this series form a dyadic tree of independent signs, creating a "hard" target for uniform approximation where the unresolved tail carries the majority of the complexity.
   • Orthogonal Heads: To enforce orthogonality without eliminating shared information, the authors compose $g$ with Sawtooth-Walsh functions. These readouts, $f_i$, are non-linear maps that decorrelate the outputs (ensuring $\langle f_i \circ g, f_j \circ g \rangle = \delta_{ij}$) while preserving the approximation hardness within the shared feature $g$.
   • Composition: The target class is $\mathcal{F} = \{ (f_0 \circ g, \dots, f_{M-1} \circ g) \mid g \in \mathcal{G} \}$.

2. Coding Schemes:

   • Separate Coding: $M$ independent encoder/decoder pairs, each allocated a bit budget $n_i$, approximating each coordinate $f_i \circ g$ individually.
   • Joint Coding: A single encoder/decoder pair with a total bit budget $N = M \cdot \min(n_i)$ approximating the entire vector function simultaneously.

3. Neural Network Realization:

   • To bridge the gap between abstract codec theory and neural networks, the authors construct explicit Heaviside activation networks. They utilize the piecewise constant nature of truncated Haar wavelets and Sawtooth-Walsh functions.
   • Pointwise multiplication (required for the composition $f_i \circ g$) is realized via a recursive triangle-wave approximation (based on Yarotsky, 2017), allowing the network to approximate the product of the shared feature tail and the fixed readouts.

Key Results
The paper establishes a sharp separation in approximation rates between joint and separate coding for the constructed class:

• Theoretical Gap (Theorem 8): For a class of $M$ orthogonal functions generated by a shared Rademacher-Haar feature, the ratio of the optimal separate approximation error to the optimal joint approximation error is bounded below by:
  $$\frac{\|F - \delta^{(S)}_D \circ \delta^{(S)}_E(F)\|_\infty}{\|F - \delta^{(J)}_D \circ \delta^{(J)}_E(F)\|_\infty} \geq \frac{M}{4} \cdot \frac{1 + 1/N}{1 + 2M^2/N}$$
  Asymptotically, this yields a gap of order $M/4$. This demonstrates that a separate codec must encode the shared "hard" feature $M$ times (once per coordinate), whereas a joint codec encodes it only once, regardless of the orthogonality constraint on the outputs.

• Optimality of Truncation: The authors prove that truncating the Rademacher-Haar series is an optimal finite-bit discretization map for this class, providing explicit upper and lower bounds with exact constants. This avoids reliance on asymptotic metric entropy estimates.

• Neural Network Realization (Corollary 19): The separation is realized by explicit Heaviside neural networks. A joint architecture with a shared trunk (encoding the Haar feature) and fixed readout heads achieves the same bit-economy as the optimal joint codec. Conversely, $M$ separate networks with matched total description length suffer the $O(M)$ degradation in approximation rate.

Significance and Claims
The authors claim to provide the first rigorous mathematical treatment of the benefit of shared representations from an approximation-theoretic perspective, specifically under geometric constraints like orthogonality.

• Refuting Redundancy Arguments: The results clarify that the advantage of joint learning is not merely an artifact of statistical averaging, linear redundancy, or distributional assumptions. Instead, the gain stems strictly from the reduction in information complexity achieved by encoding a shared latent feature once rather than $M$ times.
• Orthogonality Compatibility: The study demonstrates that orthogonality constraints, often viewed as restrictive, do not preclude the efficiency of shared representations. By moving the constraint into the non-linear readout heads, the shared feature remains the sole source of approximation hardness.
• Foundation Model Insight: The work offers a theoretical justification for the "shared trunk, multiple heads" architecture in foundational models. It suggests that even when outputs must satisfy strict physical or geometric constraints (like orthogonality), the expensive feature extraction can be amortized across tasks, provided the tasks share a latent structure.

The paper concludes modestly, noting that while the mechanism is proven for this specific constructive class (Rademacher-Haar with Sawtooth-Walsh), identifying whether this factor-$M$ separation persists for broader classes of neural networks, smoother features, or other constraints (e.g., equivariance) remains an open problem.