Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning

Technical Summary: Multiple Neural Operators Achieve Near-Optimal Rates for Multi-Task Learning

Problem Statement
The paper addresses the problem of multi-task operator learning, specifically the approximation of collections of operators within a shared neural network model. Unlike standard operator learning, which approximates a single map $G: U \to V$, multi-task learning seeks to approximate a map $G: W \to \{G[\alpha]: U \to V\}_{\alpha \in W}$, where $\alpha \in W$ acts as a descriptor for the specific operator or task (e.g., parameterized PDEs, varying boundary conditions).

The central challenge identified is the potential for parametric complexity. Previous work on the Multiple Neural Operators (MNO) architecture [47] established constructive approximation bounds that exhibited an "exponential blow-up" relative to standard single-operator learning. Specifically, the parameter count $N^\#$ required to achieve accuracy $\varepsilon$ scaled as $N^\# \lesssim \varepsilon^{-\varepsilon^{-\varepsilon^{-d_W}}}$, suggesting that multi-task learning might be intrinsically more complex than single-task learning. The paper investigates whether this complexity is an artifact of the constructive proof strategy or an intrinsic barrier of the problem class.

Methodology
The authors employ a combination of constructive approximation theory, statistical learning theory, and lower-bound complexity analysis.

1. Refined Constructive Approximation (MNO): The authors analyze the MNO architecture, defined as a separable sum of subnetworks:
   $$\text{MNO}[\alpha][u](x) = \sum_{p=1}^P \sum_{k=1}^H l_p(M_W(\alpha)) b_{pk}(M_U(u)) \tau_{pk}(x)$$
   They introduce a parallel error analysis strategy (contrasted with the previous "nested" strategy). In the nested approach, the error of the second stage (approximating the input function dependence) accumulates over the number of terms in the first stage (parameter dependence), forcing a much tighter per-term accuracy and inflating complexity. In the new parallel approach, the coefficients $l_p$ form an approximate partition of unity ($\sum l_p \approx 1$). This decouples the second-stage error from the number of first-stage terms, allowing the second stage to operate at the target accuracy $\varepsilon$ rather than a much smaller $\varepsilon^{\gamma+1}$.

2. Statistical Generalization: Using the refined approximation bounds, the authors derive statistical learning rates by combining the approximation error with the metric entropy of the hypothesis class, following the framework established in [46].

3. Lower-Bound Theory: To establish minimax rates, the authors extend the lower-complexity framework of [26] (based on infinite-dimensional cubes and pathological operators) to the multi-task setting. They define multiple operator maps of neural network type and symmetric multiple operator maps, proving that for broad classes of Lipschitz and differentiable maps, any neural network achieving accuracy $\varepsilon$ must satisfy a lower bound of the form $C(NN) \gtrsim \exp(c \varepsilon^{-1/\eta})$.

4. Architectural Comparison: The paper analyzes a concatenated DeepONet baseline, where the operator descriptor $\alpha$ and input $u$ are concatenated and fed into a single network. They derive both upper and lower bounds for this architecture to compare its scaling laws with MNO.

Key Contributions and Results

• Near-Optimal Approximation Rates: The primary theoretical contribution is a substantial improvement in the constructive upper bounds for MNO. The authors prove that for Lipschitz multiple operator maps, the total number of nonzero parameters $N^\#$ required to achieve accuracy $\varepsilon$ satisfies:
  $$N^\# \lesssim \exp\left( d \log(\varepsilon^{-1}) \varepsilon^{-\max\{d_W, d_U\}} \right)$$
  Inverting this relationship yields the error scaling:
  $$\varepsilon \lesssim \left( \frac{\log N^\#}{\log \log N^\#} \right)^{-1/\max\{d_W, d_U\}}$$
  These rates are qualitatively identical to those known for single-operator learning, demonstrating that the previous "exponential blow-up" was an artifact of the proof technique, not an intrinsic property of multi-task learning.

• Refined Statistical Learning Rates: Consequently, the statistical generalization error for MNO scales as:
  $$O\left( \left( \frac{\log n_\alpha}{\log \log n_\alpha} \right)^{-2/\max\{d_W, d_U\}} \right)$$
  where $n_\alpha$ is the number of sampled operators. This matches the scaling laws of single-operator learning, confirming that shared representations do not increase the overall statistical cost.

• Lower Complexity Bounds and Minimax Rates: The paper establishes a curse of parametric complexity for multi-task learning. They prove that for broad classes of Lipschitz and differentiable multiple operator maps, the lower bound on complexity is:
  $$C(NN) \gtrsim \exp\left( c \varepsilon^{-1/[(\eta+1+\delta)r]} \right)$$
  Combining this with the improved upper bounds, the authors derive minimax approximation-complexity rates. While the upper and lower bounds differ in the precise exponent, they both confirm a double-exponential complexity regime, indicating that while the specific "blow-up" of the previous work is avoidable, a genuine exponential barrier remains for general Lipschitz classes.

• Comparison with Concatenated DeepONet: The authors prove that the concatenated DeepONet architecture (where $\alpha$ and $u$ are concatenated) satisfies essentially the same minimax approximation-complexity scaling laws as MNO. Specifically, both architectures share the same worst-case complexity bounds.

Significance and Claims
The paper claims to resolve a fundamental question regarding the complexity of multi-task operator learning: shared representations across tasks do not increase the overall cost. The results demonstrate that multi-task operator learning follows the same scaling laws as single-operator learning.

The significance of the work lies in:

1. Clarifying Complexity Sources: It distinguishes between artifacts of constructive analysis (the previous exponential blow-up) and intrinsic barriers (the double-exponential curse of parametric complexity).
2. Theoretical Validation of MNO: It provides a theoretical justification for the MNO architecture, showing it achieves near-optimal rates without the previously suspected intrinsic penalty.
3. Architectural Equivalence in Worst-Case: The finding that MNO and concatenated DeepONet share the same minimax rates suggests that the empirical superiority of MNO observed in prior work [47] (where MNO outperformed concatenated DeepONet by orders of magnitude) is likely due to structural or geometric properties specific to the data and tasks, rather than a difference in worst-case approximation complexity. The paper posits that MNO's explicit separation of parameter and input roles is beneficial in practice, even if minimax rates do not distinguish the two architectures.

The authors conclude that while Lipschitz regularity alone cannot overcome the curse of parametric complexity, exploiting additional structures (such as holomorphic dependence or PDE-specific regularity) remains a promising direction for achieving substantially better rates.