📊 statistics

Transfer Learning in Nonparametric Regression with Deep ReLU Networks

This paper proposes a two-stage transfer learning framework for nonparametric regression using deep ReLU networks that pools data to estimate a common structure and then learns group-specific offsets, achieving optimal convergence rates that overcome the curse of dimensionality under hierarchical composition models.

Original authors: Junpeng Ren, Carlos Misael Madrid Padilla, Yanzhen Chen, Oscar Hernan Madrid Padilla

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junpeng Ren, Carlos Misael Madrid Padilla, Yanzhen Chen, Oscar Hernan Madrid Padilla

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern data science, researchers often face a problem of abundance mixed with scarcity. They possess massive amounts of information from one source, yet they need to make accurate predictions for a specific, smaller group where data is thin. Imagine a doctor who has studied millions of patient records from a general population but needs to diagnose a rare condition in a specific demographic with very few recorded cases. The challenge is to use the broad knowledge without letting it drown out the unique details of the specific group. This is the heart of transfer learning, a method that tries to borrow strength from a large, related dataset to improve performance on a smaller, target one. For decades, statisticians have known that simply pooling all data together often fails because it ignores the unique traits of the smaller group, while training a model only on the small group fails because there is not enough information to learn from. The question has been how to find the perfect balance: how to learn the shared patterns that apply to everyone while still capturing the specific deviations that make a particular group unique.

A team of researchers has now proposed a new way to solve this puzzle, specifically for complex, non-linear relationships where the rules of the data are not simple straight lines. They focused on a powerful type of computer model known as a deep neural network, which is famous for its ability to find intricate patterns in high-dimensional data, such as images or text. The authors developed a two-step strategy that mimics how a human might approach a difficult problem by first understanding the big picture and then zooming in on the details. In the first step, the computer looks at data from every available group combined together to learn a general "average" function. This is not a simple average of numbers, but a complex mathematical shape that captures the common structure shared by all the groups. Once this general shape is learned, the computer moves to the second step. Here, it looks at just one specific group and calculates the difference, or "offset," between that group's reality and the general average it just learned. By adding this specific difference back to the general average, the computer creates a final model that is both broadly informed and locally precise.

The researchers tested this approach using deep neural networks, which are designed to handle data with many variables, a task that often confounds traditional statistical methods. They found that this two-stage process works remarkably well, allowing the models to overcome a major hurdle known as the "curse of dimensionality." This is a phenomenon where the amount of data needed to learn a pattern grows exponentially as the number of variables increases, often making learning impossible in high-dimensional settings. By breaking the problem into a shared part and a specific part, the new method effectively reduces the complexity of what the computer has to learn at each stage. The shared part is learned from the entire dataset, providing a robust foundation, while the specific part is often much simpler than the whole, requiring far fewer data points to estimate accurately.

To prove their theory, the team ran extensive computer simulations with thousands of data points across various scenarios, including low-dimensional and high-dimensional settings. In these tests, their two-stage method consistently outperformed other common strategies. For instance, it beat models that tried to learn everything from scratch using only the small group's data, as well as models that simply ignored group differences and treated everyone as the same. In one high-dimensional scenario involving a hundred variables, the new method achieved significantly lower errors than its competitors, demonstrating that it could extract the signal from the noise more effectively. The researchers also applied their method to a real-world dataset of facial images to estimate the age of people from different ethnic backgrounds. In this test, the two-stage approach again produced the most accurate results, successfully leveraging the shared visual features of aging while accounting for the distinct characteristics of each ethnic group.

The study suggests that this framework is not just a theoretical curiosity but a practical tool for improving how machines learn from grouped data. The authors showed that the method works even when the groups are of very different sizes, a common situation in real life where some populations are well-represented in data and others are not. They also demonstrated that the method remains robust even when the groups are not perfectly similar, provided that the differences between them are not too extreme. While the paper focuses on the mathematical guarantees and simulation results, the underlying message is clear: by separating the universal from the specific, deep learning models can become more efficient and accurate. This approach offers a promising path forward for fields ranging from epidemiology, where disease patterns vary by region, to personalized medicine, where treatments must be tailored to individual patient profiles, all without requiring an impossible amount of data for every single subgroup.

Technical Summary: Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Problem Formulation
This paper addresses the problem of nonparametric regression in settings where data consists of multiple distinct groups that share structural similarities but exhibit group-specific deviations. The authors formalize the data generation process as:
$y_i = f_0(x_i) + f_{0, z_i}(x_i) + \epsilon_i$
where $f_0$ represents a shared function common to all groups, and $f_{0, \ell}$ denotes the group-specific deviation for group $\ell$ . The objective is to estimate the group-specific conditional mean function $g_{0, \ell}(x) = f_0(x) + f_{0, \ell}(x)$ . The framework is designed to handle scenarios where the number of groups $L$ may grow with the total sample size $n$ , reflecting modern machine learning contexts with diverse and large-scale datasets.

Methodology: Two-Stage Offset Learning
The proposed method employs a two-stage offset learning procedure inspired by pretraining:

Stage 1 (Shared Structure Estimation): Data from all $L$ groups are pooled to estimate the overall conditional mean function $\bar{f}(x) = E[Y|X=x]$ . This is achieved by minimizing the empirical risk over a function class $\mathcal{F}$ (e.g., deep ReLU networks) using the combined dataset.
Stage 2 (Group-Specific Offset Estimation): For each target group $\ell$ , the method estimates the offset between the overall mean and the group-specific mean. Specifically, it estimates the function $G_\ell(x) = f_{0, \ell}(x) - \sum_{k} f_{0, k}(x)P(Z=k|X=x)$ using only the data from group $\ell$ . This is done by minimizing the empirical risk over a group-specific function class $\mathcal{F}_\ell$ on the residuals $y_i - \hat{f}_{A_n}(x_i)$ .
Final Estimator: The final estimator for group $\ell$ is constructed additively: $\hat{g}_\ell(x) = \hat{f}_{A_n}(x) + \hat{f}_{\ell, B_n}(x)$ .

The framework supports two variations: one where stages share data and another where they use independent data (via sample splitting).

Key Contributions and Theoretical Results

General Theoretical Framework: The paper establishes $L_2$ error upper bounds for a broad class of nonparametric estimators under mild complexity and noise conditions (accommodating sub-exponential noise). Theorem 3.2 provides a general bound decomposing the error into first-stage estimation error and second-stage approximation/estimation errors.
Deep ReLU Networks and the Curse of Dimensionality: When instantiated with deep ReLU networks under hierarchical composition models, the authors derive explicit convergence rates. Theorem 3.4 demonstrates that the proposed transfer learning framework preserves the ability of deep networks to overcome the curse of dimensionality. The convergence rate is shown to be the sum of the rates for estimating the shared function and the offset function, rather than the rate for the full complex function.
Conditions for Positive Transfer: The paper identifies regimes where transfer learning yields strictly faster convergence rates than single-group estimation. This occurs when the shared function $\bar{f}$ and the offset $G_\ell$ are less complex (in terms of smoothness and intrinsic dimension) than the target function $g_{0, \ell}$ . For instance, if the offset is smoother or lower-dimensional than the full function, the two-stage approach leverages the larger pooled sample size for the complex part and the specific sample size for the simpler residual.
Sample Splitting: Theorem 3.7 and Corollary 3.8 analyze the case where Stage 1 and Stage 2 use independent data. This variant relaxes certain complexity constraints on the function classes and can yield tighter convergence rates, particularly when the proportion of the target group $\pi_\ell$ is small.
Generality: The framework is shown to be applicable beyond neural networks, providing the first convergence guarantees for trend filtering in a transfer learning setting and recovering existing rates for orthogonal series regression.

Empirical Results
The authors validate the method through extensive simulations and real-data experiments:

Simulations: Experiments cover low-dimensional (additive and nonlinear mixtures) and high-dimensional settings (latent factor models with shifted representations). Across diverse scenarios, the proposed "2-Stage" deep neural network estimator consistently outperforms competitors, including pooled training, separate training, fine-tuning only the output layer (Top-FT), and pooling with labels (Pool-w-L). It also outperforms Random Forests and parametric competitors (ptLasso) in high-dimensional settings.
Real-Data Applications:
- Beijing PM2.5: A low-dimensional study demonstrating effectiveness in environmental data.
- UTKFace: A high-dimensional facial age estimation task across five ethnic groups. The two-stage transfer learning approach achieved the lowest overall Mean Squared Error (MSE), performing optimally on White, Black, and Asian subgroups and showing substantial improvement over pooled baselines for the Indian subgroup.

Significance and Claims
The paper claims to provide a general theoretical foundation for transfer learning in nonparametric regression, specifically bridging the gap between the empirical success of deep learning and theoretical guarantees in multi-group settings. Its primary significance lies in:

Theoretical Justification: Offering rigorous convergence rates that explain why pretraining on pooled data followed by offset learning can be superior to training on target data alone, particularly when the underlying functions exhibit hierarchical or compositional structures.
Curse of Dimensionality: Demonstrating that deep ReLU networks within this framework can effectively mitigate the curse of dimensionality by exploiting the intrinsic low-dimensional structure of the shared and offset components.
Robustness: Showing that the method remains robust even when the target group size is small relative to the total dataset, a common challenge in practical transfer learning.

The authors maintain a modest stance regarding optimality, noting that lower bounds for hierarchical function classes are not fully established in general cases. They also acknowledge that negative transfer is theoretically possible if the offset is highly complex or the groups are dissimilar, though their experiments suggest strong cross-group similarity in real-world datasets. Future work is suggested to extend the framework to continuous task variations, more advanced architectures (CNNs, Transformers), and the formal detection of negative transfer.

Technical Summary: Transfer Learning in Nonparametric Regression with Deep ReLU Networks

More like this