📊 statistics

Bagged Polynomial Regression and Neural Networks

This paper proposes Bagged Polynomial Regression with Random Projections (BPR), an interpretable ensemble method that achieves neural network-level accuracy in high-dimensional climate and environmental applications while offering superior transparency, auditability, and theoretical risk bounds.

Original authors: Sylvia Klosin, Jaume Vives-i-Bastida

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sylvia Klosin, Jaume Vives-i-Bastida

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Black Box" vs. The "Messy Kitchen"

Imagine you are trying to predict the weather or identify what kind of crop is growing in a field using satellite photos. You have thousands of data points (like temperature, soil moisture, cloud cover, and light reflection).

Neural Networks (NN) are like a super-smart, but secretive, master chef. They can taste a dish and tell you exactly what ingredients are in it with incredible accuracy. However, if you ask them how they figured it out, they just say, "I just know." Their internal logic is a "black box." In science and policy, this is risky because we need to know why a prediction was made to trust it.

Standard Polynomial Regression is like a student trying to write a recipe using a massive, messy kitchen. They try to write down every possible combination of ingredients (flour + sugar + eggs + flour + sugar + eggs...). The problem? As the number of ingredients grows, the number of possible recipes explodes. The student gets overwhelmed, makes mistakes, and the recipe becomes too complicated to read.

The Solution: "Bagged Polynomial Regression" (BPR)

The authors, Sylvia Klosin and Jaume Vives-i-Bastida, propose a new method called Bagged Polynomial Regression with Random Projections (BPR).

Think of BPR as a team of junior chefs working together, rather than one overworked master chef or one overwhelmed student.

The "Bagging" (The Team): Instead of one person trying to cook the whole meal, you hire 100 junior chefs.
The "Random Projections" (The Ingredients): You don't give every chef the whole pantry. Instead, you give each chef a random, small basket of ingredients (e.g., Chef A gets flour and sugar; Chef B gets eggs and milk; Chef C gets flour and eggs).
The "Polynomial" (The Recipe): Each chef writes a simple, easy-to-read recipe using only the ingredients in their basket. They might mix them in simple ways (like "flour + sugar" or "flour squared").
The "Average" (The Final Dish): At the end, you take all 100 recipes, average them out, and serve the result.

Why is this better?

Accuracy: Because you have so many chefs, the final dish tastes just as good as the master chef's (Neural Network).
Transparency: Because each chef only used a few ingredients, you can easily read their recipe. You can see exactly how "flour" affects the taste. You can even ask, "What happens if we add more sugar?" and get a clear answer.

The Science Behind the Magic

The paper uses math to prove two main things:

1. Avoiding the "Explosion" of Complexity
If you try to mix all ingredients at once, the number of possible combinations grows so fast it becomes impossible to calculate (this is called the "curse of dimensionality"). By splitting the ingredients into small groups (partitions), the math stays manageable. The authors prove that this method converges to the right answer much faster than trying to do it all at once.

2. Understanding the "Why" (Marginal Effects)
In environmental science, we often care about marginal effects (e.g., "If the vegetation gets greener, does the chance of it being soybeans go up or down?").

Neural Networks: When the authors tested this, the Neural Network's answer was essentially a flat line. It was so complex that it hid the relationship between greenness and soybeans. It was like a chef who knows the dish is good but can't explain which ingredient made it good.
BPR: The BPR method clearly showed that as the vegetation gets greener (higher NDVI), the chance of it being soybeans goes down. This matched real-world farming knowledge. The "team of chefs" approach made the relationship visible and easy to audit.

The Real-World Test: Crop Classification

To prove their idea works, the authors tested it on a real problem: identifying 7 different types of crops (corn, peas, canola, etc.) using satellite images from Manitoba, Canada.

The Result: BPR got a 99.7% accuracy rate.
The Comparison: This was just as good as (and slightly better than) the best Neural Networks used for the same task (which got about 99.2%).
The Bonus: Unlike the Neural Network, BPR provided clear, readable charts showing exactly how specific factors (like how green the plants looked) influenced the decision.

Summary

The paper argues that we don't always need "black box" AI to get great results. By using a "team approach" (bagging) and breaking big problems into smaller, manageable chunks (random projections), we can build models that are just as accurate as the complex ones but are transparent and easy to understand. This is crucial for fields like environmental policy, where we need to trust the model's logic, not just its output.

Technical Summary: Bagged Polynomial Regression with Application to Environmental Prediction

1. Problem Statement

Deep neural networks (NNs) have become the standard for high-dimensional prediction in environmental economics, such as flood detection, wildfire forecasting, and crop yield prediction. While NNs offer superior accuracy through over-parameterization, they function as "black boxes," making them difficult to audit, align with domain knowledge, or use for policy design where interpretability is required.

Standard polynomial regression offers a transparent, coefficient-based alternative but suffers from the "curse of dimensionality." As the number of covariates ( $d$ ) and polynomial degree ( $J$ ) increase, the number of dictionary basis terms ( $k$ ) explodes (e.g., $k \approx (J+1)^d$ ). This leads to:

Computational intractability: Inverting $k \times k$ matrices becomes impossible.
Slow convergence rates: The sample size required for consistent estimation grows exponentially with dimension, rendering standard polynomial regression impractical for high-dimensional settings.

The paper seeks an econometrics-native ensemble method that retains the interpretability of polynomial models while achieving competitive predictive accuracy in high-dimensional environments.

2. Methodology

Theoretical Framework: Partitioned Series Estimators

The authors build upon the series/sieve regression literature (Newey, Stone) to derive finite-sample and asymptotic $L_2$ risk bounds. They introduce Partitioned Polynomial Regression (PPR), where covariates are divided into $B$ groups, and polynomial features are constructed independently within each group rather than across all interactions.

Trade-off Analysis: Partitioning reduces the effective basis complexity (lowering estimation error) at the cost of increased approximation error. The authors derive conditions under which an intermediate level of partitioning ( $B^*$ ) improves convergence rates, particularly for smooth target functions ( $g \in \Sigma^s(X)$ ).
Marginal Effects: The theory is extended to the estimation of Average Partial Derivatives (APDs). The authors show that if the derivatives of the dictionary function are sparsely supported, the estimation rates for marginal effects can be faster than for the function itself. This suggests that simpler models with known basis dictionaries (like polynomials) may provide more reliable marginal effect estimates than complex models, even if their overall predictive accuracy is similar.

Proposed Algorithm: Bagged Polynomial Regression (BPR)

Motivated by the theoretical benefits of partitioning, the authors propose Bagged Polynomial Regression with Random Projections (BPR). BPR combines three concepts:

Random Projections: Models are fit on random subsets of features ( $F$ ) rather than the full set.
Bagging (Ensembling): $M$ base models are trained on random subsamples of the data and averaged to reduce variance.
Regularization: $\ell_2$ penalization (Ridge) is applied to handle collinearity and improve out-of-sample fit.

Algorithm 1 (BPR):

For $m = 1$ $m = 1$ to $M$ $M$ :
- Sample $F$ features randomly.
- Sample $S$ observations (subsample).
- Generate polynomial features of degree $J$ for the selected features.
- Fit a regularized generalized linear model (logistic for classification, linear for regression).
Aggregate predictions: Average the outputs of the $M$ models (or take a majority vote for classification).

The method allows researchers to tune hyperparameters ( $J, F, M, S, \lambda$ ) via cross-validation to navigate the approximation-estimation error trade-off, effectively estimating the optimal partition size $B^*$ for a given dataset.

3. Key Results

Theoretical Contributions

Risk Bounds: The paper provides novel finite-sample and asymptotic $L_2$ risk bounds for partitioned series estimators. It demonstrates that partitioning controls the growth of the dictionary basis, allowing for faster convergence rates in high dimensions compared to standard series regression.
Marginal Effect Rates: Corollary 2.2 establishes that the estimation error for marginal effects scales with the support of the derivative ( $s_j$ ) rather than the total number of parameters ( $k$ ). This implies that models with simpler, sparse marginal structures (like BPR) can offer more precise estimates of marginal effects than complex black-box models.
Optimal Partitioning: The authors characterize the optimal group size $B^*$ , showing it increases with the smoothness of the target function and the polynomial degree, but decreases with the dimensionality of the feature space.

Empirical Application: Crop Classification

The method is applied to a multi-class crop classification problem using satellite imagery (RapidEye optical and UAVSAR radar) near Winnipeg, Canada. The task involves distinguishing seven crop types (corn, peas, canola, soybeans, oats, wheat, broadleaf) from 174 features.

Predictive Performance: BPR achieved a test accuracy of 0.997, outperforming reported neural network baselines for the same dataset (approx. 0.992). This was achieved with a low-degree (degree 2) embedding and a modest ensemble size (10 bags), using significantly fewer parameters than over-parameterized NNs.
Interpretability:
- Coefficient Summaries: The authors demonstrated how to aggregate coefficients across the bagged ensemble to estimate Average Partial Derivatives (APDs). For the Normalized Difference Vegetation Index (NDVI), BPR correctly identified a negative relationship with soybean probability (consistent with agronomic priors: soybeans are less green in early July than competing crops).
- Partial Dependence Plots (PDP): While BPR PDPs showed clear, intuitive relationships between NDVI and crop probability, the NN PDPs were essentially flat. The authors argue that NNs distribute predictive signals across correlated substitutes, obscuring the marginal effect of individual variables, whereas BPR's random feature sub-sampling forces base learners to explicitly model the relationship of included variables.

4. Significance and Claims

The paper claims to offer a "third way" between the high accuracy of black-box NNs and the interpretability of standard econometric models, which often fail in high dimensions.

Econometrics-Native Alternative: BPR provides a transparent, coefficient-based structure that supports interpretability and aligns with scientific priors, addressing emerging governance requirements (e.g., EU AI Act) for algorithmic transparency.
Theoretical Justification for Simplicity: The results suggest that for smooth target functions, simpler models with controlled basis growth can match the predictive performance of complex black-box models while offering superior interpretability and more reliable marginal effect estimates.
Practical Utility: The authors demonstrate that BPR is not just a theoretical construct but a practical tool that matches state-of-the-art performance in environmental prediction tasks while remaining straightforward to diagnose using standard econometric summaries (coefficient tables, PDPs).

The paper concludes that while standard polynomial regression is rarely used due to slow convergence rates, Bagged Polynomial Regression with Random Projections successfully mitigates these issues, making it a viable and interpretable substitute for neural networks in high-dimensional environmental applications.