Why Machine Learning Models Systematically Underestimate Extreme Values II: How to Fix It with LatentNN

1. Problem Statement: Attenuation Bias in Neural Networks

The paper addresses a fundamental statistical issue known as attenuation bias (or regression dilution) in the context of astronomical data analysis using neural networks.

• The Phenomenon: When input variables ($x$) contain measurement errors ($\sigma_x$), standard regression models (both linear and non-linear) systematically underestimate the slope of the relationship between inputs and outputs. This results in a "compression" of the predicted dynamic range: high values are predicted too low, and low values are predicted too high.
• The Cause: Standard machine learning treats observed inputs ($x_{obs}$) as ground truth. Measurement errors artificially inflate the observed variance of the input ($Var(x_{obs}) = Var(x_{true}) + \sigma_x^2$) without affecting the output variance. Consequently, the model fits a shallower slope to minimize error, pulling predictions toward the mean.
• Why it matters in Astronomy: Unlike computer vision or NLP where input noise is negligible, astronomical data often operates at low Signal-to-Noise Ratios (SNR $\sim$ 10). In this regime, measurement errors can be comparable to the intrinsic data range.
• The Limitation of Standard ML: The paper demonstrates that increasing model complexity (e.g., using deep Multi-Layer Perceptrons instead of linear regression) or applying standard regularization (L2/weight decay) does not mitigate this bias. The bias is a fundamental consequence of treating noisy inputs as exact, independent of sample size or model architecture.

2. Methodology: LatentNN

The author proposes LatentNN, a method that generalizes the classical statistical solution for errors-in-variables (Deming regression) to neural networks.

• Core Concept: Instead of treating observed inputs $x_{obs}$ as fixed, the method treats the true underlying inputs $x_{true}$ as latent variables to be estimated alongside the neural network parameters ($\theta$).
• Hierarchical Structure:
  1. Top Level: Neural network parameters $\theta$.
  2. Intermediate Level: Latent true inputs $x_{latent}$ (one set per training sample).
  3. Bottom Level: Observed data $(x_{obs}, y_{obs})$.
• Objective Function: The method maximizes the joint likelihood of observing both inputs and outputs given the latent variables. This translates to minimizing a composite loss function:
  $$J(\theta, \{x_{latent}\}) = \sum_{i=1}^N \frac{(y_{obs,i} - f_\theta(x_{latent,i}))^2}{\sigma_{y,i}^2} + \sum_{i=1}^N \frac{(x_{obs,i} - x_{latent,i})^2}{\sigma_{x,i}^2}$$
  • Term 1 (Prediction Loss): Penalizes the difference between the model's prediction (based on latent inputs) and the observed output.
  • Term 2 (Latent Likelihood): Penalizes the deviation of the latent inputs from the observed inputs, weighted by the known measurement uncertainty ($\sigma_x$).
• Optimization: The network parameters $\theta$ and the latent variables $x_{latent}$ are optimized jointly using gradient descent (e.g., Adam). The latent values are initialized at the observed values ($x_{latent}^{(0)} = x_{obs}$) and iteratively "denoised" toward the true values during training.
• Regularization: To prevent overfitting to specific latent configurations, L2 regularization (weight decay) is applied only to the network weights $\theta$, not the latent variables.

3. Key Contributions

1. Empirical Demonstration: The paper provides rigorous numerical evidence that standard Neural Networks (MLPs) suffer from the same attenuation bias as linear regression, regardless of depth or width.
2. Generalization of Deming Regression: It successfully extends the classical Deming regression framework (typically limited to linear models) to arbitrary differentiable neural network architectures.
3. LatentNN Implementation: A practical, scalable implementation using modern automatic differentiation frameworks (PyTorch/JAX) that treats inputs as learnable parameters.
4. Analysis of Dimensionality: The study reveals a non-monotonic relationship between input dimensionality and correction difficulty. While high-dimensional correlated features allow the model to "lean" on redundancy for robust correction, intermediate dimensions (e.g., 3 features) present a complex posterior landscape that is harder to optimize than both 1D and high-D cases.

4. Experimental Results

The method was validated on synthetic data and a realistic stellar spectroscopy application:

• One-Dimensional Regression:
  • Standard MLPs showed severe bias (e.g., at SNR=1, the attenuation factor $\lambda_y \approx 0.5$).
  • LatentNN recovered $\lambda_y \approx 1.0$ across all SNR regimes, effectively removing the bias.
• Multivariate Inputs (Correlated Features):
  • Standard MLPs followed the theoretical attenuation curve regardless of the number of features ($p$).
  • LatentNN maintained $\lambda_y \approx 1$ for $p \ge 10$ at SNR $\ge 2$.
  • For low dimensions ($p=3$), LatentNN still significantly outperformed standard MLPs (e.g., $\lambda_y \approx 0.7$ vs $0.2$ at SNR=1), though optimization became more sensitive.
• Stellar Spectroscopy Application:
  • Applied to inferring metallicity ([M/H]) from synthetic stellar spectra (The Payne emulator).
  • Scenario: Inferring metallicity from 3, 10, or 30 spectral pixels with varying noise levels.
  • Result: Standard MLPs compressed the metallicity distribution, systematically predicting metal-poor stars as more metal-rich (e.g., a true [M/H] = -2.0 star predicted as -1.5). LatentNN corrected this, maintaining $\lambda_y \gtrsim 0.95$ for 10+ pixels even at high noise levels.
  • Implication: This correction is critical for accurately identifying rare, metal-poor stars and constraining the formation history of the Milky Way.

5. Significance and Limitations

Significance:

• Scientific Accuracy: Provides a framework for unbiased inference in low-SNR regimes, which is characteristic of astronomical data (spectroscopy, photometry, time-series).
• Correcting Extreme Values: Specifically addresses the underestimation of extreme values, which are often the most scientifically interesting (e.g., the metal-poor tail of the stellar population).
• Scalability: Unlike full hierarchical Bayesian inference (which is computationally prohibitive for large datasets), LatentNN offers a scalable, optimization-based alternative (Maximum A Posteriori limit) that fits within standard GPU memory constraints for typical astronomical survey sizes ($N \sim 10^5$).

Limitations:

• Requirement of Uncertainties: The method requires accurate estimates of input ($\sigma_x$) and output ($\sigma_y$) uncertainties. Misspecification can lead to under- or over-correction.
• Computational Cost: The number of parameters scales with the dataset size ($N \times p$), as a latent variable is added for every training sample. While feasible for current survey sizes, it may challenge very large datasets ($N > 10^6$) with high-dimensional inputs.
• Optimization Complexity: The loss landscape is more complex than standard training. At very low SNR or intermediate dimensions, the problem can be ill-conditioned, requiring careful hyperparameter tuning (specifically weight decay) and multiple random restarts.
• Point Estimates: LatentNN provides point estimates (MAP) rather than full posterior distributions over the latent variables, though the network parameters themselves are well-constrained.

Conclusion

The paper establishes that attenuation bias is a pervasive issue in astronomical machine learning that cannot be solved by deeper networks or more data. LatentNN offers a principled, effective, and computationally feasible solution by jointly optimizing for the true underlying data and the model parameters, thereby restoring the integrity of extreme value predictions in noisy observational data.