Robust Regression with Student's T: The Role of Degrees of Freedom

1. Problem Statement

Linear regression estimators, particularly Ordinary Least Squares (OLS), are highly sensitive to outliers, which can distort parameter estimates and degrade inference reliability. While modeling errors using the Student's $t$-distribution offers a robust alternative due to its heavy tails, a critical practical challenge remains: how to choose the degrees of freedom parameter ($\nu$).

• If $\nu$ is fixed too low, the model may be inefficient if the true error distribution is closer to normal.
• If $\nu$ is fixed too high, the model loses robustness against outliers.
• Existing literature offers conflicting advice: some suggest fixing $\nu$ for small sample sizes, while others propose estimating it. There is a lack of comprehensive comparison between frequentist and Bayesian methods for estimating $\nu$ and understanding how these estimates propagate to the regression coefficients ($\beta$), especially in high-dimensional settings.

2. Methodology

The authors propose and compare several approaches to estimate $\nu$ and subsequently estimate the regression coefficients $\beta$ and scale $\sigma$.

A. Estimation of Degrees of Freedom ($\nu$)

The paper evaluates four primary methods for estimating $\nu$:

1. Profile Maximum Likelihood Estimator ($\hat{\nu}$): Maximizes the profile log-likelihood function $\ell_p(\nu) = \ell(\hat{\beta}_\nu, \hat{\sigma}_\nu, \nu)$.
2. Adjusted Profile Likelihood Estimator ($\hat{\nu}_{adj}$): Maximizes an adjusted profile log-likelihood function proposed by Cox and Reid (1987) to correct for the bias introduced by estimating nuisance parameters ($\beta, \sigma$):
   $$\ell_{adj}(\nu; y) = \ell_p(\nu; y) - \frac{1}{2} \log |j_{\lambda\lambda}(\nu, \hat{\lambda}_\nu)|$$
   where $j_{\lambda\lambda}$ is the submatrix of the observed Fisher information corresponding to nuisance parameters.
3. Bayesian MAP (Jeffreys' Independence Prior): Uses Jeffreys' independence prior where $\pi(\beta) \propto 1$ and $\pi(\sigma, \nu)$ is derived from the determinant of the relevant submatrix of the Fisher information matrix. The estimate is the Maximum A Posteriori (MAP) value.
4. Pseudo-Bayesian MAP: Treats the profile likelihood as a true likelihood and applies a prior $\pi(\nu) \propto I_{\nu\nu}^{1/2}(\nu)$ (based on the diagonal of the Fisher information) to form a pseudo-posterior.

B. Two-Stage Inference Procedure

The core methodology involves a two-stage procedure:

1. Stage 1: Estimate $\nu$ using one of the methods above.
2. Stage 2: Fix $\nu$ at the estimated value and maximize the likelihood over $\beta$ and $\sigma$ to obtain the regression coefficients.

C. Simulation Framework

The authors conducted extensive simulations on:

• Synthetic Data: Varying sample sizes ($n$), dimensions ($p$), and true $\nu$ values (2, 5, 10).
• Real Data: The "stack loss" dataset (Brownlee, 1965) expanded to high dimensions.
• Contamination: Data contaminated with various outlier distributions (Normal with high variance, $\chi^2$, $t$-distribution, and two-point contamination) at rates of 10%, 20%, and 30%.
• Comparisons: The proposed methods were compared against OLS, Huber robust regression (with data-driven tuning), and fixed-$\nu$ approaches.

3. Key Contributions

1. Systematic Comparison of $\nu$ Estimators: The paper provides a rigorous comparison of frequentist (profile vs. adjusted profile) and Bayesian (Jeffreys vs. pseudo-Bayesian) methods for estimating $\nu$.
2. Identification of the Adjusted Profile Likelihood as Superior: The study demonstrates that the Adjusted Profile Likelihood ($\hat{\nu}_{adj}$) generally yields the most accurate estimates of $\nu$ and $\beta$, particularly in high-dimensional settings ($p$ is large relative to $n$).
3. Insight on High-Dimensionality: The authors reveal a critical interaction between dimensionality and $\nu$ estimation. In high dimensions ($p/n > 0.5$), the model tends to overestimate $\nu$ (shrinking residuals artificially), making fixed-$\nu$ approaches with larger values sometimes more efficient than estimated ones.
4. Robustness Validation: The proposed method is shown to be robust across various error distributions (heavy-tailed and light-tailed) and contamination types, often outperforming Huber regression when the $p/n$ ratio is moderate.
5. Software Implementation: The authors released an R package, RobustTRegression, to facilitate the implementation of these methods.

4. Key Results

• Performance of $\nu$ Estimators:

  • In low-dimensional settings ($p \ll n$), all estimators perform similarly.
  • In high-dimensional settings ($p \approx n$ or $p > n/2$), the Adjusted Profile Likelihood ($\hat{\nu}_{adj}$) significantly outperforms the standard profile likelihood and Bayesian methods in terms of Root Mean Square Error (RMSE).
  • Standard profile likelihood tends to fail or produce unstable estimates when $p$ is large relative to $n$.

• Regression Coefficient ($\beta$) Accuracy:

  • Low/Moderate Dimensions: Estimating $\nu$ via $\hat{\nu}_{adj}$ produces $\beta$ estimates with accuracy comparable to fixing $\nu$ at its true value. It consistently outperforms OLS and often outperforms Huber regression, especially under heavy-tailed errors.
  • High Dimensions: When $p/n$ is large (e.g., $>0.5$), the $\hat{\nu}_{adj}$ approach still performs well but may yield slightly higher RMSE than the optimal fixed-$\nu$ approach (which is often a larger $\nu$). This is because high dimensionality allows the model to absorb outliers into the coefficients, reducing the apparent need for heavy tails.
  • Contamination: Under various contamination scenarios (Gaussian, $\chi^2$, $t$, two-point), $\hat{\nu}_{adj}$ maintains robust performance, often achieving lower RMSE than Huber regression, particularly at lower contamination rates (10-20%).

• Sensitivity to $\nu$ Misspecification:

  • The method is not highly sensitive to small misspecifications of $\nu$ around the true value. However, as $\nu$ moves far from the truth (e.g., fixing $\nu=20$ when true $\nu=2$), performance degrades significantly.

5. Significance and Conclusion

The paper concludes that the proper calibration of the degrees of freedom is as crucial as the choice of the robust distribution itself.

• Recommendation: For practical applications where $p/n$ is not excessively large, the authors strongly recommend using the Adjusted Profile Likelihood ($\hat{\nu}_{adj}$) to estimate $\nu$. This approach offers a "sweet spot" of high accuracy and robustness, performing comparably to the ideal scenario where the true $\nu$ is known.
• Caveat: In very high-dimensional settings where $p/n$ is large, the estimation of $\nu$ becomes unstable. In such cases, the authors suggest that fixing $\nu$ at a large value (or using a different strategy) might be more efficient, though this threshold requires further research.
• Impact: This work synthesizes the theory of $t$-based robust regression, providing a clear, data-driven pathway for practitioners to handle outliers without arbitrarily fixing distributional parameters, thereby improving the reliability of statistical inference in modern, high-dimensional data environments.