A constrained iteratively-reweighted least-squares framework for generalised linear models

1. Problem Statement

Generalised Linear Models (GLMs) are ubiquitous in statistical analysis but often lack a comprehensive framework for incorporating linear constraints on coefficients. While constraints are useful for encoding prior knowledge (e.g., monotonicity, non-negativity, or compositional balance) and improving estimator consistency, existing methods face significant barriers:

• Limited Scope: Previous work on constrained estimation and inference is largely restricted to linear models, with sparse extensions to GLMs.
• Computational Barriers: Implementing constraints in GLMs often requires complex, non-standard optimization routines that lack stability or speed.
• Inference Challenges: Standard asymptotic theory (assuming normality) fails under constraints, leading to invalid confidence intervals and p-values.
• Software Gaps: There is a lack of user-friendly software that integrates constrained estimation, inference, and model selection into standard regression workflows.

2. Methodology: The CIRLS Framework

The authors propose a unified framework centered on the Constrained Iteratively-Reweighted Least-Squares (CIRLS) algorithm.

A. The CIRLS Algorithm

• Mechanism: The algorithm adapts the standard IRLS used for GLMs. At each iteration $k$, instead of solving an unconstrained weighted least-squares problem, it solves a Quadratic Program (QP) subject to linear constraints ($\mathbf{l} \le \mathbf{C}\boldsymbol{\beta} \le \mathbf{u}$).
• Optimization: The QP is solved efficiently using dual algorithms (e.g., Goldfarb and Idnani, 1983) or Sequential Quadratic Programming (SQP) techniques.
• Convergence: Theoretically, CIRLS is a simplified SQP. Under standard conditions (twice-differentiable objective, irreducible constraint matrix), it converges quadratically to a local (and often global, due to log-likelihood concavity with canonical links) optimum.

B. Inferential Procedures

Since constrained estimators are biased and their distribution is no longer standard Gaussian, the authors propose:

• Truncated Multivariate Normal (TMVN) Distribution: The distribution of the constrained coefficients $\mathbf{C}\hat{\boldsymbol{\beta}}$ is modeled as a TMVN.
• Simulation-Based Inference: To compute confidence intervals and moments, the framework simulates from the TMVN distribution (using exponential tilting/importance sampling) and transforms the results back to the coefficient space. This accounts for the bias introduced by constraints while providing valid coverage probabilities.

C. Degrees of Freedom and Model Selection

To address model complexity and selection criteria (like AIC or BIC), the authors distinguish between two metrics:

• **Observed Degrees of Freedom ($odf$):**$p - m_a$, where $m_a$ is the number of active constraints in the fitted sample. This is a random variable and can be unstable.
• **Expected Degrees of Freedom ($edf$):**$p - \sum k \cdot P(m_a=k)$. This estimates the expected number of active constraints by simulating from the unconstrained model. The paper argues $edf$ is a superior metric for model selection as it accounts for the randomness of constraint activation, preventing over-penalization of complex models.

D. Software Implementation

The framework is implemented in the R package cirls. It integrates seamlessly with R's standard glm() syntax, allowing users to specify constraints via formula interfaces or constraint matrices. It provides methods for vcov(), confint(), logLik(), and standard model selection functions.

3. Key Contributions

1. General Framework: A unified approach for fitting GLMs with linear constraints across various distributions (Gaussian, Poisson, Binomial, etc.) and link functions.
2. Algorithmic Efficiency: The CIRLS algorithm leverages fast QP solvers, offering rapid convergence comparable to unconstrained IRLS.
3. Valid Inference: A rigorous derivation of the TMVN distribution for constrained coefficients, enabling accurate confidence intervals and hypothesis testing where standard methods fail.
4. Complexity Estimation: The introduction of $edf$ provides a robust method for estimating model complexity in constrained settings, crucial for AIC/BIC calculations.
5. Accessibility: The cirls package lowers the barrier to entry, allowing researchers to apply shape constraints (monotonicity), non-negativity, and compositional constraints using familiar R syntax.

4. Results

Simulation Studies

The authors evaluated the framework using two Data-Generating Mechanisms (DGMs): non-negative regression and non-decreasing strata regression.

• Estimation Performance: Constraints introduce bias but significantly reduce variance. The Root Mean Squared Error (RMSE) of the constrained model is often lower than the unconstrained model, even when the true parameters are slightly outside the feasible region (slightly over-constrained).
• Inference Accuracy: Confidence intervals derived from the TMVN simulation approach achieve near-nominal coverage (95%) when constraints are feasible. Variance estimation is slightly underestimated for feasible models but remains within acceptable error margins (<30%).
• Degrees of Freedom: The $edf$ metric outperforms $odf$ in estimating true model complexity, particularly in scenarios with multiple constraints, as it is more stable and less prone to overfitting penalties.

Real-World Case Studies

1. Global Temperature Anomaly (Monotonic Regression):
   • Application: Estimating global warming trends from 1850–2015 with a non-decreasing constraint.
   • Result: The constrained model produced a smoother, more stable fit during periods of low temperature change compared to the unconstrained model, while accurately capturing rapid warming periods. The $edf$ indicated a reduction in effective complexity (8.6 vs. 12 for the unconstrained spline).
2. GDP Composition and Life Expectancy (Compositional Regression):
   • Application: Modeling the effect of GDP sector proportions on life expectancy in EU countries, enforcing a sum-to-zero constraint on coefficients.
   • Result: The framework successfully estimated balanced effects without requiring a reference category (unlike log-ratio transformations). It identified that higher proportions of "Other" activities (health/education) correlated with higher life expectancy, while transport/communication correlated with lower expectancy.

5. Significance and Limitations

Significance:
The CIRLS framework fills a critical gap in statistical methodology by making constrained GLMs computationally feasible, statistically rigorous, and practically accessible. It allows researchers to incorporate domain-specific knowledge (e.g., physical laws, biological monotonicity) directly into regression models, improving estimation accuracy and interpretability in fields like epidemiology, environmental science, and economics.

Limitations:

• Constraint Dimension: The current inference method requires the number of constraints ($m$) to be less than or equal to the number of coefficients ($p$). If $m > p$, the back-transformation is difficult, though a potential accept-reject sampling workaround is noted.
• High-Dimensional Inference: In scenarios where $p \ge n$ (e.g., Lasso), the unconstrained model required for the TMVN simulation cannot be fitted.
• Boundary Issues: As with many constrained methods, inference can be challenging when true parameters lie exactly on the boundary of the feasible region, though the simulation approach handles this better than asymptotic approximations.
• Non-Linear Constraints: The current framework is limited to linear constraints; non-linear constraints (e.g., Ridge-type smoothing) would require more general SQP extensions.

Conclusion:
The paper successfully demonstrates that constrained GLMs can be fitted efficiently and with valid inference. The cirls package democratizes these advanced techniques, enabling a unified approach to constrained estimation across diverse research domains.