📊 statistics

Maximum softly penalised likelihood in factor analysis

This paper introduces a maximum softly-penalised likelihood framework for exploratory factor analysis that guarantees interior parameter estimates to resolve Heywood cases while preserving the consistency and asymptotic normality of maximum likelihood estimators through carefully scaled penalties.

Original authors: Philipp Sterzinger, Ioannis Kosmids, Irini Moustaki

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Sterzinger, Ioannis Kosmids, Irini Moustaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using a set of clues. In the world of statistics, this "mystery" is understanding hidden patterns in data (like what makes people anxious, or what drives their spending habits). The tool you use to solve this is called Factor Analysis. It tries to find invisible "factors" (the hidden causes) that explain the visible data (the clues).

However, sometimes the math breaks down. The paper you're asking about tackles a specific problem called a Heywood case.

The Problem: The "Impossible" Clues

Imagine you are trying to figure out how much of a person's weight is due to genetics versus diet. Your math might tell you that genetics accounts for 100% of the weight, and diet accounts for 0%. That's a "zero variance" case.

But sometimes, the math goes even weirder. It might tell you that diet accounts for -20% of the weight. In the real world, you can't have negative weight or negative error. This is a Heywood case. It's like the calculator screaming, "I don't know what I'm doing!"

When this happens:

The computer gets stuck and can't finish the calculation.
Even if it forces an answer, the results are nonsense (like saying a factor explains more than 100% of the data).
Any conclusions you draw from this are unreliable.

The Old Solutions: The "Brute Force" and the "Magic Wand"

For decades, statisticians tried to fix this in two ways:

Brute Force: Just tell the computer, "If you get a negative number, change it to zero." The problem? This breaks the mathematical rules, making the results shaky and hard to trust.
The Magic Wand (Bayesian Priors): Some researchers added a "magic rule" (a penalty) to the math to gently push the numbers away from zero. This worked, but it was like using a sledgehammer to crack a nut. It often introduced bias, meaning it systematically pulled the answers in the wrong direction, making them inaccurate even when they weren't "broken."

The New Solution: The "Softly Penalised" Approach

The authors of this paper (Sterzinger, Kosmidis, and Moustaki) propose a new method called Maximum Softly Penalised Likelihood (MSPL).

Here is the analogy:
Imagine you are walking a tightrope (the edge of the parameter space).

The Old Way: You put a giant wall at the edge. If you hit it, you bounce back, but you might get hurt (biased results).
The New Way (MSPL): You put a soft, invisible cushion just before the edge. As you get closer to the danger zone (zero or negative variance), the cushion gets softer and softer, gently nudging you back to safety.

Why is this "Soft" approach special?

It guarantees safety: The math proves that with this cushion, you will never fall off the tightrope. You will never get a negative variance again.
It's gentle: The cushion is designed to be "soft" enough that as you get more data (more clues), the cushion disappears. It doesn't push you off course; it just stops you from falling.
It keeps the truth: Because the cushion fades away as you get more data, the final answer is just as accurate as if you had never needed the cushion at all. It preserves the "gold standard" of statistical accuracy.

The "Goldilocks" Scaling

The paper's biggest breakthrough is figuring out exactly how soft the cushion should be.

If the cushion is too hard (like the old methods), it distorts the answer.
If the cushion is too soft (or non-existent), you fall off the cliff (Heywood cases).
The authors found the "Goldilocks" zone: A specific mathematical scaling (related to the square root of the sample size) that is just right. It's strong enough to prevent the crash, but weak enough to let the true answer shine through.

Real-World Impact

The authors tested this on fake data and real historical datasets (like old psychological tests).

Before: The standard method often crashed or gave weird results (like negative variances).
After: The new method worked smoothly every time. It didn't just fix the crashes; it actually gave more accurate answers than the old "brute force" methods, especially when the data was messy or the sample size was small.

The Takeaway

This paper is like inventing a new kind of shock absorber for statistical cars.
Previously, if you hit a bump (a Heywood case), the car would either crash or the suspension would be so stiff it broke the passengers' backs (bias).
Now, with MSPL, the car has a smart suspension that absorbs the shock perfectly, ensuring a smooth ride to the correct destination, no matter how bumpy the road gets.

In short: They found a way to stop factor analysis from breaking down, without making the results less accurate. It's a "best of both worlds" solution for statisticians.

1. Problem Statement

Exploratory Factor Analysis (EFA) frequently encounters Heywood cases, where the Maximum Likelihood (ML) estimation procedure yields improper solutions. These occur when:

Error variance estimates ( $\psi_j$ ) are zero or negative.
Correlation estimates exceed 1 in absolute value.

Consequences:

Numerical Instability: Optimization algorithms often fail to converge or produce misleading inferences.
Statistical Invalidity: Standard errors, factor scores, and model selection criteria (AIC/BIC) become unreliable.
Current Solutions & Limitations:
- Constrained ML: Forcing variances to be non-negative violates regularity conditions, leading to difficult-to-evaluate asymptotic properties.
- Bayesian/Penalized Approaches: Previous methods (e.g., Akaike 1987; Hirose et al. 2011) use penalties to prevent zero variances. However, these "vanilla" penalties often lack formal proofs of existence guarantees and, more critically, introduce severe finite-sample bias in parameter estimates and factor loadings due to excessive penalization strength that does not decay with sample size.

2. Methodology: Maximum Softly Penalised Likelihood (MSPL)

The authors propose a framework called Maximum Softly Penalised Likelihood (MSPL). This approach modifies the log-likelihood function by adding a penalty term that ensures estimates remain in the interior of the parameter space while preserving the optimal asymptotic properties of the ML estimator.

The Framework:
The objective function is defined as:
$\ell^*(\theta; S) = \ell(\theta; S) + P^*(\theta)$
Where:

$\ell(\theta; S)$ is the standard profile log-likelihood for EFA.
$P^*(\theta)$ is a penalty function designed to discourage estimates of error variances ( $\psi_j$ ) from approaching zero.

Key Technical Components:

Penalty Selection: The paper re-evaluates existing penalties proposed by Akaike (1987) and Hirose et al. (2011):
- Akaike: $P^*(\theta) = -\frac{\rho n}{2} \text{tr}(\Psi^{-1/2}\Lambda\Lambda^\top\Psi^{-1/2})$
- Hirose: $P^*(\theta) = -\frac{\rho n}{2} \text{tr}(\Psi^{-1/2}S\Psi^{-1/2})$
- Crucial Innovation: The authors demonstrate that the standard choice of $\rho$ (constant or scaling with $n$ ) causes bias. They propose a "soft" scaling where the penalty strength decays with the sample size: $\rho = 2\sqrt{2/n^3}$ (effectively making the penalty term $O_p(n^{-1/2})$ ).
Soft Penalisation: The term "softly penalised" implies that the penalty is strong enough to prevent boundary solutions (Heywood cases) in finite samples but weak enough to vanish asymptotically, ensuring the estimator behaves like the ML estimator as $n \to \infty$ .

3. Key Contributions & Theoretical Results

A. Existence of Estimates (Theorem 4.1)

The authors derive sufficient conditions for a penalty function $P^*$ to guarantee that the Maximum Penalised Likelihood (MPL) estimator exists in the interior of the parameter space (i.e., $\psi_j > 0$ for all $j$ ).

Conditions: The penalty must be continuous, bounded from above, and diverge to $-\infty$ as any variance approaches zero (provided the covariance matrix remains positive definite).
Result: They formally prove that both the Akaike and Hirose penalties satisfy these conditions, providing the first rigorous proof that these specific penalties eliminate Heywood cases.

B. Asymptotic Properties (Theorems 5.1 & 5.2)

The paper establishes that MSPL estimators possess the same desirable asymptotic properties as standard ML estimators:

Consistency: The estimators converge to the true parameters (up to orthogonal rotation) as $n \to \infty$ .
$\sqrt{n}$ -Consistency & Asymptotic Normality: If the penalty scales as $O_p(n^{-1/2})$ (the "soft" scaling), the MSPL estimator is asymptotically equivalent to the ML estimator. Specifically:
$\sqrt{n}(\tilde{\theta} - \hat{\theta}) \xrightarrow{p} 0$
Where $\tilde{\theta}$ is the MSPL estimator and $\hat{\theta}$ is the ML estimator.
Significance: This ensures that hypothesis testing and confidence intervals derived from MSPL are valid, unlike constrained ML or heavily penalized Bayesian approaches which may have distorted asymptotic distributions.

C. Equivariance

The framework preserves two critical equivariance properties of the ML estimator:

Rescaling: Invariance to linear scaling of the observed variables.
Rotation: Invariance to orthogonal rotations of the factor loadings.

4. Empirical Results

Simulation Studies

The authors conducted extensive simulations comparing:

Standard ML (prone to Heywood cases).
Vanilla MPL (Akaike/Hirose with constant $\rho$ ).
MSPL (Akaike/Hirose with soft scaling $\rho \propto n^{-1/2}$ ).

Findings:

Heywood Cases: ML produced a high percentage of Heywood cases (up to 75% in some settings). Both MPL and MSPL reduced this to near zero.
Bias & RMSE: Vanilla MPL exhibited large finite-sample bias and Root Mean Squared Error (RMSE) due to excessive penalization. MSPL showed the smallest bias and RMSE, closely matching the performance of ML in non-problematic scenarios.
Underestimation: Vanilla penalties consistently underestimated factor loadings and communalities. MSPL estimates were well-calibrated (probability of underestimation $\approx 0.5$ ).
Model Selection: AIC and BIC based on MSPL estimates correctly identified the number of factors more frequently than ML (which fails when Heywood cases occur) and significantly outperformed vanilla MPL (which selected incorrect models due to bias).

Real Data Applications

The method was applied to three classic datasets (Davis, Emmett, Maxwell) known to produce Heywood cases with ML.

Results: MSPL successfully resolved the improper solutions (negative variances) without distorting the estimates. The resulting communalities and factor loadings were reasonable, and AIC/BIC values remained stable compared to ML, confirming that the penalty did not artificially alter the model fit.

5. Significance and Conclusion

This paper provides a comprehensive solution to the long-standing problem of Heywood cases in factor analysis.

Theoretical Breakthrough: It bridges the gap between constrained estimation (which fixes the boundary issue but breaks asymptotics) and penalized estimation (which fixes the boundary but introduces bias).
Practical Utility: The MSPL framework allows practitioners to use standard optimization routines to obtain valid, interior estimates that are free from Heywood cases while retaining the optimal statistical properties of Maximum Likelihood.
Recommendation: The authors advocate for the use of softly penalized likelihood with the specific scaling derived in the paper ( $\rho = 2\sqrt{2/n^3}$ ) as a robust alternative to both unconstrained ML and naive Bayesian/penalized approaches.

In summary, the paper transforms a heuristic fix (penalization) into a rigorous statistical framework, ensuring that factor analysis models are both numerically stable and statistically sound.