📊 statistics

A New Fit Assessment Framework for Common Factor Models Using Generalized Residuals

This paper proposes an extended framework using generalized residuals to develop flexible fit test statistics for common factor models, demonstrating through simulations and empirical analysis that this approach effectively detects misfits in distributional and functional assumptions that conventional mean-and-covariance-based methods often overlook.

Original authors: Youjin Sung, Youngjin Han, Yang Liu

Published 2026-03-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Youjin Sung, Youngjin Han, Yang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about how people think. You can't see thoughts directly, so you use a "lie detector" test (a survey or a set of questions) to infer what's going on inside their heads. In statistics, this is called a Common Factor Model. You assume that the answers people give (the visible clues) are caused by hidden traits (the invisible suspects).

For decades, detectives have used a standard checklist to see if their theory fits the clues. They check the average answers and how the answers vary together. If the averages and variations look right, they say, "Case closed! The model fits."

The Problem:
The authors of this paper argue that this standard checklist is like checking a car engine only by listening to the hum. It tells you if the engine is running, but it doesn't tell you if the pistons are misfiring or if the oil is the wrong color. You might have a car that sounds fine but is actually broken in ways the standard test can't see.

In statistical terms, the old methods only check the "mean and covariance" (the average and the spread). They miss subtle, critical errors, like:

Are the hidden traits actually distributed normally, or are there two distinct groups of people mixed together?
Is the relationship between the hidden trait and the answer actually a straight line, or is it curved?
Is the "noise" (error) in the answers the same for everyone, or does it get wilder for some people?

The Solution: Generalized Residuals
The authors propose a new, super-powered magnifying glass called Generalized Residuals.

Think of it this way:

The Old Way: You take the whole pile of survey answers, calculate the average, and compare it to your prediction. If they match, you're happy.
The New Way (Generalized Residuals): Instead of looking at the whole pile, you slice the data into tiny, specific groups based on the hidden trait. You ask: "For people with exactly this level of hidden trait, does their answer match the prediction?"

If you do this for every possible level of the hidden trait, you get a detailed map of where the model is failing.

The Three Main Checks (The "Detective's Toolkit")

The paper introduces three specific ways to use this new magnifying glass:

The "Shape" Check (Latent Variable Density):
- The Metaphor: Imagine you expect a crowd of people to be a smooth, bell-shaped curve (like a normal distribution). But what if the crowd actually has two distinct humps (like a bimodal distribution)?
- The Old Way: Might miss this if the average height is the same.
- The New Way: It looks at the shape of the crowd and screams, "Hey! There are two groups here, not one smooth group!"
The "Line" Check (Mean Function):
- The Metaphor: You assume that as a person's hidden trait increases, their answer goes up in a straight line (like walking up a ramp).
- The Old Way: Might say "Looks straight enough" because the average slope is correct.
- The New Way: It zooms in and sees, "Wait, at the very top and bottom, the line is curving like a banana!" It catches non-linear relationships that the old method smooths over.
The "Consistency" Check (Variance/Homoscedasticity):
- The Metaphor: You assume everyone makes mistakes with the same amount of randomness (like throwing darts with the same wobble).
- The Old Way: Checks the average wobble.
- The New Way: It notices that while the average wobble is fine, the people with high hidden traits are throwing darts all over the place (huge wobble), while the others are very steady. It catches "uneven" error.

Why Does This Matter? (The Real-World Example)
The authors tested this on Response Time data (how long it takes students to answer questions).

The Old Check: Said, "Great fit! The model works perfectly."
The New Check: Said, "Hold on! The model is lying to you.
- The students aren't a single smooth group; there are distinct types of responders.
- For some questions, the relationship isn't a straight line; it curves.
- For slow responders, the answers are way more chaotic than the model predicts."

The Takeaway
This paper is a call to stop settling for "good enough" when checking if a statistical model works. Just because the averages look right doesn't mean the whole picture is correct.

By using Generalized Residuals, researchers can now see the hidden cracks in their models. It's like upgrading from a black-and-white photo to a high-definition, 3D scan. It helps them find the specific places where their theories are wrong, leading to better science, fairer tests, and more accurate conclusions about human behavior.

1. Problem Statement

Conventional Goodness-of-Fit (GOF) assessments for common factor models (CFMs) rely primarily on comparing observed and model-implied mean vectors and covariance matrices. While standard indices (e.g., $\chi^2$ , CFI, TLI, RMSEA, SRMR) are effective at detecting misfit in these first- and second-order moments, they often fail to detect critical aspects of model-data disagreement, such as:

Non-normality of Latent Variables (LVs): Conventional methods may indicate good fit even if the LV distribution is bimodal or skewed, leading to misleading interpretations of correlations (e.g., interpreting a linear correlation coefficient when the relationship is non-linear or subgroup-specific).
Misspecification of Manifest Variable (MV) Functions: Standard diagnostics often miss violations in the functional form of MVs, such as non-linear relationships between LVs and MVs (violating linearity) or heteroscedasticity (violating constant variance assumptions).
Masked Misfit: In many cases, conventional GOF indices suggest a "good fit" while significant structural misfit exists, potentially invalidating subsequent inferences about latent traits or their relationships.

2. Methodology

The authors extend the theory of Generalized Residuals (originally developed by Haberman & Sinharay, 2013, for categorical IRT data) to continuous common factor models.

A. Theoretical Framework

Generalized Residuals Definition: The framework defines a summary quantity $H(Y_i; \xi)$ for each individual $i$ , where $Y_i$ represents manifest variables and $\xi$ represents model parameters. The residual is the difference between the sample average of $H$ and its model-implied population expectation:
$e = \hat{\eta}(\hat{\xi}) - \eta(\hat{\xi})$
where $\hat{\eta}$ is the sample mean and $\eta$ is the population expectation.
Transformation: To increase flexibility, a transformation function $\phi$ is applied to the residuals, allowing for the construction of ratios (e.g., conditional means) and other non-linear combinations.
Asymptotic Properties: The authors establish that the transformed residuals are asymptotically multivariate normal under the null hypothesis of a correctly specified model. They derive the asymptotic covariance matrix (ACM) and propose a Monte Carlo (MC) estimation method to calculate the ACM components ( $A(\xi)$ and $\Sigma_H(\xi)$ ) to ensure stability, particularly with small sample sizes or extreme LV values.

B. Test Statistics

Two types of statistics are proposed based on the asymptotic normality:

Pointwise $z$ -statistic: Tests a single residual at a specific LV value.
$z = \frac{e_{\phi, r}}{\hat{\sigma}_{\phi, r} / \sqrt{n}}$
This allows for localized diagnostics (e.g., checking fit at specific points on the latent trait continuum).
Summary $\chi^2$ -statistic: Tests multiple residuals simultaneously to provide an overall assessment.
$T = n e_{\phi}^\top \hat{\Sigma}^{-}_{\phi} e_{\phi}$
The authors propose a specific weight matrix $\hat{\Sigma}^{-}_{\phi}$ (a generalized inverse) to ensure the statistic follows a $\chi^2$ distribution with degrees of freedom $s$ (often set to 1 to avoid numerical instability).

C. Specific Examples of Application

The paper illustrates the framework with three specific test cases:

LV Density Normality: Tests if the latent variable distribution is normal by comparing the empirical posterior density of LVs against the model-implied normal density.
MV-Level Linearity: Tests if the conditional mean of an MV given the LV is linear. This involves constructing residuals based on the ratio of the expected product of the MV and LV density to the LV density.
MV-Level Homoscedasticity: Tests if the conditional variance of an MV is constant across LV values.

3. Key Contributions

Extension of Theory: Successfully extends generalized residuals from categorical data (IRT) to continuous linear factor models, introducing transformed residuals and establishing their joint multivariate normality.
Novel Estimation Strategy: Proposes a Monte Carlo approach for estimating the asymptotic covariance matrix of residuals, which prevents the Type I error inflation seen in previous methods when using sample covariances directly.
Diagnostic Granularity: Moves beyond global fit indices to provide localized diagnostics. Researchers can now visualize and test fit at specific points of the latent continuum, identifying where and why a model fails (e.g., non-linearity only at extreme trait levels).
Robustness to "Good" Global Fit: Demonstrates that models can pass conventional GOF tests (CFI, RMSEA) while failing critical parametric assumptions (normality, linearity, homoscedasticity).

4. Results

The authors conducted two simulation studies and one empirical analysis:

Simulation Study 1 (LV Density)

Setup: Compared a correctly specified bivariate normal LV distribution against a misspecified bimodal mixture distribution.
Findings:
- Type I Error: Well-controlled across sample sizes ( $n=100, 500, 1000$ ).
- Power: The proposed tests ( $z_1, T_1$ ) successfully detected the bimodal misfit, with power increasing with sample size.
- Conventional GOF: Standard indices (CFI, TLI, RMSEA) and the $\chi^2$ test failed to detect the misfit, often indicating "good fit" even when the LV distribution was severely misspecified.

Simulation Study 2 (MV-Level Linearity & Homoscedasticity)

Setup: Tested models with misspecified mean functions (quadratic instead of linear) and variance functions (log-linear instead of constant).
Findings:
- Specificity: The linearity test ( $z_2, T_2$ ) detected mean misspecification but not variance misspecification, and vice versa for the homoscedasticity test ( $z_3, T_3$ ).
- False Detection: The tests maintained low false detection rates for correctly specified items even in the presence of misspecified items.
- Conventional GOF: Again, standard diagnostics failed to detect these specific functional form violations.

Empirical Example (Response Time Data)

Data: Log-transformed response times from the Test of Relational Reasoning-Junior (TORRjr).
Conventional Results: Standard GOF indices (CFI=0.97, RMSEA=0.05) suggested a good fit.
Proposed Method Results:
- LV Density: Detected significant non-normality (leptokurtic and negatively skewed), indicating more fast responders than the normal model assumed.
- Linearity: Identified specific items (2, 6, 7) where the linear mean function failed, particularly at extreme latent slowness values.
- Homoscedasticity: Detected heteroscedasticity in several items, showing that variance increased for slower responders.

5. Significance and Implications

Deeper Diagnostic Insight: The framework provides a "microscope" for model fit, allowing researchers to move from "Does the model fit?" to "Where does the model fit, and what assumptions are violated?"
Model Modification Guidance: The visualizations (plotting empirical vs. model-implied conditional moments) offer direct guidance for model improvement. For instance, detected non-linearity suggests fitting a nonlinear factor model; detected bimodality suggests mixture modeling.
Complement to Standard Practice: The authors argue that generalized residuals should not replace conventional GOF but complement them. Relying solely on mean/covariance recovery can lead to false confidence in models that are structurally flawed.
Future Directions: The paper suggests extending this framework to non-linear factor analysis, count data IRT, and developing effect size measures to address the sensitivity of $\chi^2$ tests to large sample sizes. It also highlights the need for efficient computational strategies for high-dimensional latent spaces.

In conclusion, this paper presents a robust, flexible, and statistically rigorous framework for diagnosing common factor models that uncovers misfit patterns invisible to traditional methods, thereby enhancing the validity of latent variable inferences in psychology and education.