📊 statistics

From Structural Equation Modelling to Double Machine Learning: Robustness Analysis for Survey-Based Research

This paper proposes and demonstrates a staged robustness analysis framework that integrates Structural Equation Modelling, Ordinary Least Squares regression, and Double Machine Learning to validate the stability of survey-based structural relationships across different estimation methods, using a FinTech Digital Customer Intimacy model as a case study.

Original authors: Ka Ching Chan, Qiana Liu, Sanjib Tiwari, Ranga Chimhundu

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ka Ching Chan, Qiana Liu, Sanjib Tiwari, Ranga Chimhundu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about how people feel about a new financial app (FinTech). You have a bunch of clues (survey answers) about things like "trust," "satisfaction," and "how often they use the app."

For a long time, researchers have used a specific tool called Structural Equation Modelling (SEM) to connect these clues. Think of SEM as a high-tech, custom-built map. It's great because it can handle "invisible" concepts (like "trust," which you can't measure with a ruler) by combining many survey questions into one big score. It draws a map showing how one thing leads to another (e.g., "Trust leads to Satisfaction").

The Problem:
The paper argues that relying only on this custom map is risky. What if the map looks good only because of how you drew it? What if the connections disappear if you look at the data a slightly different way? The authors say, "Don't just trust the map; let's test the terrain with different vehicles to see if the roads are actually there."

The Solution: A Three-Stage Robustness Check
The authors propose a new workflow that acts like a three-stage vehicle test to see if the roads (relationships) on your map are real or just an illusion.

Stage 1: The Custom Map (SEM)

What it is: The standard method. You build your theory-based map.
The Analogy: This is like drawing a blueprint of a house based on your architectural theory. You know where the walls should be.
The Paper's Rule: Don't throw away any walls just because the blueprint looks shaky yet. Keep the full blueprint for the next steps.

Stage 2: The Transparent Walk (OLS Regression)

What it is: A simpler, standard statistical method.
The Analogy: This is like walking through the house with a flashlight. You aren't using the fancy blueprint anymore; you are just looking at the raw materials (the survey scores) and asking, "If I walk from the kitchen to the living room, is there actually a path, or is it just a wall?"
Why do it? It's a "sanity check." If the blueprint says there's a path, but your flashlight walk shows a dead end, you know something is wrong.

Stage 3: The All-Terrain Vehicle (Double Machine Learning - DML)

What it is: A sophisticated computer method that uses AI (Machine Learning) to handle distractions.
The Analogy: Imagine you are driving an all-terrain vehicle (ATV) through a jungle. The jungle is full of vines, mud, and rocks (other factors like "age," "income," or "other app features") that might hide the path.
- Standard methods might get stuck in the mud.
- DML is like a smart ATV that can automatically cut through the vines and smooth out the mud to see the path underneath. It asks: "Even if we ignore all these distractions, does the path from 'Trust' to 'Satisfaction' still exist?"
- The authors test this with three different "ATV engines" (Random Forest, Gradient Boosting, and Support Vector Machines) to make sure the path isn't just a trick of one specific engine.

The "Reverse Direction" Check

Sometimes, the map says "A causes B," but maybe "B causes A."

The Analogy: Imagine a river. The map says the water flows from the mountain to the sea. But what if the sea is actually pushing the water up the mountain?
The authors check this by driving the ATV backwards. If the path looks strong in both directions, it might be a two-way street (a reciprocal relationship) rather than a one-way road.

What They Found (The FinTech Case Study)

They tested this on a survey about Digital Customer Intimacy (how close customers feel to a financial app).

The Strong Roads: Most of the paths on their map were real. Whether they used the blueprint, the flashlight, or the ATV, the connections between things like "Perceived Quality," "Attitude," and "Digital Intimacy" held up.
The Shaky Bridges: Some paths were tricky.
- Usage vs. Satisfaction: The blueprint said "Using the app more makes you less satisfied." The flashlight agreed, but the ATV got confused depending on how it measured "usage." This means the relationship is sensitive to how you measure things.
- Trust vs. Intimacy: The map said "Trust leads to Intimacy," but the reverse check showed "Intimacy might also lead to Trust." This suggests a two-way relationship, not just a one-way street.
The Dead Ends: Some paths (like "Customer Awareness" leading to "Attitude") were weak in the blueprint and stayed weak in all the other tests. These are likely dead ends that should be removed from the final theory.

The Big Takeaway

The paper doesn't say "Machine Learning replaces the old way." Instead, it says: "Use the old way to build your theory, but use Machine Learning as a stress-test to see if your theory is strong enough to survive different conditions."

They even made a Google Colab workbook (a free, online coding notebook) so other researchers can use this exact "Map + Flashlight + ATV" workflow for their own studies. It's a toolkit to stop researchers from making decisions based on a single, potentially flawed map.

Technical Summary: From Structural Equation Modelling to Double Machine Learning

Problem Statement
Structural Equation Modelling (SEM) is the standard framework for survey-based research involving latent constructs (e.g., trust, satisfaction, intention). While SEM effectively assesses measurement models and estimates theory-driven structural relationships, its path significance is contingent on a specific model specification. A significant path coefficient in SEM does not inherently demonstrate stability under alternative estimation frameworks, score representations, or flexible control adjustments. Consequently, researchers lack a systematic, transparent workflow to determine whether key relationships remain robust when moving from latent-variable modeling to conventional regression or machine-learning-based adjustments. Recent literature has introduced Double Machine Learning (DML) to information systems and applied it in FinTech contexts, yet DML is rarely utilized as a post-SEM, path-by-path robustness diagnostic tool for latent-construct models.

Methodology: The Staged SEM–OLS–DML Framework
The paper proposes a two-layer, eight-stage robustness framework that integrates SEM, Ordinary Least Squares (OLS) regression, and Double Machine Learning (DML). The workflow is designed to separate the generation of robustness evidence from its interpretation, ensuring that machine learning complements rather than replaces SEM.

Layer 1: Evidence Generation
- Measurement Refinement & Baseline SEM: The process begins with refining the measurement model (validating indicators, reliability, and fit). Crucially, the full theory-specified structural model is re-estimated as a "robustness-baseline SEM model" before any paths are removed based on significance.
- Score Construction: Two types of respondent-level construct scores are generated: (a) SEM factor scores (preserving the latent structure and factor loadings) and (b) mean composite scores (a transparent, simple benchmark).
- SEM-to-Regression Translation: Each structural path in the baseline SEM is translated into a path-level specification for OLS and DML. For a structural equation $Y \sim D + X$ , the focal predictor $D$ is the variable of interest, while other predictors $X$ in the same equation are treated as co-predictor controls.
- OLS Benchmark: OLS regression is run on the construct scores to provide a transparent, linear benchmark. This checks if the SEM-implied path pattern remains visible under conventional regression assumptions.
- DML-Style Residualisation: DML is applied to examine stability under flexible control adjustment. The outcome ( $Y$ ) and focal predictor ( $D$ ) are regressed on the control vector ( $X$ ) using machine learning learners (Random Forest, Gradient Boosting, and Support Vector Machine) to estimate nuisance functions. The residuals are then used to estimate the focal relationship. Cross-fitting is employed to prevent overfitting.
- Sensitivity Checks: The framework includes checks for learner sensitivity (comparing RF, GBM, SVM) and score sensitivity (comparing factor vs. composite scores). Selected reverse-direction diagnostics are also performed to test directional robustness.
Layer 2: Integrated Interpretation
- Outputs from SEM, OLS, and DML are compared across methods, score types, and learners.
- Paths are classified based on convergence or divergence:
  - Strongly Robust: Stable direction and significance across all methods.
  - Partially Robust/DML-Sensitive: Direction is stable, but significance weakens under flexible control adjustment.
  - Score-Sensitive: Results depend heavily on whether factor scores or composite scores are used.
  - Directionally Unstable: Signs flip or reverse-direction diagnostics show stronger support for the opposite direction.

Empirical Demonstration
The framework is applied to a FinTech Digital Customer Intimacy (DCI) survey model containing 17 structural paths and constructs such as Perceived Quality, User Experience, Trust, and Emotional Connection.

Results: Most paths (e.g., $PQ \to AT$ , $AT \to BI$ , $EC \to DCI$ ) remained strongly robust across SEM, OLS, and DML learners for both score types.
Nuanced Findings:
- $UB \to SA$ (Usage Behaviour to Satisfaction): Significant and negative in SEM and factor-score DML, but weak/non-significant in composite-score checks, indicating score-representation sensitivity.
- $TR \to EC$ (Trust to Emotional Connection): Non-significant in SEM but significant in score-based checks with sign sensitivity, requiring theoretical review.
- $CA \to AT$ (Customer Awareness to Attitude): Weak across all methods, suggesting it may be a candidate for removal if theory is weak.
- Reverse Diagnostics: The relationship between Emotional Connection and DCI showed stability in both directions, suggesting a reciprocal association.

Key Contributions

Practical Workflow: The paper provides a reproducible, staged workflow for translating SEM-implied paths into OLS and DML-style robustness checks without treating machine learning as a substitute for latent-variable modeling.
Diagnostic Clarity: It clarifies how to interpret score-representation sensitivity (factor vs. composite) and learner sensitivity (RF vs. GBM vs. SVM) within survey-based research.
Resource Availability: The authors provide a Google Colab workbook and generated result files via Zenodo, offering a reusable template for researchers to adapt this robustness workflow to other survey-based latent-construct studies.

Significance and Claims
The paper positions this framework as a "model-decision support process" rather than a simple robustness appendix. Its primary significance lies in shifting the research practice from "trim-then-check" (removing non-significant paths immediately after SEM) to a "check-then-decide" approach. By retaining the full theoretical path system for downstream robustness analysis, researchers can make final structural decisions based on a convergence of evidence from latent modeling, transparent regression, and flexible machine-learning adjustments.

The authors explicitly state that this approach strengthens robustness assessment but does not establish definitive causal claims (interventional or counterfactual) in cross-sectional survey data. The framework is designed to identify which relationships are stable and which require cautious interpretation, further theoretical discussion, or longitudinal validation. It addresses a methodological gap by positioning DML specifically as a post-SEM diagnostic tool for latent-construct models, distinct from its use as a primary estimator in causal inference or as a predictive tool in SEM-ML integration studies.