📊 statistics

Identifying Direct Causal Effects in Latent Factor Models by Accounting for Unidentified Parents

This paper introduces a new identification criterion for linear structural equation models with latent variables that improves upon existing methods for densely confounded graphs by leveraging recursive schemes that explicitly account for causal parents with unidentified direct effects, utilizing network-flow computations to provide a practical algorithmic solution.

Original authors: Tom Hochsprung, Nils Sturma, Jakob Runge, Mathias Drton, Andreas Gerhardus

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tom Hochsprung, Nils Sturma, Jakob Runge, Mathias Drton, Andreas Gerhardus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out who is really pulling the strings in a complex organization. You can see the employees (the observed variables) and you can see their daily interactions, but you know there are invisible managers (the latent variables) pulling the strings behind the scenes that you can't see.

Your goal is to map out exactly who influences whom directly. For example, does Employee A directly influence Employee B, or is it just that they both report to the same invisible Manager C?

This paper presents a new, smarter way to solve this detective work, specifically for situations where the "invisible managers" are causing a lot of confusion (confounding) among the employees.

The Problem: The "Blind Spot" of Old Methods

Previously, detectives used a method called the Latent-Factor Half-Trek Criterion (LF-HTC). Think of this like trying to solve a puzzle by looking at the pieces from a distance.

How it worked: It tried to "project away" the invisible managers, treating them as just background noise.
The flaw: When the invisible managers are very busy and influencing many employees at once (a "dense" graph), this old method gets confused. It often gives up and says, "I can't tell who is influencing whom," even when the answer is actually there. It's like trying to hear a whisper in a crowded room by just turning up the volume; the noise drowns out the signal.

The Solution: The "eLF-HTC" Detective Tool

The authors of this paper developed a new tool called the eLF-HTC (Edgewise Latent-Factor Half-Trek Criterion). Instead of ignoring the invisible managers, this new method explicitly accounts for them and uses a clever strategy to handle the noise.

Here is how their new approach works, using a few analogies:

1. The "Unidentified Parents" Strategy
Imagine you are trying to figure out who is the boss of Employee X.

Old way: You needed to know the identity of every single person who influenced X before you could solve the puzzle. If you were missing one name, you were stuck.
New way (eLF-HTC): The authors realized you don't need to know everyone immediately. You can start solving for the boss of X even if you haven't fully identified some of X's other connections yet. It's like solving a Sudoku puzzle: you don't need to fill in every number to know that a specific square must be a 7. You can use the "unknowns" as part of your calculation to find the answer.

2. The "Network Flow" Traffic System
To make this math work, the authors turned the problem into a traffic flow problem.

Imagine the relationships between employees are roads.
The goal is to send "traffic" (information) from a group of starting points to a group of destinations without any traffic jams (collisions).
The paper uses computer algorithms (specifically network-flow computations) to check if there is a clear path for the information to flow. If the traffic can flow smoothly through a specific pattern, it proves that the direct influence exists and can be calculated. This is much faster and more powerful than trying to solve the whole puzzle by brute force.

3. The "Recursive" Cleanup Crew
The paper also introduces a "recursive" method. Think of this as a cleanup crew.

Once the detective figures out one direct influence (e.g., "A definitely influences B"), they can "delete" that connection from the map.
By removing that known connection, the map becomes simpler.
The detective then looks at the simplified map again to find new connections that were previously hidden by the complexity of the first one. They repeat this process, peeling back layers of the onion until the whole picture is clear.

What They Found

The authors tested their new tool against the old one using thousands of different scenarios (simulated graphs).

The Result: The old method (LF-HTC) could only solve about 25% to 71% of the puzzles, depending on how messy the graph was.
The New Method: Their combined tool (using the new eLF-HTC, the traffic flow math, and the recursive cleanup) solved nearly 98% of the puzzles.
The "Dense" Advantage: The improvement was most dramatic in the "dense" scenarios where many employees were influenced by the same invisible managers. In these tough cases, the old method solved less than 1% of the puzzles, while the new one solved 96%.

The Takeaway

This paper doesn't just say "we have a better idea." They actually built a software tool (available in R) that allows researchers to:

Check if a causal relationship can be identified.
Get the exact mathematical formula to calculate that relationship.
Do this much faster and for much more complex situations than before.

In short, they gave detectives a new set of glasses that lets them see through the fog of invisible managers to clearly identify who is directly influencing whom, even in the most crowded and confusing organizational charts.

Technical Summary: Identifying Direct Causal Effects in Latent Factor Models by Accounting for Unidentified Parents

Problem Statement

The paper addresses the problem of identifying direct causal effects in linear structural equation models (LSEMs) that include explicitly modelled latent variables. In such models, observed variables $X$ and latent variables $L$ are related via linear equations with stochastic noise. The primary goal is to determine which coefficients (direct causal effects) in the structural equations can be uniquely recovered (identified) from the observed covariance matrix $\Sigma$ of $X$ , and to derive rational formulas for these coefficients.

Existing approaches often rely on latent projections, where latent variables are marginalized out, resulting in dependent error terms represented by bidirected edges. However, the authors note that when observed variables are densely confounded by even a few latent variables, projection-based methods frequently fail to certify identifiability. While methods that explicitly use latent variables exist, such as the Latent-Factor Half-Trek Criterion (LF-HTC) proposed by Barber et al. (2022a), they remain inconclusive for many dense causal graphs, despite the fact that rational identifiability might theoretically exist (verifiable via computationally expensive Gröbner basis methods).

Methodology

The authors propose a new identification framework that generalizes the LF-HTC by explicitly accounting for unidentified parents during the identification process. The methodology consists of three main pillars combined into a recursive algorithm:

1. The Edgewise Latent-Factor Half-Trek Criterion (eLF-HTC)

The core contribution is the eLF-HTC, a generalized version of the LF-HTC.

Relaxation of Constraints: Unlike the LF-HTC, which requires auxiliary sets $Z$ to have no observed parents (or for all parents to be already identified), the eLF-HTC allows $Z$ to contain nodes with unidentified parents.
Mechanism: The criterion introduces sets $W_v \subseteq pa_V(v)$ (unidentified parents of the target $v$ ) and $W_z \subseteq pa_V(z)$ (unidentified parents of auxiliary nodes $z$ ). It checks for the existence of a system of latent-factor half-treks from a set of source nodes $Y$ to the union of these unidentified parents and auxiliary nodes, subject to specific "no-sided intersection" constraints.
Algorithmic Implementation: The authors demonstrate that checking the eLF-HTC can be reduced to network-flow computations (specifically max-flow problems) on a constructed flow graph. This allows for practical implementation despite the combinatorial nature of the search space.

2. Determinantal Identification Result

The paper incorporates a result adapted from Weihs et al. (2018), originally formulated for mixed graphs with dependent errors.

Transfer to Latent Models: The authors transfer this result to the setting of explicitly modelled latent variables.
Mechanism: This approach uses max-flow computations on a determinantal flow network to identify edges based on the rank of submatrices of the covariance matrix $\Sigma$ . It provides a rational formula involving determinants of submatrices of $\Sigma$ .
Synergy: This method can identify edges that the eLF-HTC cannot, particularly in specific structural configurations.

3. Recursive Subgraph Identification

The authors introduce a meta-strategy to enhance the power of the above criteria.

Procedure: The algorithm recursively deletes edges that have already been identified as rationally identifiable from the graph.
Covariance Adjustment: When edges are deleted, the covariance matrix of the resulting subgraph is not simply a submatrix of the original $\Sigma$ . The authors provide Proposition 4.5, which gives rational formulas to calculate the required covariances for the subgraph using the original $\Sigma$ and the coefficients of the deleted edges.
Benefit: By reducing the graph, the constraints for the eLF-HTC and determinantal criteria may be relaxed, allowing for the identification of further edges that were previously blocked by the presence of the deleted edges.

Combined Algorithm

The paper presents Algorithm 1, which iteratively applies the eLF-HTC, the determinantal result, and the recursive subgraph procedure. It maintains a set of "solved" nodes and edges, updating the set of allowed covariances as edges are identified and removed.

Key Results and Experiments

Theoretical Contributions

Generalization: The eLF-HTC strictly subsumes the LF-HTC. Any edge identifiable via LF-HTC is also identifiable via eLF-HTC.
Rational Formulas: The proofs provide explicit rational formulas for the identified coefficients, ensuring that the identification is constructive.
Complexity: While the general problem of LSEM identifiability can have double-exponential time complexity (via Gröbner bases), the proposed algorithm achieves polynomial time complexity under specific simplifying assumptions (e.g., bounding the size of search sets $H$ , $S$ , and $T$ ). Without simplifications, the complexity is super-exponential but significantly better than algebraic methods for practical graph sizes.

Numerical Experiments

The authors evaluate their combined algorithm against the LF-HTC and computer algebra methods (Gröbner bases) on two simulation settings:

Setting 1: 6 observed variables, 1 latent variable.
Setting 2: 6 observed variables, 2 latent variables.

Findings:

Performance: The combined algorithm identifies significantly more graphs than the LF-HTC, especially in dense settings (high number of observed edges).
- In Setting 1 with 9 observed edges, LF-HTC identified <1% of identifiable graphs, whereas the combined algorithm identified ~96%.
- In Setting 2 with 6 observed edges, LF-HTC identified ~65%, while the combined algorithm identified ~97%.
Completeness: The combined approach identifies nearly all (approx. 98%) of the rationally identifiable graphs in the tested settings, approaching the theoretical limit set by computer algebra methods but with much lower computational cost.
Synergy: The results demonstrate that the three components (eLF-HTC, determinantal, recursive) provide strict improvements over one another; their combination yields more identifiable cases than the union of their individual results.

Significance and Claims

The paper claims to provide the most general graphical results for latent-factor graphs currently known.

Practical Utility: The authors emphasize that their approach offers a practical alternative to Gröbner basis computations, which are often infeasible for moderate-sized graphs due to double-exponential time complexity.
Software Availability: The authors provide an R implementation (available at a GitLab repository) that allows users to check identifiability, obtain LaTeX-formatted identification formulas, and compute estimates of causal effects from covariance matrices.
Modesty: The authors acknowledge that while their method covers nearly all identifiable cases in their experiments, there may remain a small fraction of rationally identifiable graphs for which no graphical criterion (including theirs) currently provides a certificate. They note that future work could involve deriving necessary conditions or combining their ideas with methods for more general latent structures (e.g., where latents are not source nodes).

In summary, the paper presents a robust, computationally tractable framework for causal discovery in the presence of latent confounders, overcoming the limitations of projection-based methods and previous half-trek criteria by explicitly handling unidentified parents and leveraging recursive graph reduction.