📊 statistics

Sanity Checking Causal Representation Learning on a Simple Real-World System

This paper evaluates state-of-the-art causal representation learning methods on a simple, real-world optical experiment with known ground truth and finds that they consistently fail to recover underlying causal factors due to reproducibility issues and the violation of key theoretical assumptions, highlighting a significant gap between theoretical promise and practical application.

Original authors: Juan L. Gamella, Simon Bing, Jakob Runge

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan L. Gamella, Simon Bing, Jakob Runge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For decades, scientists have sought a way to teach computers to see the world not just as a blur of pixels, but as a collection of distinct causes and effects. Imagine trying to understand a complex machine by looking only at the smoke coming out of its exhaust; you might guess what is happening inside, but you cannot be sure. This is the challenge of causal representation learning. It is a field of artificial intelligence that aims to peel back the layers of raw data to find the hidden, fundamental factors that actually drive a system. If a computer can learn these underlying causes, it could make better predictions, understand why things go wrong, and adapt to new situations much like a human does. The promise is that if we can teach machines to identify these true causes, they will become far more reliable and useful in the real world, rather than just memorizing patterns in a dataset.

However, a significant gap has emerged between the theory and reality. Many new methods for teaching computers to find these causes have been tested only on computer-generated data, where the rules are perfectly known and the noise is carefully controlled. It is like testing a new car engine only on a treadmill in a laboratory; the engine might run perfectly there, but that does not guarantee it will start on a rainy road. Researchers have long suspected that these methods might struggle when faced with the messy, unpredictable nature of the real world, but until now, there has been no simple, real-world test to prove it.

A team of researchers has now built such a test, using a physical device that is simple enough to understand but real enough to matter. They constructed a "light tunnel," a long chamber containing a controllable light source, two rotating filters that change the direction of light waves, and a camera along with several sensors to measure the light intensity. The researchers could precisely control the brightness of the red, green, and blue lights, as well as the angle of the two filters. Because they set these controls themselves, they knew the exact cause of every change in the light. The camera and sensors then recorded the resulting images and measurements, which were a tangled mix of all those inputs. This setup provided a perfect ground truth: the researchers knew exactly what the causes were, so they could see if the computer algorithms could successfully find them again.

The team took three different, leading approaches to causal representation learning and asked them to solve this puzzle. The first method relied on the computer being shown data from different "environments" where the controls were deliberately changed. The second method asked the computer to look at the data from multiple different angles or "views" simultaneously. The third method tried to learn from a sequence of events over time, watching how the system evolved. In a perfect world, these methods should have easily identified the red, green, and blue brightness levels and the two filter angles as the hidden causes.

The results were a stark reality check. None of the methods succeeded in consistently recovering the true underlying causes from the real-world data. When the researchers fed the algorithms the actual images and sensor readings from the light tunnel, the computers failed to untangle the causes. They produced results that were weak, inconsistent, or completely wrong. To understand why this happened, the researchers created a simplified, computer-generated version of the same experiment. In this synthetic version, the relationship between the controls and the images was perfectly smooth and predictable, without the tiny, random fluctuations that occur in real electronics.

When they ran the same algorithms on this clean, synthetic data, the results were mixed. One of the methods, which relied on seeing different environments, finally worked well on the synthetic data, recovering the causes with high accuracy. This suggested that the method was sound in theory but too sensitive to the tiny, random noise present in the real sensors. However, the other two methods failed just as badly on the synthetic data as they did on the real data. This was a surprising discovery. It meant that for these methods, the problem was not just the noise of the real world, but something deeper in how the algorithms were built or how they were trained. They simply could not solve the puzzle, even when the rules were made as simple as possible.

The study highlights a critical issue in the field: the difference between a method working in a controlled simulation and working in the real world. The researchers found that the assumptions these algorithms make about how data is generated often do not hold up when faced with real physical systems. For instance, some methods assume that the relationship between the cause and the observation is perfectly smooth and predictable, but real sensors introduce small, unpredictable jitters that break these assumptions. Furthermore, the team discovered that the success of these methods often depended heavily on specific choices made by the programmers, such as the exact shape of the computer network or the way the data was prepared, rather than the core theory itself.

This work does not mean that the goal of teaching computers to understand cause and effect is impossible. Instead, it serves as a necessary "sanity check," a simple reality test that reveals where the current tools fall short. By showing that even the most advanced methods struggle on a simple, well-understood physical system, the researchers have identified a clear path forward. The field must move beyond testing on perfect, computer-generated data and begin to grapple with the imperfections of the real world. Only by building methods that can handle the noise and complexity of actual physical systems can we hope to create artificial intelligence that truly understands the world it lives in.

Technical Summary: Sanity Checking Causal Representation Learning on a Simple Real-World System

Problem Statement

Causal Representation Learning (CRL) aims to recover the underlying causal latent ground-truth model from high-level, entangled observations of a system. While significant theoretical progress has been made regarding the identifiability of causal factors under specific assumptions (e.g., specific mixing functions, intervention types, or temporal structures), the field lacks meaningful real-world benchmarks. Existing evaluations predominantly rely on synthetic datasets generated according to the specific assumptions of the methods being tested. This practice validates theoretical foundations but offers limited insight into the applicability of these methods to real-world problems where data-generating processes are complex, noisy, and often violate theoretical assumptions.

Methodology

The authors introduce a "sanity check" based on a controlled, real-world physical system known as a "light tunnel" (Gamella et al., 2025). This system serves as a testbed where the ground truth is known, allowing for direct evaluation of CRL methods.

The Physical System

The system consists of a controllable light source (Red, Green, Blue LEDs), two linear polarizers mounted on rotating frames, a camera, and various light and angle sensors.

Ground Truth (Latent Factors): The control inputs: light source brightness ( $R, G, B$ ) and polarizer angles ( $\theta_1, \theta_2$ ).
Observations: Images captured by the camera and numeric sensor readings (light intensity, current, and noisy angle measurements).
Mixing Function: The transformation from inputs to observations is governed by physical principles (e.g., Malus' law for polarization, inverse-square law for light intensity). Crucially, this process is stochastic due to light source fluctuations and sensor noise, and it is not purely additive.

Experimental Design

The authors evaluated three representative families of CRL methods:

Contrastive CRL: Methods requiring interventional data (Buchholz et al., 2023).
Multiview CRL: Methods utilizing multiple disjoint sets of observations (Yao et al., 2024c).
Temporal CRL: Methods exploiting causal temporal structures (CITRIS, Lippe et al., 2022).

To isolate the causes of failure, the authors performed a synthetic ablation. They replaced the real-world data-generating process with a deterministic simulator that produces synthetic images and sensor measurements from the same inputs. This simulator adheres strictly to the theoretical assumptions of the evaluated methods (e.g., deterministic mixing, additive noise where assumed), serving as a control to distinguish between "misspecification" (violated assumptions) and "optimization" (failure to converge) failures.

Key Contributions

Real-World Benchmark: The introduction of the lt_crl_benchmark_v1 dataset, derived from a physical optical experiment, providing a ground-truth evaluation platform for CRL.
Synthetic Ablation Framework: A methodology to substitute real data with a deterministic surrogate to test the robustness of CRL methods against the specific noise and stochasticity of real-world mixing functions.
Comprehensive Evaluation: A systematic evaluation of state-of-the-art methods across different theoretical paradigms on both real and synthetic data, accompanied by extensive hyperparameter searches and code reproduction efforts.
Public Resources: Release of the dataset, data collection procedures, and reproduction code to facilitate future research.

Results

The evaluation revealed that all tested methods failed to consistently recover the underlying causal factors on the real-world data.

Contrastive CRL (Buchholz et al., 2023):
- Synthetic Data: Performed well, recovering latent variables and the causal graph with high accuracy (MCC $\approx$ 0.89).
- Real Data: Performance collapsed (MCC $\approx$ 0.28). The method failed to recover latent variables and produced inconsistent causal graphs across random initializations.
- Conclusion: The failure is attributed to the misspecification of the mixing function. The method assumes a deterministic mixing process, but the real system involves complex, non-additive stochastic noise.
Multiview CRL (Yao et al., 2024c):
- Performance: Failed to recover content variables (specifically polarizer angles $\theta_1, \theta_2$ ) from the intersection of image and sensor views, even when theoretical assumptions (like injectivity) were met in the synthetic ablation.
- Observation: Performance was similarly poor on both real and synthetic data. The authors hypothesize that while supervised models can easily recover polarizer angles from images, self-supervised multiview methods struggle to capture this information from the image modality, possibly due to data modality dependencies.
Temporal CRL (CITRIS, Lippe et al., 2022):
- Performance: Failed to recover ground-truth factors on both real and synthetic data (low diagonal $R^2$ scores, high off-diagonal correlations).
- Conclusion: The failure likely stems from a combination of misspecification and optimization issues. The complexity of the method's sub-modules may compound errors, and the specific architecture or training regime may not be robust to the data characteristics, even in the deterministic setting.

Significance and Claims

The paper positions itself as a "sanity check" rather than a definitive critique of the entire field. The authors make the following modest claims:

Gap Between Theory and Practice: There is a stark contrast between the theoretical promise of CRL methods and their performance in a simple, controlled real-world setting.
Reproducibility Challenges: The results highlight a "reproducibility problem," where methods often fail even on simplified synthetic ablations, suggesting that implementation details, pre-processing, and optimization choices play a larger role than theoretical guarantees alone.
Role of Assumptions: The study demonstrates that common assumptions regarding the mixing function (e.g., determinism, additivity) are crucial for performance but frequently do not hold in real-world data.
Negative Results are Informative: A failure on this simple testbed indicates a high probability of failure in more complex real-world scenarios. However, success on this benchmark does not guarantee success in all applications.
Call to Action: The authors argue that while theoretical advancements are necessary, the field must also focus on the "engineering endeavor" of implementing robust, reproducible algorithms. They advocate for the use of real-world physical testbeds to break down the hard problem of CRL into manageable steps, moving beyond purely synthetic validation.

The paper concludes that without bridging the gap between theoretical assumptions and the messy reality of physical data generation, the application of CRL to real-world problems remains limited.