📊 statistics

Factorizable joint shift revisited

This paper proposes a general framework for analyzing factorizable joint shift across both classification and regression tasks with general label spaces, extending existing results and introducing a generalized Expectation-Maximization algorithm for label distribution estimation.

Original authors: Dirk Tasche

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dirk Tasche

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has perfected a recipe for a delicious soup (the Source Distribution) using a specific set of ingredients and a specific number of diners. You know exactly how the flavors interact.

Now, you want to serve this soup to a new group of people (the Target Distribution). However, there's a problem: the new group has different tastes, different dietary needs, and perhaps even different ingredient availability. This change in the "environment" is what data scientists call Distribution Shift.

If you just serve the old recipe to the new crowd, it might taste terrible. This paper is about figuring out how to adjust your recipe so it works for the new crowd, even when you can't ask them directly what they like (you don't have their "labels" or feedback).

Here is a breakdown of the paper's main ideas using simple analogies:

1. The Problem: Two Types of Changes

Usually, when a recipe fails for a new crowd, it's because of one of two things:

Covariate Shift (The Ingredients Changed): The new crowd is eating with different vegetables or spices available, but they still like the same flavor profile. (e.g., The training data had mostly red peppers, but the test data has mostly green peppers).
Label Shift (The Crowd Changed): The ingredients are the same, but the new crowd has different preferences. Maybe they are all spicy-food lovers now, whereas the old crowd liked mild food. (e.g., The training data had 50% spicy dishes, but the test data is 90% spicy).

2. The New Idea: Factorizable Joint Shift (FJS)

The paper introduces a concept called Factorizable Joint Shift (FJS). Think of this as a "Double Trouble" scenario where both the ingredients and the crowd's preferences have changed, but in a very specific, predictable way.

The author argues that FJS isn't just a random mess; it's actually a two-step dance:

First, the crowd's preferences change (Label Shift).
Then, the ingredients change based on those new preferences (Covariate Shift).
(Or vice versa: Ingredients change first, then preferences adjust).

The magic of FJS is that even though both things changed, they didn't change chaotically. They changed in a way that allows us to mathematically "untangle" the two shifts. It's like realizing that the new soup tastes different not just because the carrots are different, but because the ratio of carrots to potatoes changed in a specific, calculable pattern.

3. The Big Leap: From Categories to Continuum

Previous research on this "two-step dance" only worked for simple, categorical labels (like "Spicy" vs. "Mild" or "Cat" vs. "Dog").

This paper's main contribution is expanding the math to handle general label spaces.

Old way: Only worked for distinct buckets (Classification).
New way: Works for buckets and for continuous scales like temperature, height, or money (Regression).

Imagine the old math could only tell you if a soup was "Hot" or "Cold." This new framework can tell you exactly how many degrees hotter it is, even if the ingredients and the crowd have both shifted.

4. The Solution: The "EM Algorithm" as a Smart Adjuster

The paper proposes a method to fix the recipe using an algorithm called Expectation Maximization (EM).

Think of the EM algorithm as a smart sous-chef who is trying to guess the new crowd's preferences without asking them directly.

The Setup: You know the old recipe (Source) and you know the new ingredients available (Target Features), but you don't know the new crowd's taste preferences (Target Labels).
The Process:
1. Guess: The sous-chef makes a guess about the new crowd's preferences.
2. Check: They see if this guess explains the new ingredients.
3. Refine: If the guess doesn't fit, they tweak it.
4. Repeat: They keep guessing and tweaking until the math says, "Okay, this is the most likely preference profile that explains the new ingredients."

The paper proves that this "guess-and-check" loop works even when the labels are continuous numbers (like predicting the exact price of a house) and not just simple categories.

5. The "Perfect Fit" vs. "Good Enough"

The paper also discusses a tricky situation: What if the new crowd's preferences are so weird that no amount of adjusting the ingredients can perfectly match the new reality?

The "Exact Fit" Dream: Ideally, we want to find a new recipe that perfectly matches the new ingredients and the new crowd.
The Reality: Sometimes, mathematically, you can't get a perfect match. The paper shows that in these cases, the algorithm finds the "best possible" approximation. It minimizes the "distance" (a mathematical measure of error) between what you predicted and what actually happened.

6. Connection to "Generalized Label Shift"

The paper also revisits a related concept called Generalized Label Shift (GLS).

The Metaphor: Imagine you are trying to understand a new crowd by translating their language into your own. GLS suggests that if you translate the ingredients into a "simplified language" (a representation), the shift looks like a simple change in preferences.
The Finding: The paper proves that if this translation is done correctly, it is mathematically the same as the "two-step dance" (FJS) described earlier. This means you don't always need to find a complex new translation; you can often just use the FJS method directly.

Summary

In short, this paper takes a complex mathematical problem about how data changes over time and:

Generalizes it: It works for any type of data, not just simple categories.
Clarifies it: It shows that complex shifts are often just a sequence of simpler shifts happening one after another.
Provides a tool: It offers a robust mathematical method (an updated EM algorithm) to estimate what the new data looks like, even when you are missing half the information.

It's essentially a new, more powerful toolkit for chefs (data scientists) to ensure their soup (models) tastes right, even when the kitchen (data environment) has completely changed.

1. Problem Statement

The paper addresses the challenge of distribution shift (or dataset shift) in machine learning, where a model trained on a source distribution $P$ (training data) fails when deployed on a target distribution $Q$ (test data). Specifically, the paper focuses on scenarios where:

Feature observations ( $X$ ) are available for the target dataset.
Label observations ( $Y$ ) are missing for the target dataset (unsupervised domain adaptation).
The label space $\Omega_Y$ is general, encompassing not just categorical variables (classification) but also continuous variables (regression) and mixed spaces.

Existing literature has largely focused on specific shift assumptions like Covariate Shift (change in $P(X)$ , $P(Y|X)$ constant) or Label Shift (change in $P(Y)$ , $P(X|Y)$ constant). However, these are often insufficient for real-world tasks. Factorizable Joint Shift (FJS) was proposed as a more general assumption covering both, but previous analyses were restricted to categorical labels. This paper aims to generalize FJS to arbitrary label spaces and provide robust estimation frameworks.

2. Methodology and Framework

2.1. Theoretical Framework

The author establishes a rigorous probabilistic framework using measure theory:

Source ( $P$ ) and Target ( $Q$ ): Defined on a product space $\Omega = \Omega_X \times \Omega_Y$ .
Absolute Continuity: The target distribution $Q$ is assumed to be absolutely continuous with respect to the source $P$ ( $Q \ll P$ ), allowing the definition of a Radon-Nikodym derivative (density) $f = \frac{dQ}{dP}$ .
FJS Definition: $P$ and $Q$ are related by FJS if the density $f$ factorizes into a product of a function of features and a function of labels:
$\frac{dQ}{dP} = h(X)g(Y)$
where $h$ and $g$ are measurable functions. This implies that the joint shift is a combination of a covariate shift and a label shift.

2.2. Characterization of FJS

The paper proves that FJS is equivalent to a sequence of consecutive shifts (e.g., a label shift followed by a covariate shift).

Theorem 3.5: Establishes the relationship between the factorization functions ( $h, g$ ) and the marginal densities of the target distribution. It shows that under FJS, the target conditional densities are "scaled" versions of the source conditional densities.
Generalization: The paper extends previous results (limited to categorical $Y$ ) to general label spaces, showing that FJS implies that the ratio of target to source conditional densities depends only on the conditioning variable.

2.3. Estimation Algorithms

The paper addresses two specific problems under the FJS assumption:

Problem (A): Both target marginals $Q_X$ and $Q_Y$ are known.
Problem (B): Only the target feature marginal $Q_X$ is known (the standard unsupervised setting).

The Generalized EM Algorithm (Section 3.3):
To solve Problem (B), the author proposes a generalization of the Expectation-Maximization (EM) algorithm (originally by Saerens et al., 2002, for class priors).

Goal: Estimate the unknown target label density $g(Y)$ (or the target label distribution) given the known target feature density $h(X)$ .
Iteration:
1. Initialize a label density $g_0(Y)$ .
2. E-step: Compute the implied feature density $h_n(X) = E_P[g_n(Y) | X]$ .
3. M-step: Update the label density $g_{n+1}(Y) = g_n(Y) E_P[\frac{h(X)}{h_n(X)} | Y]$ .
Convergence: Theorem 3.8 proves that this iteration minimizes the Kullback-Leibler (KL) divergence between the predefined target feature density and the implied feature density. The algorithm converges to a solution where the target label distribution is estimated such that the resulting joint distribution satisfies the FJS constraint.

Iterative Solution for Problem (A):
For the case where both marginals are known, an alternating iteration is proposed to find the factorization functions $h$ and $g$ that satisfy the marginal constraints simultaneously.

2.4. Relation to Generalized Label Shift (GLS)

The paper analyzes the relationship between FJS and Generalized Label Shift (GLS), a concept used in representation learning where features are mapped to a space where only label shift occurs.

Proposition 4.1: If a representation mapping $R$ is sufficient for the features (i.e., $P(Y|X) = P(Y|R)$ ), then GLS in the mapped space implies FJS in the original space.
Implication: This suggests that one might not need to explicitly learn a representation mapping to achieve GLS; instead, one can directly apply the FJS-based EM algorithm to the original features.

3. Key Contributions

Generalization to Continuous/Mixed Labels: The primary contribution is extending the theory of Factorizable Joint Shift from categorical labels to general label spaces (including regression). This allows the application of FJS concepts to regression problems and mixed data types.
Generalized EM Algorithm: The paper presents a rigorous derivation of an EM algorithm for label distribution estimation in general spaces. It proves that this algorithm minimizes the KL divergence between the target feature distribution and the model's implied distribution, providing a theoretical guarantee for its use in unsupervised domain adaptation.
Decomposition of FJS: It formally demonstrates that FJS can be viewed as a sequence of consecutive label and covariate shifts, clarifying the structural relationship between these shift types.
Connection to GLS: It establishes a formal link showing that GLS with a sufficient representation implies FJS, suggesting that direct FJS modeling may be an alternative to complex representation learning.
Handling "Exact Fit": The paper discusses the conditions under which a target feature distribution can be perfectly matched ("exact fit") by a label shift or constrained FJS. It notes that while exact fit is not always guaranteed (e.g., in normal mixture models), the EM algorithm provides the best possible approximation in terms of KL divergence.

4. Key Results

Theorem 3.5: Provides the necessary and sufficient conditions for a joint distribution to satisfy FJS in terms of marginal densities and conditional expectations.
Theorem 3.8: Proves the convergence properties of the generalized EM algorithm, showing that the sequence of implied feature densities monotonically approaches the target feature density in terms of KL divergence.
Corollary 3.17: Shows that in the case of finite categorical features and general labels, if the posterior probabilities are linearly independent, the EM algorithm achieves an exact fit of the target feature distribution.
Example 3.16: A numerical experiment demonstrates the algorithm's ability to estimate a continuous target label distribution (Gaussian) given a discrete target feature distribution, showing that while the final label density depends on initialization, the feature distribution is matched with high precision.

5. Significance

Bridging Classification and Regression: By moving beyond categorical labels, this work unifies the treatment of distribution shift for both classification and regression tasks, which is a significant step forward as most existing domain adaptation literature focuses heavily on classification.
Practical Domain Adaptation: The proposed EM algorithm offers a practical, computationally feasible method for adapting models when target labels are unavailable, provided the FJS assumption holds. This is crucial for real-world applications like medical diagnosis (where test data labels are delayed) or financial risk modeling.
Theoretical Clarity: The paper clarifies the mathematical structure of FJS, resolving ambiguities about its relationship with covariate and label shifts and providing a solid theoretical foundation for future research in distribution shift.
Alternative to Representation Learning: By linking GLS and FJS, the paper suggests that in some cases, complex representation learning might be bypassed in favor of direct density estimation under FJS, potentially simplifying the adaptation pipeline.

In summary, "Factorizable joint shift revisited" provides a comprehensive theoretical and algorithmic framework for handling distribution shifts in general settings, offering a robust alternative to standard covariate or label shift assumptions and extending these capabilities to regression problems.