Unsupervised Domain Adaptation for Binary Classification with an Unobservable Source Subpopulation

This paper addresses unsupervised domain adaptation in binary classification where a source subpopulation is unobservable, proposing a distribution matching method with theoretical guarantees to recover target domain predictions and outperform naive benchmarks that ignore this missing data.

The Gist

Imagine you are a chef trying to teach a new apprentice how to cook a specific dish. You have a massive cookbook (the Source Domain) filled with recipes and photos. However, there's a catch: due to a strange quirk in how the book was compiled, one specific type of ingredient is completely missing from the photos.

For example, let's say you are teaching the apprentice to identify Birds (Label $Y$) based on their Background (Label $A$).

• The Missing Group: In your cookbook, you have photos of "Landbirds on Land," "Landbirds on Water," and "Waterbirds on Land." But, for some reason, there are zero photos of "Waterbirds on Water."
• The Goal: You want the apprentice to be able to identify birds in a new, real-world environment (the Target Domain) where all types of birds and backgrounds exist, including those missing "Waterbirds on Water."

If you just blindly tell the apprentice, "Look at the photos I have and guess," they will fail miserably. They might think, "Oh, all waterbirds are on land," or they might get confused when they see a waterbird on water in the real world. This is the problem of Unsupervised Domain Adaptation with Structured Missingness.

Here is how this paper solves that problem, broken down into simple concepts:

1. The Problem: The "Invisible" Group

In many real-world situations, data isn't missing randomly. It's missing because of how the world works.

• Real-world example: In a hospital database, you might have data on "Men with Disease X" and "Women without Disease X," but perhaps "Men without Disease X" were never recorded because of an old rule.
• The Risk: If you train an AI on this incomplete data and send it to a new hospital where all groups exist, the AI will make biased, wrong predictions for the missing group. It's like trying to navigate a city using a map that has a whole neighborhood erased.

2. The Secret Sauce: The "Conditional Invariance" Rule

The authors make a clever assumption to bridge the gap. They say:

"Even though the mix of birds is different in the new world, the way a bird looks is the same."

In technical terms, they assume that if you look at a "Waterbird on Water," its visual features (feathers, beak shape) look the same whether it's in the old cookbook or the new real world. The only thing that changes is how many of them there are.

This is like saying: "A Ferrari looks like a Ferrari whether it's in a showroom or on a race track. The only difference is that the showroom has 100 Ferraris and the race track has 1, or vice versa."

3. The Solution: The "Distribution Matching" Detective

Since the "Waterbirds on Water" are invisible in the source data, how do we figure out how the AI should handle them?

The authors propose a method called Distribution Matching. Here is the analogy:

Imagine you have a smoothie (the Target Domain) that contains four fruits: Apples, Bananas, Cherries, and Dates.

• You have a Source Smoothie that only has Apples, Bananas, and Cherries. The Dates are missing.
• You know the taste of Apples, Bananas, and Cherries perfectly from the Source.
• You want to know the recipe (proportions) of the Target Smoothie so you can predict what it will taste like.

The authors' method works like this:

1. Look at the "Bananas" (The Visible Group): In the Target, you can see the Bananas. You know what a "Banana" tastes like from the Source.
2. Do the Math: By comparing how the "Bananas" are distributed in the Target versus the Source, you can mathematically deduce how much "Date" (the missing fruit) must be hiding in the mix to make the total flavor balance out.
3. The KL-Divergence: This is just a fancy mathematical tool (like a "flavor distance meter") that helps them find the exact proportions of the missing group that make the Target smoothie taste consistent with the Source's rules.

4. The Result: A Better Map

Once they figure out the proportions of the missing group, they can rewrite the "recipe" for the new world.

• Naive Approach (The Old Way): "Ignore the missing group, just guess based on what you see." -> Result: The AI thinks all waterbirds are landbirds.
• This Paper's Approach: "Use the visible groups to mathematically reconstruct the invisible group." -> Result: The AI correctly identifies the waterbirds on water, even though it never saw a single photo of them during training.

Why This Matters

This isn't just about birds. This is about fairness and safety in AI.

• Healthcare: If a drug trial only included young men, an AI trained on that data might fail to predict side effects for elderly women. This method helps the AI "fill in the blanks" for those missing groups.
• Self-Driving Cars: If training data lacks images of cars in heavy snow, the car might crash when it finally sees snow. This method helps the car understand the "missing" scenario by learning from the "present" ones.

Summary

The paper is a guide on how to teach an AI to be smart about what it doesn't know. Instead of giving up when a chunk of data is missing, it uses the data that is there to mathematically reconstruct the missing piece, ensuring the AI works safely and accurately in the real world. It turns a "blind spot" into a "calculated guess" that is actually quite accurate.

Technical Summary

1. Problem Statement

The paper addresses a specific and challenging variant of Unsupervised Domain Adaptation (UDA).

• Standard Setting: Typically, UDA assumes a labeled source domain and an unlabeled target domain with distribution shifts.
• The Novel Challenge: The authors consider a setting where the source domain suffers from structured missingness. Specifically, the data is defined by a binary label $Y$ and a binary background/environment variable $A$. In the source domain, one specific subpopulation defined by the combination $(Y=1, A=1)$ is completely unobservable (probability is zero), while the other three subpopulations $(0,0), (0,1), (1,0)$ are present.
• Target Domain: The target domain contains all four subpopulations, but the labels $Y$ are unobserved.
• Real-world Motivation: This scenario arises in datasets like Waterbirds (where waterbirds on water backgrounds are rare/absent in training) or healthcare (where specific disease-demographic combinations are missing due to inclusion criteria). Naively applying standard UDA methods leads to biased estimates and poor performance on the unobserved subpopulation in the target.

2. Methodology

The proposed framework relies on Structured Conditional Invariance and Distribution Matching to recover the target prediction probabilities.

2.1. Core Assumptions

• Structured Conditional Invariance: The conditional distribution of features $X$ given the label and background remains invariant across domains:
  $$p(X | Y, A, R=1) = p(X | Y, A, R=0) = p(X | Y, A)$$
  This is a stronger assumption than standard label shift, as it conditions on both $Y$ and $A$.
• Structured Missingness: In the source, $pr(Y=1, A=1 | R=1) = 0$.

2.2. Theoretical Derivation

The authors derive closed-form expressions for the target predictive probabilities:

• $\eta_1(x) = pr(Y=1 | X=x, A=1, R=0)$
• $\eta_0(x) = pr(Y=1 | X=x, A=0, R=0)$
• $\eta(x) = \eta_1(x)\tau_0(x) + \eta_0(x)(1-\tau_0(x))$ (Overall probability)

Crucially, they show that $\eta_1(x)$ (the probability for the missing source group) can be recovered using observable quantities from the source and target, specifically by estimating the subpopulation proportions ($\beta$) in the target domain.

2.3. Estimation via Distribution Matching

Since the target proportions are unknown, the authors propose a Distribution Matching method to estimate the parameter vector $\beta = (\beta_{10}, \beta_{00})^T$ (the proportions of $Y=1$ and $Y=0$ given $A=0$ in the target).

• Objective: Minimize the Kullback-Leibler (KL) divergence between the observed feature distribution in the target subgroup ($R=0, A=0$) and a mixture of the source subgroup distributions ($R=1, Y=1, A=0$ and $R=1, Y=0, A=0$).
• Optimization: The problem is reformulated to maximize a log-likelihood objective involving the source classifier $\xi_0(x)$ and the target mixture proportions, avoiding the need to model high-dimensional feature densities directly.
• Algorithm:
  1. Train classifiers on observable source subgroups to get $\xi(x)$ and $\xi_0(x)$.
  2. Estimate conditional probabilities $\tau(x)$ and $\kappa(x)$ using available data.
  3. Solve the distribution matching problem to estimate target proportions $\hat{\beta}$.
  4. Plug $\hat{\beta}$ into the derived closed-form equations to obtain $\hat{\eta}(x)$.

3. Key Contributions

1. New Problem Formulation: The paper formally defines and analyzes the UDA setting with a completely unobservable source subpopulation, a scenario often ignored by existing methods.
2. Theoretical Framework: They provide rigorous derivations showing that accurate target prediction is possible despite the missing data, provided the conditional invariance assumption holds.
3. Identifiability Analysis: The authors establish conditions under which the model parameters are identifiable, linking the problem to Positive-Unlabeled (PU) learning and Open Set Label Shift (OSLS).
4. Statistical Guarantees:
   • Consistency: The estimator for the subpopulation proportions is shown to be statistically consistent.
   • Error Bounds: They derive upper bounds on the parameter estimation error and the generalization error (prediction risk) of the resulting classifier, showing that the error depends on the convergence rate of the proportion estimator.
5. Practical Algorithm: A distribution matching algorithm (Algorithm 1) is proposed that avoids complex density estimation, making it scalable.

4. Experimental Results

The method was evaluated on both synthetic data and real-world datasets (Waterbirds and CelebA).

• Synthetic Data: The proposed method ($\hat{\eta}$) consistently outperformed two naive baselines:
  • Naive1: Blindly applying source probabilities (ignoring missingness).
  • Naive2: Assuming standard label shift without accounting for the missing subgroup.
  • The proposed method showed improved Accuracy and F1 scores, particularly as sample sizes increased.
• Real-World Data (Waterbirds):
  • The dataset was partitioned such that the $(Y=1, A=1)$ group was excluded from the source.
  • The proposed method significantly outperformed baselines in accuracy and F1 score.
  • Feature Extraction: Using ViT-16 features yielded better results than ResNet-18, especially for the naive baselines which suffered more from spurious correlations.
  • Robustness: The method remained effective even when the proportion of the "bridge" subgroup (e.g., $Y=0, A=1$) in the source varied, though performance degraded if multiple subgroups became too small.

5. Significance and Impact

• Addressing Data Bias: This work provides a solution for scenarios where data collection constraints lead to systematic gaps in training data (e.g., rare disease subgroups, specific environmental conditions).
• Beyond Spurious Correlation: By explicitly modeling the missing subpopulation structure, the method prevents the "collapse" phenomenon seen in adversarial domain adaptation, where the model incorrectly aligns the target's missing group to an existing source group.
• Fairness and Reliability: The approach ensures that predictions for the previously unobserved subpopulations in the target domain are not systematically biased, contributing to more robust and fair machine learning systems in critical fields like healthcare and ecology.
• Theoretical Foundation: It establishes the first rigorous characterization of model adaptation under structured subpopulation absence, offering a blueprint for handling similar "missingness" problems in other statistical learning contexts.