Technical Summary: Semi-Supervised Conditional Generative Learning through Stochastic Interpolation and Sufficient Representations

1. Problem Statement

Conditional generative modeling aims to learn the underlying conditional distribution $P_{X|Y}$ of high-dimensional data $X \in \mathbb{R}^d$ given a condition $Y \in \mathbb{R}^q$. While standard regression provides point estimates, learning the full distribution is essential for uncertainty quantification and capturing complex structures like multimodality. However, in semi-supervised settings, labeled pairs $(X, Y)$ are scarce while unlabeled samples of $X$ are abundant.

Direct approaches that inject $Y$ into the high-dimensional generation process suffer from poor sample efficiency. They require substantial labeled data to learn the complex ambient conditional distribution, failing to leverage the abundance of unlabeled data. Existing representation-based methods (e.g., VAEs, latent diffusion) often rely on autoencoder structures where theoretical guarantees for conditional generation remain unexplored, particularly regarding how information loss in imperfect representations propagates to generative error.

2. Methodology: RepG Framework

The authors propose RepG, a two-stage framework that factorizes the conditional generation process using sufficient representations and conditional stochastic interpolation.

2.1 Structural Factorization via Sufficient Representations

RepG introduces a low-dimensional latent representation $R(X) \in \mathbb{R}^k$ (where $k \ll d$) designed to capture all information linking $X$ and $Y$. The method relies on the conditional independence assumption $X \perp\!\!\!\perp Y \mid R$, establishing the Markov chain $Y \to R \to X$. This allows the generation process to be decomposed into:

1. Latent Sampling (Label-Dependent): Generating the sufficient representation $r$ from the condition $y$ ($P_{R|Y}$). This is a low-dimensional task estimated using scarce labeled data.
2. Reconstruction (Label-Free): Generating the high-dimensional target $x$ from the representation $r$ ($P_{X|R}$). Since $P_{X|R}$ is independent of $Y$, this stage can be estimated using the pooled dataset of both labeled and abundant unlabeled data.

2.2 Implementation via Conditional Stochastic Interpolation

To implement the two stages, RepG utilizes conditional stochastic interpolation (Albergo et al., 2025; Huang et al., 2023).

• Interpolation Path: A path $Z_t = a_t Z_0 + b_t Z_1$ connects a reference Gaussian distribution ($Z_0$) to the target distribution ($Z_1$) over $t \in [0, 1]$.
• Drift Estimation: The generative task reduces to estimating the drift field $v^*_F$ of a Stochastic Differential Equation (SDE). This drift is decomposed into a velocity field and a score function.
• Neural Estimators: The velocity fields for both stages are parameterized by deep ReLU feedforward neural networks (FNNs).
  • Stage 1: The velocity field $u$ is trained on labeled pairs $(R_i, Y_i)$ to minimize the empirical risk of the interpolation path.
  • Stage 2: The velocity field $v$ is trained on the augmented dataset $(X_m, R_m)$ (combining labeled and unlabeled data) to learn the reconstruction mapping.
• Generation: Given a condition $y$, the model first simulates the SDE to generate a latent representation $\hat{r}$, then simulates a second SDE conditioned on $\hat{r}$ to generate the final sample $\hat{x}$.

2.3 Representation Learning

To construct the sufficient representation $R(\cdot)$, the authors employ a Deep Dimension Reduction (DDR) framework (Huang et al., 2024). This involves learning a mapping that maximizes the distance covariance between $R(X)$ and $Y$ while regularizing $R(X)$ to follow a standard Gaussian distribution, ensuring $X \perp\!\!\!\perp Y \mid R(X)$ is approximately satisfied.

3. Key Contributions

3.1 Theoretical Error Decomposition

The paper establishes a rigorous decomposition of the expected Kullback-Leibler (KL) divergence between the true conditional distribution and the RepG estimator. The total error comprises three components:

1. Structural Bias: Quantified by the Conditional Mutual Information (CMI), $I(X; Y \mid R)$. This measures the information loss if the representation is not perfectly sufficient.
2. Low-Dimensional Estimation Error: The error in learning $P_{R|Y}$ using labeled data.
3. High-Dimensional Estimation Error: The error in learning $P_{X|R}$ using the pooled dataset.

This decomposition proves that strict conditional independence is not required; a small CMI combined with controlled estimation errors suffices for low total generative error.

3.2 Non-Asymptotic Convergence Rates

Under smoothness assumptions (local Hölder classes) and sub-exponential moment bounds, the authors derive convergence rates for the FNN estimators:

• The first-stage error scales with the sample size $n$ and dimension $k+q$.
• The second-stage error scales with the pooled sample size $n+N$ and dimension $d+k$.
• Crucial Result: In the regime where unlabeled data is abundant ($N \gg n$), the overall convergence rate is dominated by the low-dimensional supervised component ($n^{-2\alpha/(2\alpha+q+k+1)}$). This demonstrates that RepG effectively mitigates the curse of dimensionality by confining the statistical burden of labeled data to the reduced dimension $k$.

3.3 Minimax Lower Bound

The paper provides a minimax lower bound for direct conditional generation (without latent factorization), showing the error scales with the full ambient dimension $d+q+1$. By comparing this with the upper bound of RepG, the authors confirm that the representation-based factorization significantly improves sample complexity when $k \ll d$.

4. Experimental Results

4.1 Simulation Studies

• Discrete Conditioning: RepG was tested on synthetic datasets (Two Moons, Concentric Circles, 2D/3D Gaussian Mixtures). The method successfully learned sufficient representations that preserved class separability and reconstructed complex geometries without mode collapse.
• Continuous Conditioning: In scenarios involving heteroscedasticity, skewness, and crossing distributions, RepG outperformed Gaussian Process Regression (GPR), Mixture Density Networks (MDN), and Quantile Random Forests (QRF) in terms of joint distributional accuracy (Energy Score). While marginal interval widths were comparable, RepG better captured multivariate dependence structures.

4.2 Real-World Application (MNIST)

RepG was applied to MNIST digit generation with fixed labeled sets ($n_\ell \in \{1000, 3000\}$) and varying unlabeled sets ($n_u$).

• Performance: RepG consistently outperformed a baseline direct conditional model.
• Scalability: As the number of unlabeled samples increased, RepG showed monotonic accuracy gains in a downstream classification task, stabilizing near 87-90% accuracy. In contrast, the baseline's performance remained flat, as it could not utilize the unlabeled data.
• Visual Quality: Generated images transitioned from fragmented artifacts to coherent structures as the unlabeled sample size increased.

5. Significance and Claims

The paper claims that RepG offers a principled solution to the semi-supervised conditional generation problem by:

1. Decoupling Complexity: Separating the difficult high-dimensional generation into a low-dimensional supervised task and a label-free reconstruction task.
2. Theoretical Guarantees: Providing the first rigorous error decomposition and convergence rates for representation-based conditional generation, explicitly linking structural bias (CMI) to statistical risk.
3. Sample Efficiency: Demonstrating that by confining the supervised learning burden to the intrinsic dimension of the latent space, the method bypasses the curse of dimensionality inherent in direct ambient-space modeling.
4. Practical Utility: Validating that even with approximate sufficiency, the framework remains effective, as shown by empirical results on both synthetic and real-world data.

The authors conclude that this approach provides a strong theoretical basis for allocating limited labeled data in semi-supervised settings and effectively leverages abundant unlabeled data to learn high-dimensional data structures.