Technical Summary: PromptSplit

Problem Statement

Prompt-guided generative AI models (spanning vision and language) have proliferated, yet systematic methods to compare their behaviors relative to specific input prompts remain limited. Existing evaluation metrics, such as FID, Inception Score, and CLIP-Score, provide aggregate quality indicators that compress model behavior into single scalar values. These metrics often obscure prompt-dependent discrepancies. For instance, two models might generate high-quality images for a given prompt but diverge significantly in the content or style of the output based on specific prompt categories (e.g., one model consistently generating male figures while another generates female figures for the prompt "a person"). Current spectral and distributional comparison methods typically operate in a prompt-free setting, marginalizing over prompts and failing to isolate the specific input categories responsible for behavioral divergence.

Methodology

The authors propose PromptSplit, a prompt-aware, unsupervised spectral framework designed to detect and analyze prompt-dependent disagreement between two generative models.

1. Joint Prompt-Output Representation

PromptSplit constructs a joint representation by forming tensor-product embeddings of the prompt features ($\phi_T$) and output features ($\phi_X$ or $\phi_Y$). For a prompt $t$ and output $x$, the joint feature is defined as:
$$\phi^\otimes(t, x) = \phi_T(t) \otimes \phi_X(x)$$
This induces a product kernel $k^\otimes([t, x], [t', x']) = k_T(t, t') \cdot k_X(x, x')$, capturing multiplicative interactions between the input and output spaces.

2. Kernel Covariance Difference

For two datasets $D_X$ (from model $X$) and $D_Y$ (from model $Y$), the method computes empirical joint kernel covariance operators $\hat{C}_{T \otimes X}$ and $\hat{C}_{T \otimes Y}$. The core of the analysis is the covariance-difference operator:
$$\hat{\Lambda}_{X,Y|T} = \hat{C}_{T \otimes X} - \eta \hat{C}_{T \otimes Y}$$
where $\eta$ is a hyperparameter. The principal eigenvectors of this operator reveal the directions in the joint space where the two models exhibit the most significant behavioral differences.

3. Kernel Trick and Spectral Analysis

Direct eigendecomposition of the high-dimensional covariance operator is computationally infeasible. The authors apply the kernel trick to formulate a block kernel matrix $K_{X,\eta Y|T}$ of size $(n+m) \times (n+m)$, which shares the same non-zero eigenvalues as the high-dimensional operator. The eigenvectors of this block matrix correspond to the disagreement modes in the joint space.

4. Scalable Random Projection

To handle large-scale datasets (where $n, m$ exceed tens of thousands), PromptSplit employs a random-projection approximation. The joint tensor features are projected to a lower dimension $r$ using Gaussian random matrices. This reduces the computational complexity of the spectral decomposition from $O((n+m)^3)$ to $O((n+m)r^2 + r^3)$.

• Theoretical Guarantee: The paper proves that the expected deviation of the eigenspace estimate from the full-dimensional result is bounded by $O(1/r^2)$, ensuring that the approximation preserves the disagreement directions with high probability.

Key Contributions

1. Prompt-Level Comparison Formulation: The work formulates model comparison as a joint prompt–output spectral problem, moving beyond aggregate metrics to identify specific prompt categories that cause divergent model behaviors.
2. PromptSplit Framework: Introduction of a kernel-based framework that couples prompts and outputs via tensor-product embeddings to analyze the eigenspectrum of joint kernel covariance differences.
3. Scalable Approximation: Development of a random-projection method that enables the analysis of large datasets with a theoretical error bound of $O(1/r^2)$, making the method practical for real-world generative model evaluation.
4. Interpretability: The method provides interpretable "modes" of disagreement, explicitly linking specific prompt clusters to the nature of the output divergence.

Experimental Results

The authors evaluated PromptSplit across text-to-image (T2I) and text-to-text (LLM) settings:

• Synthetic Validation: In controlled settings (e.g., MNIST-M with known color/grayscale prompt mismatches), PromptSplit successfully identified the ground-truth prompt categories responsible for the disagreement.
• Text-to-Image Models: Comparisons between Stable Diffusion XL, PixArt-Σ, and Kandinsky revealed prompt-dependent divergences in style, composition, and alignment. For example, the method identified specific occupation-based prompts (e.g., "nurse," "carpenter") where models produced significantly different visual representations.
• Large Language Models (LLMs): Comparisons between Qwen 3 and Gemma 3 on the NQ-Open dataset identified distinct clusters of questions (e.g., regarding US presidents or actors) where the models generated divergent answers.
• Guidance Application: The authors demonstrated using PromptSplit to guide the latent diffusion process, successfully aligning the distribution of generated images with a reference dataset (e.g., specific painting styles) by incorporating the disagreement gradient into the sampling process.

Significance and Claims

The paper claims that PromptSplit offers a principled, interpretable tool for detecting where and how generative models disagree, addressing a gap left by aggregate quality metrics. By explicitly modeling the joint prompt-output space, it isolates the specific input categories driving behavioral differences.

The authors position PromptSplit as an embedding-based, second-order spectral method. They modestly note that while the design enables scalability and interpretability, it does not aim to characterize all higher-order aspects of conditional generative behavior. The work is presented as a step toward more granular, prompt-conditioned evaluation of the growing ecosystem of generative AI models. The implementation is made publicly available to facilitate further research in this domain.