Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

Technical Summary: Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

Problem Statement
Current evaluation metrics for prompt-guided generative models (e.g., text-to-image, LLMs) primarily focus on fidelity and prompt alignment, often neglecting the assessment of output diversity. Existing diversity metrics, such as the Vendi score and R´enyi Kernel Entropy (RKE), were designed for unconditional generative models. When applied to conditional models, these metrics conflate two distinct sources of variability: prompt-induced diversity (variation arising from differences in input prompts) and model-induced diversity (randomness or variability in outputs generated for similar or identical prompts). This conflation biases diversity comparisons, as a model may appear diverse simply because it was evaluated on a broad range of prompts, rather than because it generates varied outputs for a given prompt.

Methodology
The authors propose Conditional-Vendi and Conditional-RKE, two prompt-aware diversity metrics derived from the conditional entropy of positive semidefinite (PSD) matrices.

1. Theoretical Foundation: The method extends the matrix-based entropy framework of Giraldo et al. (2014). For a set of generated outputs $X$ and corresponding prompts $T$, the authors define kernel matrices $K_X$ and $K_T$. The joint entropy is defined via the von Neumann entropy of the trace-normalized Hadamard product of these kernels. The conditional entropy $H(X|T)$ is then defined as $H(X, T) - H(T)$.
2. Metric Definitions:
   • Conditional-Vendi: Defined as $\exp(H(\frac{1}{n}(K_X \odot K_T)) - H(\frac{1}{n}K_T))$. This metric isolates model-induced diversity by subtracting the entropy attributable to the prompts.
   • Conditional-RKE: Defined using order-2 R´enyi entropy, resulting in a closed-form expression involving Frobenius norms: $\frac{\|K_T\|_F^2}{\|K_X \odot K_T\|_F^2}$.
3. Decomposition Property: Both metrics satisfy a product decomposition property where the unconditional score (Vendi or RKE) equals the product of the Conditional score (model-induced diversity) and an "Information" score (mutual information between prompt and output).
4. Statistical Convergence and Truncation:
   • The authors prove that Conditional-RKE converges at a rate of $O(1/\sqrt{n})$.
   • Direct estimation of Conditional-Vendi suffers from high sample complexity due to dimension dependence ($d_X d_T$). To address this, the authors introduce a Truncated Conditional-Vendi score, which approximates the entropy using only the top-$t$ eigenvalues. This truncated estimator achieves a convergence rate independent of the feature dimensions $d_X$ and $d_T$, enabling scalable computation.
5. Guidance Application: The authors extend the Conditional-Vendi score to guide text-conditioned latent diffusion models. By adding a gradient ascent step on the Truncated Conditional-Vendi score during the sampling process, the model is encouraged to increase prompt-aware entropy, thereby generating more diverse samples.

Key Contributions

• Introduction of Conditional-Vendi and Conditional-RKE as prompt-aware diversity metrics that disentangle model-induced variability from prompt-induced variability.
• Theoretical analysis establishing finite-sample convergence guarantees for both scores, including a truncated approximation for Conditional-Vendi to mitigate the curse of dimensionality.
• Demonstration that these scores can guide diffusion models to produce more diverse outputs while maintaining alignment.
• Empirical validation across text-to-image, text-to-video, image-captioning, and LLM tasks, showing that the conditional scores recover ground-truth diversity orderings better than unconditional metrics.

Results

• Convergence: Experiments on the MS-COCO dataset show that Conditional-RKE and Truncated Conditional-Vendi stabilize with moderate sample sizes (e.g., 15,000 samples), whereas the non-truncated Conditional-Vendi fails to converge within computationally feasible limits.
• Diversity Disentanglement: In controlled experiments with Stable Diffusion XL, unconditional Vendi scores increased when prompts specified different animal types (prompt-induced diversity). However, Conditional-Vendi remained low in this scenario, correctly identifying that the model’s internal diversity was constrained by the specific prompts. Conversely, when prompts were vague (e.g., "an animal"), Conditional-Vendi increased, reflecting the model’s ability to generate varied breeds.
• Model Comparison: The metrics successfully distinguished between models. For instance, in text-to-video evaluation, Open-Sora exhibited higher Conditional-Vendi scores than VideoCrafter, aligning with visual observations of greater diversity.
• LLM Evaluation: For Llama 2, Gemma 3, and Phi 4 Mini, both Conditional-Vendi and Conditional-RKE scores increased with higher temperature parameters, confirming that the metrics capture the expected increase in output diversity.
• Guidance: Applying Conditional-Vendi guidance to Stable Diffusion XL and PixArt-Σ improved sample diversity (measured by Vendi and in-batch similarity) while maintaining text-image alignment (CLIPScore), outperforming standard Vendi guidance.

Significance
The paper argues that existing diversity metrics are insufficient for conditional generative models because they fail to account for the conditioning signal. By isolating model-induced diversity, Conditional-Vendi and Conditional-RKE provide a more accurate assessment of a model’s generative capability. The authors highlight that these scores not only serve as evaluation tools but can also be used as regularizers or guidance mechanisms to improve the diversity of generated content. The work addresses a critical gap in the evaluation of prompt-guided AI systems, offering a theoretically grounded and computationally tractable solution for assessing and enhancing diversity in conditional generation.