Technical Summary: Conf-Gen: Conformal Uncertainty Quantification for Generative Models

Problem Statement

Recent breakthroughs in artificial intelligence, particularly in unsupervised generative models like Large Language Models (LLMs) and image generators, have shifted the landscape away from traditional supervised learning. However, uncertainty quantification (UQ) remains a critical challenge for these generative systems, hindering their deployment in high-stakes domains such as healthcare and scientific discovery.

While Conformal Prediction (CP) and Conformal Risk Control (CRC) provide rigorous, distribution-free frameworks for UQ in supervised learning, they are not directly compatible with generative tasks. Existing attempts to apply conformal ideas to generative models are often highly task-specific, rely on restrictive assumptions (such as strict monotonicity of utility functions), and are limited to set-valued outputs. Furthermore, current implementations often require ground truth labels and callable utility functions, which are not always available in open-ended generative scenarios.

Methodology: Conf-Gen Framework

The authors propose Conf-Gen (Conformal Generation), a general framework that adapts CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies previous task-specific attempts and extends conformal methodology to new domains, including sequences and structured outputs beyond simple sets.

Core Components

The framework operates on a calibration dataset $G_{:n} = \{(X^{(i)}, Y^{(i)}, Y^{(i)}_{GT})\}_{i=1}^n$, where $X$ is the input, $Y$ is a sequence of generations, and $Y_{GT}$ is optional ground truth. It consists of three main components:

1. Selection Functions ($C_\lambda$): A family of potentially stochastic functions $C_\lambda: \mathcal{X} \times \mathcal{Y}^* \to \mathcal{S}$, indexed by a parameter $\lambda$. Unlike traditional CP which outputs sets, $\mathcal{S}$ can be sequences ($\mathcal{Y}^*$) or other structured outputs. Intuitively, larger $\lambda$ values produce more conservative outputs (e.g., shorter sequences or stricter subsets) that achieve higher admissibility.
2. Admissibility Function ($A$): A function $A: \mathcal{X} \times \mathcal{S} \times \mathcal{Y}_{GT} \to [0, \infty)$ that generalizes the utility function in CRC. It measures the quality of the selected output. Crucially, $A$ does not need to be a callable function; it can be defined by human evaluation or procedures where ground truth is undefined.
3. Calibration: The method finds a threshold $\hat{\lambda}$ such that the average admissibility on the calibration dataset meets a target $\gamma$. Formally, $\hat{\lambda} = \inf \{ \lambda \in \Lambda : \bar{A}_n(\lambda) \ge \frac{n+1}{n}\gamma \}$.

Theoretical Contributions

Conf-Gen introduces a $\gamma$-sensible instance definition that relaxes the assumptions of CRC:

• Relaxed Monotonicity: Instead of requiring the admissibility function to be non-decreasing almost surely (a strict requirement in CRC), Conf-Gen only requires that the map $\lambda \mapsto \mathbb{E}[A^{(n+1)}(\lambda) \mid \hat{\lambda}, \lambda^*]$ is non-decreasing in conditional expectation. This allows for scenarios where the relationship between $\lambda$ and admissibility is not strictly monotonic in every realization but holds in expectation.
• Generalized Output Spaces: The framework supports outputs beyond sets (e.g., sequences), enabling applications like selecting a subsequence of answers or a subset of actions.
• No Ground Truth Requirement: The framework does not strictly require a well-defined ground truth $Y_{GT}$ for every instance, accommodating open-ended tasks where admissibility is determined by human evaluation or semantic consistency.

Computational Efficiency

The paper identifies design patterns to make Conf-Gen computationally tractable:

• Score-Based Selection: By defining $C_\lambda$ based on scores assigned to individual elements (e.g., selecting elements where $S_\downarrow(x, y_t) \le \lambda$), the image of the selection function becomes finite, allowing efficient calibration even with infinite $\Lambda$.
• Instance-Level Decomposition: If the admissibility function can be decomposed into an aggregation of instance-level scores (e.g., $A = \text{agg}(A'_1, \dots, A'_T)$), calibration can be performed by evaluating $A'$ on all generated elements once, decoupling the evaluation from the choice of $\lambda$.
• Partial Generation: In many cases, the system can stop generating a sequence early once the selection criteria are met, saving computational resources.

Key Contributions

1. Framework: Proposes Conf-Gen, a rigorous UQ framework for generative models that generalizes CRC to sequences and structured outputs.
2. Theory: Relaxes the monotonicity assumption of CRC from an almost-sure condition to a conditional expectation condition, broadening the applicability to tasks where strict monotonicity does not hold.
3. Software: Provides a flexible Python package (conf-gen) supporting efficient calibration and inference across various configurations.
4. Empirical Validation: Demonstrates the framework's versatility across novel tasks, outperforming state-of-the-art baselines in LLM question answering and establishing the first conformal guarantees for several new application domains.

Experimental Results

The authors validate Conf-Gen on diverse tasks, showing that it satisfies conformal guarantees (empirical admissibility $\ge \gamma$) while maintaining practical utility (shorter sequences or higher quality outputs compared to baselines).

• Open-Domain Question Answering: On the TriviaQA dataset, Conf-Gen produces shorter answer sequences than the baseline (Quach et al., 2024) while maintaining the same coverage guarantee, reducing the number of required LLM calls.
• Non-Memorized Image Generation: Applied to Stable Diffusion, Conf-Gen selects images that are verified by human evaluators as "non-memorized" (i.e., not copies of training data) with a guaranteed probability $\gamma$. This addresses copyright risks associated with generative models.
• Conversational AI: The framework ensures that a chatbot asks enough clarifying questions to resolve ambiguity before answering. The results show a trade-off where higher $\gamma$ leads to more clarifications, but the system does not default to asking the maximum possible number.
• Agentic AI: For web-based tasks (WebVoyager dataset), Conf-Gen guarantees that a selected sequence of agent trajectories contains at least one successful solution with probability $\gamma$.
• Random Forests: The framework is applied to supervised learning by selecting a subset of trees from a random forest that guarantees a minimum number of correct predictions, demonstrating the method's generality beyond generative models.

Significance and Claims

The paper claims that Conf-Gen represents a significant step towards improved uncertainty quantification in generative models. By relaxing the theoretical assumptions of CRC, it enables the application of conformal guarantees to a wider array of generative tasks, including those without ground truth or strict monotonicity.

The authors emphasize that their work establishes a foundation for novel conformal applications, such as ensuring generated images are not memorized and verifying that AI agents have asked sufficient clarifying questions. They posit that these improvements could increase the reliability of AI systems, facilitating their deployment in safety-critical applications. The paper maintains a modest tone regarding societal impact, acknowledging that while UQ can improve reliability, it could also potentially enhance malicious applications, though the net impact is expected to be positive.

The work does not claim to solve all UQ challenges but rather provides a flexible, theoretically grounded framework that unifies existing approaches and opens new avenues for rigorous uncertainty control in the era of generative AI.