Technical Summary: Bayes-Sufficient Representations in Supervised Learning

Problem Statement

Representation learning is frequently characterized as the task of preserving input information relevant for prediction while discarding nuisance variation. However, this definition leaves a fundamental ambiguity: relevance is not an intrinsic property of the input distribution alone but is jointly determined by the data-generating law and the specific loss function used for evaluation.

For instance, in binary classification, two inputs with conditional probabilities $P(Y=1|X=x)$ of 0.55 and 0.95 require the same Bayes-optimal class decision under zero-one loss. Conversely, under log loss or other calibrated probabilistic losses, these inputs are distinct and require different probability estimates. Consequently, the "relevant" information varies depending on the loss. The paper addresses the need to formalize this loss-dependence, distinguishing between representations that are merely sufficient for optimal prediction and those that are minimal (containing no more information than strictly required).

Methodology and Theoretical Framework

The paper establishes a decision-theoretic framework for Bayes-sufficient representations.

1. Definitions

• Bayes Sufficiency: A representation $H = h(X)$ is Bayes-sufficient for a joint distribution $P$ and loss $\ell$ if there exists a prediction head $c$ such that the composite predictor $c \circ h$ implements a Bayes-optimal action rule almost surely.
• Bayes Quotient: In the common case where the Bayes action is almost surely unique, the paper defines a canonical object called the Bayes quotient. This quotient identifies all inputs $x$ that require the same Bayes-optimal action. Formally, it is the sigma-algebra generated by the Bayes action $a^*_{\ell, P}(X)$.
• Bayes Minimality: A representation is Bayes-minimal if it is informationally equivalent to the Bayes quotient (i.e., it generates the same sigma-algebra).

2. Characterization

The paper proves a factorization characterization: a representation is Bayes-sufficient if and only if at least one Bayes-optimal predictor is measurable with respect to that representation.

• Sufficiency: The representation's sigma-algebra must contain the Bayes quotient ($\sigma(Q) \subseteq \sigma(H)$).
• Minimality: The representation's sigma-algebra must equal the Bayes quotient ($\sigma(H) = \sigma(Q)$).

3. Connection to Property Elicitation

The framework links loss functions to specific conditional properties they elicit. Different losses induce different Bayes quotients:

• Zero-one loss: Elicits the Bayes class (coarsest quotient).
• Squared loss: Elicits the conditional mean.
• Binary Brier loss: Elicits the conditional probability.
• Log loss / Strictly proper scoring rules: Elicit the full conditional distribution (finest quotient).

Thus, for a fixed distribution, the loss determines the minimal information required for optimal prediction.

Key Contributions

1. Decision-Theoretic Definition: The paper provides a formal definition of Bayes sufficiency for supervised representations, proving that sufficiency is equivalent to the factorization of a Bayes-optimal predictor through the representation.
2. The Bayes Quotient: It identifies the Bayes quotient as the canonical, loss-dependent object defining relevant information. It distinguishes between sufficiency (retaining the quotient) and minimality (retaining only the quotient), clarifying that a representation can be sufficient yet non-minimal by retaining extra, non-required information.
3. Property Elicitation Link: It connects representation targets to property elicitation theory, demonstrating how standard supervised losses induce specific representation targets (e.g., class vs. probability vs. full distribution).
4. Handling Non-Unique Actions: The framework extends to cases where Bayes actions are not unique, defining sufficiency based on the existence of a common optimal action on representation fibers rather than strict equality of action sets.

Experimental Results

The paper validates the theory through three distinct experimental settings:

1. Controlled Finite Experiments

Using a synthetic latent law with known analytical quotients, the authors compared hand-defined representations:

• Result: A coarse representation ($S$) was sufficient and minimal for zero-one classification but insufficient for log loss. A finer representation ($S, T$) was sufficient and minimal for log loss but non-minimal for classification. A representation with irrelevant noise ($S, T, U$) was sufficient but non-minimal.
• Finding: The experiments confirmed that sufficiency is loss-dependent and that representations can retain non-required information while maintaining optimality.

2. Learned Neural Bottleneck Experiments

Neural encoders were trained on continuous observations derived from the synthetic latent law under different objectives (classification vs. probability prediction) and capacities (tight vs. wide bottlenecks).

• Result: Encoders trained for classification (coarse target) recovered the coarse quotient ($S$) perfectly but failed to recover finer variables ($T$, $\eta$). Encoders trained for probability prediction (fine target) or with wider capacities recovered both coarse and fine variables.
• Finding: The objective function and bottleneck capacity govern which quotient variables remain recoverable. Probability training encourages preservation of the finer quotient, while tight classification training approaches the coarse quotient.

3. Real-Data Taxonomic Refinement (iNaturalist)

Using the iNaturalist dataset, the authors utilized the biological taxonomy (Species $\to$ Genus $\to$ Family) as a structured analogue of quotient refinement.

• Result: Representations trained on fine-grained species labels retained significantly more species-level information than those trained on coarse family labels. Conversely, family-trained representations (especially with bottleneck constraints) discarded much of the species-level information while maintaining family-level accuracy.
• Finding: This demonstrated the "sufficient refinement" phenomenon in a real-world setting: a representation can be sufficient for a coarse task while retaining (or discarding) finer, non-required distinctions depending on the training objective and capacity.

Significance and Claims

The paper claims to formalize the often-conflated notions of "task-relevant" information and "minimal" information in representation learning. Its primary significance lies in establishing that:

• Relevance is Loss-Relative: There is no single "correct" representation for a dataset; the minimal predictive information is determined jointly by the distribution and the loss function.
• Sufficiency $\neq$ Minimality: A representation can be optimal for a task (sufficient) while containing extraneous information (non-minimal). The Bayes quotient provides the precise boundary between these two states.
• Decision-Theoretic Grounding: By framing representation learning through the lens of statistical decision theory and property elicitation, the work provides a rigorous basis for understanding why different losses (e.g., classification vs. calibration) require different levels of information retention.

The authors note limitations, acknowledging that the theory is population-level and does not yet fully address finite-sample estimation, optimization dynamics, or the complexities of non-unique Bayes actions in practical deep learning scenarios. However, the framework offers a clear theoretical target for analyzing what information is strictly necessary for Bayes-optimal prediction under a specified loss.