🤖 machine learning

Loss-Shift Transfer via Bayes Quotients

This paper introduces the concept of "loss shift," demonstrating that even with a fixed data distribution, a representation optimal for a coarser loss may be insufficient for a strictly finer loss, a phenomenon formalized via Bayes quotients where the resulting performance gap is quantified by the conditional information about the target variable discarded by the representation.

Original authors: Vasileios Sevetlidis

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vasileios Sevetlidis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a student to be a master of a specific subject. Usually, when we talk about "transfer learning" (using what you learned in one situation to help in another), we assume the problem changes. Maybe the student learned math in a quiet classroom but now has to solve math problems in a noisy cafeteria. That's a change in environment.

This paper introduces a different, often overlooked problem called Loss Shift. Here, the environment (the data) stays exactly the same, but the rules of the game change.

The Core Idea: The "Map" vs. The "GPS"

Think of a Loss Function as the specific goal you give your student.

Goal A (Classification/Accuracy): "Just tell me if the car is going left or right."
Goal B (Probabilistic Prediction/Log Loss): "Tell me exactly how likely it is to go left (e.g., 51% vs. 99%)."

The paper argues that a representation (a mental map or a compressed summary of the data) that is perfect for Goal A might be useless for Goal B, even if the data hasn't changed at all.

The Analogy: The "Good Enough" Summary

Imagine you are describing a room to a friend.

Scenario 1 (The Source Task): Your friend asks, "Is the room dark or light?"
- You look at the room and create a mental summary: "It's Light."
- You throw away all the extra details. You don't remember the exact shade of the curtains or the brightness of the lamp. You just know: Light.
- This summary is perfect for the question "Dark or Light?" It is the most efficient, "minimal" summary possible.
Scenario 2 (The Target Task): Now, your friend asks, "What is the exact brightness level in lumens?"
- You try to use your old summary ("It's Light").
- Problem: You threw away the details! You can't answer the new question because your summary collapsed all the different shades of "light" into a single bucket.
- Even though you are looking at the exact same room (the data distribution is fixed), your old summary is now insufficient.

The Paper's Technical Terms, Translated

Bayes Quotient: This is the paper's fancy way of saying "The specific details that matter for the current goal."
- For "Dark or Light," the quotient is just the direction of the light.
- For "Exact Brightness," the quotient is the precise measurement.
- The paper shows that the second quotient is a "finer" version of the first. It contains everything the first one has, plus more.
Strict Refinement: This is when the new goal requires more detail than the old one.
- If your old summary (the frozen representation) only kept the "Dark/Light" info, it is strictly insufficient for the "Exact Brightness" goal. You can't get the extra detail back once you've thrown it away.
The "Frozen" Representation: This is like taking a photo of your summary and freezing it. You can't go back and re-look at the room to get more details. You are stuck with the photo.
- The paper proves that if you freeze a summary that was optimized for a simple goal (like accuracy), you will inevitably lose points when you try to use it for a complex goal (like probability), even if you have a super-smart new teacher (the "downstream head") trying to fix it.

The "Math" of the Mistake

The paper provides a precise formula for this failure. It says the "penalty" you pay for using the wrong summary is exactly equal to the information you threw away.

If you threw away the difference between "51% chance" and "99% chance," the math measures exactly how much confusion that causes.
The more you compress the data to fit the simple goal, the more "information loss" you suffer when the goal gets harder.

What the Experiments Showed

The authors tested this in four ways, all using the same data but changing the goal:

The Controlled Test: They built a fake world where they knew exactly what information was thrown away. The results matched their math perfectly: The "Light/Dark" summary got a perfect score on the simple test but failed the hard test by a predictable amount.
The Learned Bottleneck: They trained a computer to compress data for a simple task (like a bottleneck in a pipe). When they froze that compression and tried to do a harder task, the performance dropped, exactly as predicted.
The Image Test (dSprites): They used images of shapes. One task was "Is the shape on the left or right?" (Simple). The other was "How likely is it to be on the left?" (Hard). A model trained for the simple task kept the "Left/Right" info but forgot the "Likelihood" info.
The Real-World Test (CIFAR-10H): They used real photos where humans gave "soft" answers (e.g., "It's 70% a cat, 30% a dog").
- A model trained to just guess the "Most Likely" answer (Hard Label) was great at guessing the top choice.
- But when asked to guess the full distribution (Soft Label), it struggled.
- A model trained specifically for the full distribution from the start did much better, proving that the "Hard Label" summary had discarded the nuance needed for the harder task.

The Bottom Line

The main takeaway is simple: A representation is only as good as the question it was built to answer.

If you train a system to just "get the right answer" (Accuracy), it will learn to ignore the subtle details that distinguish a "maybe" from a "definitely." If you later ask it to give a "confidence score" (Log Loss), it will fail, not because the data changed, but because it threw away the very details you now need.

This is a new kind of failure in AI, distinct from the usual problem of "the data changed." Here, the data stays the same, but the definition of success changes, and the old map no longer fits the new territory.

Technical Summary: Loss-Shift Transfer via Bayes Quotients

Problem Statement
Transfer learning is traditionally analyzed through the lens of distribution shift, where the joint law $P(X, Y)$ changes between source and target settings. This paper identifies an orthogonal failure mode termed loss shift. In this setting, the joint distribution $P(X, Y)$ remains fixed, but the predictive loss function changes. The core problem is that a representation optimized for a "coarse" source loss (e.g., zero-one classification accuracy) may discard information that is irrelevant for the source objective but essential for a "finer" target loss (e.g., probabilistic prediction under log loss). Once a representation is frozen, downstream heads cannot recover the information discarded during the source training phase, leading to a fundamental transfer failure even when the data distribution is identical.

Methodology and Theoretical Framework
The paper formalizes this phenomenon using Bayes quotients, a concept drawn from decision theory and property elicitation.

Bayes Quotients and Sufficiency: For a fixed distribution $P$ and a loss $\ell$ , the Bayes quotient $q_{\ell, P}(X)$ identifies the equivalence classes of inputs that yield the same Bayes-optimal action. A representation $H = h(X)$ is Bayes-sufficient for $(P, \ell)$ if it preserves this quotient (i.e., $\sigma(q_{\ell, P}(X)) \subseteq \sigma(H)$ ). It is Bayes-minimal if it contains exactly this quotient.
Loss Preorder: The paper defines a preorder $\preceq_P$ on losses based on the refinement of their induced Bayes quotients. $\ell_1 \preceq_P \ell_2$ if every distinction required by $\ell_1$ is also required by $\ell_2$ . Strict refinement ( $\ell_1 \prec_P \ell_2$ ) implies that $\ell_2$ requires strictly more information than $\ell_1$ .
Qualitative Obstruction: The central theoretical result (Proposition 3.2) states that if $\ell_{source} \prec_P \ell_{target}$ , any representation that is Bayes-minimal for the source loss is insufficient for the target loss. Consequently, the frozen-transfer excess risk is strictly positive.
Quantitative Identity for Log Loss: For finite-output log loss, the paper derives an exact identity for the frozen-transfer gap (Theorem 3.3). The excess risk is precisely the conditional mutual information lost by the representation:
$R^*_{\log, P}(H) - R^*_{\log, P}(X) = I(Y; X | H) = \mathbb{E} [ D_{KL}(P(Y|X) \parallel P(Y|H)) ]$
This quantifies the penalty as the conditional information about $Y$ discarded by the frozen representation $H$ .

Key Contributions

Identification of Loss Shift: The paper establishes loss shift as a distinct transfer failure mode, separate from distribution shift, where the change in objective function necessitates different representations even under a fixed data distribution.
Representation-Theoretic Substrate: It utilizes the Bayes-sufficient quotient framework to order losses by refinement, providing a rigorous mechanism to compare the informational demands of different objectives.
Exact Characterization of Transfer Penalty: For log loss, the paper provides an exact information-theoretic formula for the transfer gap, linking it directly to conditional mutual information and expected KL divergence.
Binary Classification Example: It explicitly demonstrates that an accuracy-minimal representation (preserving only the Bayes class decision) is insufficient for log-loss prediction, which requires the full conditional probability vector.

Experimental Results
The paper validates the theory through four experimental settings where $P(X, Y)$ is fixed but the loss changes:

Controlled Binary Model: A synthetic discrete model where the Bayes quotient for classification is known exactly. Results show that a representation preserving only the Bayes class ( $S$ ) achieves identical accuracy to the full representation ( $X$ ) but incurs a significantly higher Negative Log Likelihood (NLL). The empirical NLL gap matches the theoretical conditional-KL gap with high precision (correlation > 0.999).
Learned Bottleneck Representations: Using a learned MLP with a bottleneck and source-time noise, the study shows that a representation trained for classification accuracy (hinge loss) and then frozen fails to achieve optimal log-loss performance downstream. Fine-tuning the encoder recovers the performance, confirming the information was present in $X$ but discarded by the frozen representation.
dSprites Image Experiment: An image-based setup using latent factors for class ( $S$ ) and probability level ( $T$ ). Representations trained with hinge loss preserved $S$ but compressed $T$ , leading to higher NLL compared to representations trained with log loss, which preserved more information about $T$ .
CIFAR-10H Human Labels: Using human soft-label distributions, the paper compares hard-label (majority vote) training against soft-label training. Hard-label frozen encoders, despite achieving comparable majority-label accuracy, perform significantly worse on soft-label NLL and KL divergence, particularly on ambiguous images where human uncertainty is high.

Significance and Claims
The paper claims that the usefulness of a representation is determined by the pair $(P, \ell)$ . A representation can be optimal for one loss and intrinsically insufficient for another, even when source and target data are drawn from the same distribution.

The significance lies in complementing distribution shift as a fundamental mechanism in transfer learning. It identifies that frozen transfer fails not only when the data changes but also when the objective changes, specifically when the target loss strictly refines the source loss. The framework provides a diagnostic tool: if a frozen representation is compressed relative to the target-loss quotient, downstream heads cannot recover the discarded target-relevant information. This explains why classification-equivalent representations can have different optimal log-loss performance and motivates the use of target-aligned losses during pretraining or the avoidance of aggressive compression when the downstream task requires finer probabilistic distinctions.