🔢 mathematics

Log-Likelihood Loss for Semantic Compression

This paper investigates lossy source coding under a log-likelihood distortion measure that models semantic reconstruction as a probabilistic generation process, characterizing the resulting rate-distortion function and its connections to log-loss compression, classical rate-distortion, and rate-distortion with perfect perception.

Original authors: Anuj Kumar Yadav, Dan Song, Yanina Shkel, Ayfer Özgür

Published 2026-01-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anuj Kumar Yadav, Dan Song, Yanina Shkel, Ayfer Özgür

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a message to a friend, but you have a strict limit on how many words you can use. This is the classic problem of data compression: how do you shrink a file without losing too much of the original meaning?

Usually, we measure "loss" by comparing the original file pixel-by-pixel or letter-by-letter with the reconstructed file. If a single pixel is slightly off, we count it as an error. This paper proposes a completely different way of thinking about "loss," especially for modern tasks like AI image generation or summarizing text.

Here is the core idea, broken down with simple analogies:

1. The Old Way vs. The New Way

The Old Way (Point-to-Point): Imagine you are describing a photo of a cat to a friend. The old method says, "If you draw the cat's ear slightly too high, that's a mistake." It cares about the exact shape and position of every detail.
The New Way (Log-Likelihood Loss): This paper suggests a different approach. Instead of asking, "Is the drawing exactly like the photo?", we ask, "If I give you this drawing, how likely is it that a professional artist could paint the original photo based on it?"

In this new system, the "reconstruction" isn't a copy; it's a set of instructions or a recipe. The goal isn't to make the copy look identical; the goal is to make sure the original source is the most probable outcome if you follow those instructions.

2. The "Magic Recipe" Analogy

Think of the source data (like an image or a text document) as a complex dish, like a gourmet stew.

Traditional Compression: You try to send the stew itself, but you have to freeze it and shrink it. When your friend thaws it, they check if it tastes exactly the same. If a carrot is slightly mushy, they say, "This is a bad compression."
This Paper's Approach: You don't send the stew. You send a recipe card.
- The "distortion" (error) is measured by how well the recipe card predicts the stew.
- If the recipe says "Add salt," and the original stew was salty, the recipe is a good fit.
- If the recipe says "Add sugar," but the stew was salty, the recipe is a bad fit (high distortion).
- The "reconstruction" isn't the stew; it's the probability that the stew exists given the recipe.

The paper calls this Log-Likelihood Loss. It measures how surprised you would be to see the original data if you only had the compressed "recipe" (the semantic representation).

3. Why This Matters (The "Denoising" Effect)

The authors show that this method naturally handles "noise" (messy data).

Analogy: Imagine you are trying to guess the plot of a movie, but you only have a blurry, static-filled recording of the dialogue.
If you try to copy the static exactly, you get garbage.
But if you use this "recipe" method, your brain (the decoder) looks at the blurry audio and asks, "What is the most likely movie scene that would produce this sound?"
The result is that the compression process actually cleans up the noise. It forces the system to focus on the "semantic" meaning (the plot) rather than the static (the noise).

4. The "Universal Translator" Claim

One of the paper's biggest claims is that this specific way of measuring loss is actually a master key.

The authors prove that almost any other way of measuring compression error (like standard pixel differences or text similarity) can be translated into this "recipe" language.
Metaphor: It's like discovering that all the different languages of the world (Hamming distance, Squared Error, etc.) are actually just different dialects of one single, universal language (Log-Likelihood). If you can solve the problem in this universal language, you can solve it for all the others.

5. Perfect Perception

Finally, the paper connects this to "Perfect Perception."

The Problem: Sometimes, a compressed image looks mathematically perfect but feels "fake" or "uncanny" to a human.
The Solution: The authors show that if you use this "recipe" method correctly, you can guarantee that the reconstructed image (or text) doesn't just look similar; it feels statistically identical to the original. It ensures the "vibe" or the "distribution" of the data is preserved perfectly, even if the individual pixels aren't.

Summary

This paper introduces a new way to measure how well we compress data. Instead of asking, "Is the copy exact?", it asks, "Is the copy a good hint for the original?"

By treating compression as a game of probabilistic guessing rather than exact copying, the authors show we can:

Better preserve the "meaning" (semantics) of data.
Automatically clean up noise.
Unify many different compression problems under one mathematical framework.
Achieve "perfect perception," where the reconstructed data feels just as real as the original.

Technical Summary: Log-Likelihood Loss for Semantic Compression

Problem Statement

This paper investigates the fundamental limits of lossy source coding under a novel distortion measure: the log-likelihood loss. Given a source $X \sim P_X$ and a prescribed conditional distribution $P_{X|U}$ , the distortion between a source realization $x$ and a reconstruction $y$ (where $y$ acts as a latent semantic representation) is defined as:
$d_{\ell\ell}(x, y) \triangleq \log \frac{1}{P_{X|U}(x|y)}$
Unlike traditional rate-distortion problems that minimize pointwise discrepancies (e.g., Mean Squared Error or Hamming distance), this framework models scenarios where the reconstruction $Y$ is not a direct approximation of $X$ , but rather a semantic representation from which $X$ can be probabilistically generated. The goal is to characterize the rate-distortion function $R_{\ell\ell}(D)$ , which minimizes the mutual information $I(X; Y)$ subject to an expected log-likelihood distortion constraint.

Methodology and Theoretical Framework

The authors formulate the problem using standard information-theoretic tools, defining the Rate-Likelihood Loss Distortion Function (RLLDF) as:
$R_{\ell\ell}(D) = \min_{W_{Y|X}: E[d_{\ell\ell}(X,Y)] \le D} I(X; Y)$
The analysis proceeds by establishing the properties of this function and its relationship to existing frameworks:

Connection to Log-Loss: The paper relates log-likelihood loss to the standard log-loss distortion (where the decoder outputs a predictive distribution). The log-likelihood framework is shown to be a restricted version of log-loss, where the decoder is constrained to output distributions from a specific family $\{P_{X|U}(\cdot|u)\}_{u \in \mathcal{U}}$ . Consequently, the standard log-loss rate-distortion function serves as a lower bound for $R_{\ell\ell}(D)$ .
Generalization of Classical Problems: A central methodological contribution is demonstrating that many classical rate-distortion problems (e.g., Hamming distortion, Squared Error) can be reformulated as log-likelihood loss problems. This is achieved by identifying conditions under which a distortion measure $d(x,y)$ can be expressed via a conditional distribution $P_{X|Y} \propto e^{-\lambda d(x,y)}$ . When such a mapping exists, the classical rate-distortion function $R(D)$ is shown to be equivalent to $R_{\ell\ell}(\tilde{D})$ via an affine transformation of the distortion parameter.
Rate-Distortion-Perception: The framework is applied to the problem of rate-distortion with perfect perception (where the reconstruction distribution must match the source distribution). The authors propose an achievable scheme involving a two-step process: passing the source through an arbitrary noisy channel and then compressing the output under log-likelihood loss.

Key Results

1. Fundamental Properties of $R_{\ell\ell}(D)$

Operating Range: The function is defined for distortion values $D \in [D_{\min}, D_{\max}]$ , where $D_{\min}$ corresponds to the minimum log-likelihood achievable by a maximum-likelihood decoder, and $D_{\max}$ corresponds to the case where no information is transmitted ( $R=0$ ).
Special Operating Point: At a specific distortion level $D^* = H(X|U)$ , the rate-distortion function admits a closed-form expression: $R_{\ell\ell}(D^*) = I(U; X)$ . At this point, the optimal reconstruction $Y$ is a sample from the posterior distribution $P_{U|X}$ .
Lower Bound: $R_{\ell\ell}(D) \ge R_{\ell}(D)$ , where $R_{\ell}(D)$ is the rate-distortion function under standard log-loss. Equality holds at the special operating points.

2. Equivalence to Classical Rate-Distortion

The paper establishes that if a distortion measure $d$ satisfies specific conditions (related to the existence of a "coupling" and a normalizing function $\mu(x, \lambda)$ ), the classical rate-distortion problem is equivalent to a log-likelihood loss problem.

Binary Source (Hamming Distortion): The authors derive the exact $R_{\ell\ell}(D)$ for a binary source with Hamming distortion, showing it recovers the standard binary rate-distortion function $H(p) - H(D)$ under the appropriate transformation.
Gaussian Source (Squared Error): Similarly, for a Gaussian source with squared error distortion, the framework recovers the classical result $R(D) = \frac{1}{2}\log(\sigma^2/D)$ .
Limitations: The paper notes that not all distortion measures satisfy these conditions (e.g., Gaussian source with absolute error distortion), implying the log-likelihood framework, while general, does not trivially solve all rate-distortion problems.

3. Rate-Distortion-Perfection

The paper addresses the problem of achieving zero perception error (where the marginal distribution of the reconstruction matches the source).

Achievability: The authors show that for a class of problems where the element-wise exponential matrix of the distortion measure is Completely Positive (CP), the proposed log-likelihood compression scheme is optimal.
Mechanism: The optimal strategy involves processing the source $X$ through a channel $P_{Z|X}$ and compressing $Z$ under log-likelihood loss at distortion level $H(Z|X)$ . This ensures the final reconstruction $Y$ has the same distribution as $X$ while minimizing the rate.

Significance and Claims

The paper claims that the log-likelihood loss provides a unified and natural distortion measure for modern compression settings involving semantic representations. Its significance lies in three main areas:

Semantic Modeling: It moves beyond pointwise fidelity to model compression where the goal is to preserve the ability to probabilistically generate the source from a latent semantic code. This aligns with modern generative AI and semantic compression paradigms.
Theoretical Unification: By showing that classical rate-distortion problems are special cases of log-likelihood loss, the paper suggests that the log-likelihood framework is as general as the classical framework for a wide class of problems.
Perception-Aware Compression: The work provides a concrete, achievable scheme for rate-distortion problems with perfect perception, linking the concept of "semantic fidelity" (via the prescribed $P_{X|U}$ ) with distributional matching.

The authors maintain a modest stance, acknowledging that while the framework is powerful, it does not characterize the rate-distortion function for all possible distortion measures, specifically those that do not satisfy the coupling conditions required to map them to the log-likelihood domain. The detailed proofs are deferred to the appendices, and the paper focuses on establishing the theoretical properties and connections rather than proposing new experimental algorithms.