Investing Is Compression

This paper argues that investing is fundamentally a compression problem, demonstrating that maximizing long-term wealth via the Kelly criterion is equivalent to minimizing the divergence between an investor's chosen distribution and the unknown true distribution, thereby framing portfolio optimization as an information-theoretic task.

The Gist

Imagine you are a captain steering a ship across a vast, stormy ocean. Your goal isn't just to survive the next wave, but to reach a distant treasure island with the most gold possible after 100 years.

Most captains (and most investors) try to maximize their speed right now. They ask, "Which wave is biggest today? Let's ride that!" But the author of this paper, Oscar Stiffelman, argues that this approach is a trap. If you chase the biggest wave every single time, you will eventually hit a rock and sink.

This paper argues that investing is actually a game of compression, similar to how your phone compresses a photo to save space. Here is the simple breakdown of the paper's big ideas:

1. The "Log" Secret: Why Being Boring Wins

In 1956, a mathematician named John Kelly figured out the secret to not going broke while trying to get rich. He realized that you shouldn't try to maximize your average money. Instead, you should maximize the average of the logarithm of your money.

• The Analogy: Imagine you have a rubber band. If you stretch it a little bit, it snaps back easily. If you stretch it too far, it breaks.
  • Normal Investing: Tries to stretch the rubber band as far as possible every time.
  • Kelly Investing: Stretches it just enough to gain speed, but never so much that it snaps.
• The Result: By being slightly more conservative and avoiding "ruin" (going to zero), you actually end up with more money in the long run because your money keeps compounding (growing on top of itself) without ever hitting a wall.

2. The "Horse Race" and the Perfect Guess

The paper uses a "Horse Race" analogy. Imagine a race with 10 horses. You have to bet your money on them.

• The Mistake: Betting based on what the crowd thinks (the market price) or just guessing.
• The Kelly Solution: You should bet in exact proportion to how likely you think each horse is to win.
  • If you think Horse A has a 50% chance, you bet 50% of your money on it.
  • If you think Horse B has a 10% chance, you bet 10%.

The Twist: The paper proves that if you do this, you don't even need to know the payout (how much money you win). You just need to know the probability of winning. If your guess matches the "true reality" of the race, you win the most.

3. The Big Reveal: Investing is "Compression"

This is the paper's most creative and important idea. The author takes a complex math trick used by a professor named Tom Cover and applies it to investing.

Usually, we think of investing as a chain of events:

Year 1 Return × Year 2 Return × Year 3 Return...

The paper says: "Wait, let's look at this differently." Instead of a chain, imagine you are trying to compress a story.

• The Story: The history of the stock market.
• The Compression: Your investment strategy.
• The Goal: To describe the market's behavior using the fewest "bits" of information possible.

The paper breaks your growth rate into three parts:

1. The Money Term: The rules of the game (how much the market pays out). You can't change this.
2. The Entropy Term: The natural chaos or randomness of the market. You can't change this either.
3. The Divergence Term (The Friction): This is the only part you control. It measures how wrong your guess is compared to reality.

The Analogy: Imagine you are trying to zip up a heavy coat (your portfolio).

• The coat is heavy (Market Rules).
• The zipper is sticky (Market Chaos).
• Your job: You are trying to align the teeth of the zipper perfectly. If your teeth (your strategy) don't match the other side (reality), the zipper gets stuck. This "stuckness" is the Divergence.

The Conclusion: To grow your wealth, you don't need to be a genius who predicts the future perfectly. You just need to minimize the friction. You need to make your "zipper teeth" (your portfolio) match the "other side" (the true market probabilities) as closely as possible.

4. Why This Changes How We Look at Backtests

When investors test strategies on past data (backtesting), they usually look at the average return. The paper says this is misleading because the past is full of weird, one-time events (like a pandemic or a bubble).

However, because the "Money" and "Entropy" parts are the same for everyone in a specific test, the only thing that matters is the Divergence.

• If Strategy A beats Strategy B, it's not because Strategy A is "smarter."
• It's because Strategy A's "zipper teeth" were aligned slightly better with the truth than Strategy B's.
• The difference in their success is measured in bits (like data compression).

5. The "Winner Fraction" Heuristic (For Venture Capital)

The paper also offers a practical tip for things like Venture Capital, where you don't know the exact odds, but you know some startups will win big and others will fail.

• The Old Way: Try to predict exactly how much money each startup will make. (Impossible).
• The New Way (Winner Fraction): Just guess the probability that a startup will be the "winner" of its group.
  • If you think Startup A has a 20% chance of being the #1 winner, and Startup B has a 10% chance, you put twice as much money in A.
• Why it works: The paper proves that even if you don't know the exact dollar amounts, simply betting on the likelihood of winning gets you very close to the perfect strategy. The "error" in your guess is limited by how chaotic the group of startups is (its entropy).

Summary

Investing isn't about predicting the future; it's about compressing the present.

Think of the market as a giant, noisy signal. Your job is to tune your radio (your portfolio) to the exact frequency of the truth.

• If you are tuned perfectly, the static (friction/divergence) disappears, and the music (wealth) plays loud and clear.
• If you are slightly off, the static grows, and your wealth grows slower.

The paper tells us that the best way to invest is to stop trying to be a fortune teller and start trying to be a perfect listener, aligning your bets exactly with the true probabilities of the world.

Technical Summary

1. Problem Statement

The paper addresses the fundamental nature of the investment problem, challenging the traditional separation of risk and reward found in Modern Portfolio Theory (MPT). It seeks to resolve the historical debate regarding the Kelly Criterion:

• The Conflict: John Kelly (1956) proposed maximizing the logarithm of wealth as the optimal strategy for long-term growth. However, economist Paul Samuelson argued this was an arbitrary, subjective utility function and successfully excluded it from mainstream economics.
• The Gap: While Tom Cover (Stanford) later proved the Kelly criterion is objectively optimal (maximizing long-term wealth, minimizing ruin, and being competitively optimal), the mechanism behind this optimality remained abstract. Specifically, the paper aims to demystify why the log-optimal strategy works by revealing the underlying information-theoretic structure of investing, particularly in general, non-stationary markets where asset returns are not mutually exclusive (unlike a simple "horse race").

2. Methodology

The author employs techniques from Information Theory, specifically leveraging work by Tom Cover on Universal Portfolios and Universal Compression. The core methodological innovation is a mathematical transformation of the multi-period investing problem.

• The "Product of Sums" to "Sum of Products" Transformation:

  • Traditionally, multi-period wealth $V_n$ is viewed as a product of sums (dot products of weights and returns over time): $V_n = \prod_{t=1}^n (\sum w_i r_{i,t})$.
  • The author rewrites this product of sums as a sum of products. By expanding the terms, the problem is reinterpreted as a weighted average of an exponential number of "horse-race" style sequences (where only one asset wins per period).
  • This transformation allows the application of Type Class analysis (from information theory), grouping sequences by their empirical frequencies.

• Asymptotic Analysis:

  • Using Stirling's approximation and Kullback-Leibler (KL) Divergence, the author analyzes the concentration of measure.
  • The analysis demonstrates that as the number of periods ($n$) increases, the weight of the portfolio concentrates exponentially on the "type class" where the empirical frequencies of asset returns match the portfolio weights.
  • Sequences that deviate from the portfolio weights are penalized by a factor of $e^{-n D_{KL}(P \| W)}$, where $D_{KL}$ is the divergence between the true distribution and the chosen allocation.

3. Key Contributions

A. The Three-Term Decomposition

The most significant theoretical contribution is the derivation of a closed-form decomposition for the expected log-growth rate ($g$) in the most general investing case. The growth rate is factored into three distinct terms:
$$g(W) = \underbrace{\log(\bar{R}_T(W^*))}_{\text{Money Term}} - \underbrace{H(W^*)}_{\text{Entropy Term}} - \underbrace{D_{KL}(W^* \| W)}_{\text{Divergence Term}}$$

1. Money Term: Represents the payoff/rules of the game (independent of the investor's allocation).
2. Entropy Term ($H$): Represents the inherent uncertainty of the market (independent of the investor's allocation).
3. Divergence Term ($D_{KL}$): Measures the "friction" or "drag" caused by the difference between the investor's chosen distribution ($W$) and the unknown true distribution ($W^*$).

Implication: Since the first two terms are constant regardless of strategy, maximizing growth is equivalent to minimizing the KL Divergence. This reframes investing fundamentally as a compression problem: the goal is to compress the uncertainty of the market into a distribution that matches the truth as closely as possible.

B. The Winner Fraction Heuristic

The author introduces a practical heuristic for scenarios where modeling the full joint return distribution is infeasible (e.g., Venture Capital).

• Concept: Allocate capital in proportion to the probability that an asset will "win" (dominate) the candidate set, rather than trying to predict exact returns.
• Theoretical Bound: The author proves that the growth shortfall of this heuristic relative to the optimal Kelly portfolio is bounded by the entropy of the winner-fraction distribution:
  $$g(W^*) - g(W_{\text{winner}}) \leq H(W_{\text{winner}})$$
• Utility: This makes the heuristic highly effective when the candidate set is small or when returns are extreme (low entropy), as the bound tightens.

C. Reinterpretation of Backtesting

The paper offers a new statistical framework for comparing investment strategies in non-stationary markets.

• Because the Money and Entropy terms are constant across strategies in a given backtest, the difference in log-growth between two strategies ($g(W_B) - g(W_A)$) is exactly equal to the difference in their KL Divergences from the ideal.
• This allows investors to measure strategy performance in bits, providing a robust metric that is less sensitive to extreme historical events than raw return averages.

4. Results

• Validation of Kelly: The paper confirms that the Kelly criterion is not a subjective utility function but the objectively optimal strategy for maximizing long-term wealth and minimizing ruin, valid even in the short term under game-theoretic conditions.
• Universality: The analysis confirms that the "friction" of suboptimal allocation grows exponentially with the divergence from the truth and the time horizon.
• Heuristic Performance: The "Winner Fraction" heuristic is proven to be approximately optimal, with the error strictly bounded by the entropy of the prediction distribution.

5. Significance

• Paradigm Shift: The paper moves the discourse from "Risk vs. Reward" (MPT) to "Compression vs. Truth." It argues that investing is not about balancing variance but about minimizing the information distance (divergence) between one's model and reality.
• Unification of Fields: It bridges Information Theory (compression, entropy, universal portfolios) and Finance (portfolio optimization, Kelly criterion), showing that the tools used to compress data are mathematically identical to those used to optimize wealth.
• Practical Application: By reducing the complex problem of joint distribution modeling to the simpler problem of estimating "winning probabilities" (with a known entropy bound), the paper provides a rigorous theoretical basis for heuristic approaches in high-uncertainty environments like Venture Capital.
• Resolution of Historical Debate: It definitively settles the Samuelson-Kelly debate by proving that log-utility is not arbitrary but is the unique solution to the problem of maximizing asymptotic growth in a competitive, information-theoretic framework.

In conclusion, Stifelman demonstrates that investing is fundamentally a compression problem. The optimal strategy is the one that minimizes the "bits" of divergence between the investor's allocation and the market's true distribution.