🔢 mathematics

Risk-Aware Information Theory

This paper introduces a risk-aware information theory based on expectiles that extends beyond Shannon's framework to capture extreme risks and heterogeneous risk-sensitivity in multiuser systems, thereby enabling safer autonomous systems and advanced machine intelligence.

Original authors: Hamidou Tembine

Published 2026-06-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamidou Tembine

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why "Average" Isn't Good Enough

Imagine you are a captain steering a ship. For over 60 years, navigators have used a map based on Shannon's Information Theory. This map works by calculating the average weather conditions. If there is a 99% chance of calm seas and a 1% chance of a massive tsunami, the "average" says the sea is mostly calm.

The problem? If that 1% tsunami hits, your ship sinks. The "average" didn't warn you because it treated the tiny chance of disaster the same as a tiny chance of a sunny day.

This paper argues that for modern, high-stakes systems (like self-driving cars, AI, or financial markets), we need a new map. We need to stop looking at the average and start looking at the worst-case (or best-case) scenarios. The author introduces a new mathematical tool called Expectiles to build this new map.

The New Tool: The "Expectile"

Think of a Expectile as a "weighted average" that cares about the edges of the story, not just the middle.

The Standard Average (Shannon): Imagine a classroom of 100 students. 99 get a grade of 80, and 1 gets a grade of 20. The average is 79.8. The teacher says, "Great job, class!" The disaster (the failing student) is hidden in the math.
The Risk-Averse Expectile (τ > 0.5): This is like a strict parent who only cares about the failing student. They look at that 1% chance of a 20 and say, "We have a problem!" They weight the bad outcomes heavily.
The Risk-Seeking Expectile (τ < 0.5): This is like an optimistic gambler. If there's a 1% chance of winning a million dollars, they focus entirely on that jackpot and ignore the 99% chance of losing their bet.

The paper replaces the standard "average" math with these "weighted" math tools to measure information.

What Changes When We Use This New Math?

The paper proves that when you switch from "Average" to "Expectile," the rules of information change completely. Here are the main discoveries:

1. Information Can Be Negative

In the old world, information is always a positive number (you can't have "less than zero" knowledge).

The New World: Under a "risk-seeking" mindset (gambling on rare wins), the math can actually produce negative information. This sounds weird, but it means that in a high-risk environment, a rare event might actually confuse the system more than it helps it, effectively subtracting from your understanding.

2. The "Two-Way Street" Breaks

In the old world, if Alice sends a message to Bob, the amount of information Alice gives Bob is exactly the same as the amount of information Bob gains from Alice. It's a perfect two-way street.

The New World: With risk-aware math, the street becomes one-way and bumpy.
- Analogy: Imagine Alice is a cautious driver (risk-averse) and Bob is a reckless driver (risk-seeking).
- Alice looks at the road and sees a tiny chance of a rockslide, so she says, "This road is dangerous!" (High information about risk).
- Bob looks at the same road, sees a tiny chance of a shortcut, and says, "This road is perfect!" (Low information about risk).
- The paper shows that the "information" Alice sends to Bob is not the same as the "information" Bob receives from Alice. The direction matters.

3. The "Sum" Rule Fails

In the old world, if you have two sources of information, the total information is just Source A + Source B.

The New World: Because the math is now "non-linear" (it bends), Source A + Source B does not equal the Total.
- Analogy: Imagine mixing two colors of paint. In the old world, Red + Blue = Purple (a simple sum). In this new world, mixing Red and Blue might create a color that is either darker than expected (if you are risk-averse) or brighter than expected (if you are risk-seeking), depending on how the colors interact. You can't just add them up; you have to mix them first.

Real-World Applications Mentioned in the Paper

The paper applies these ideas to three specific areas:

1. Communication Channels (The "Super-Channel")

The Scenario: Sending data over a noisy wire (like Wi-Fi).
The Result: If you are risk-averse (you hate errors), the paper shows you can actually achieve a higher data rate than the old "average" theory predicted.
Why? By designing your signal to specifically handle rare, noisy bursts (the "tsunamis"), you unlock a "risk premium." You can push more data through because you are prepared for the worst, whereas the old theory was too conservative.

2. Multi-User Networks (The "Traffic Jam")

The Scenario: Many users trying to talk to one receiver at the same time (like a crowded conference call).
The Result: The "shape" of how much everyone can talk changes based on their risk personality.
- If everyone is risk-averse, the "capacity map" (the limit of how much data can flow) expands outward.
- If everyone is risk-seeking, the map shrinks inward.
- If users have different risk levels (one is cautious, one is reckless), the map gets twisted and skewed, creating a strange, irregular shape that the old math couldn't predict.

3. Artificial Intelligence (The "Super-Intelligence" Gap)

The Scenario: Training an AI to make decisions.
The Result: The paper claims that current AI (Shannon Intelligence) is "blind" to tail risks. It optimizes for the average outcome.
The "Super-Intelligence" Claim: To build a truly robust AI (Superintelligence), the system must be able to adapt its "risk setting" (τ) on the fly. It needs to know when to be cautious (avoiding catastrophic failure) and when to be bold (seeking rare breakthroughs). The paper argues that an AI trained only on "averages" will fail when it encounters extreme, rare events because it was never taught to value them.

The Bottom Line

This paper says: "Stop averaging everything."

In a complex, dangerous, or high-stakes world, the "average" hides the most important parts of the story—the rare disasters and the rare jackpots. By using Expectiles, we can build a new theory of information that sees these extremes, allowing us to design safer autonomous systems, more efficient networks, and smarter AI that understands the true cost of risk.

Technical Summary: Risk-Aware Information Theory via Expectiles

Problem Statement

Shannon's classical information theory relies fundamentally on the expectation operator ( $E[\cdot]$ ) to quantify entropy, divergence, and mutual information. This framework implicitly assumes risk-neutrality, treating large deviations and rare events strictly in proportion to their probability without additional weighting. The paper argues that this assumption is insufficient for modern applications involving autonomous systems, adversarial environments, financial signal processing, and advanced machine intelligence. In these contexts, the tail behavior of information measures often dominates performance; a low-probability catastrophic failure or a rare high-reward event may be invisible or misvalued by risk-neutral metrics. The core problem is the lack of a systematic information-theoretic framework that incorporates coherent risk measures to account for asymmetric risk sensitivity (risk-aversion or risk-seeking) in the evaluation of information.

Methodology

The paper proposes a systematic replacement of the expectation operator with expectiles, a class of coherent risk measures that are elicitable and computationally tractable.

Expectile Definition: For a random variable $Z$ and a parameter $\tau \in (0, 1)$ , the $\tau$ -expectile $e_\tau(Z)$ is defined as the minimizer of an asymmetric quadratic loss function:
$e_\tau(Z) \in \arg\min_{c \in \mathbb{R}} \mathbb{E}[\ell_\tau(Z - c)], \quad \text{where } \ell_\tau(u) = |\tau - \mathbb{1}_{\{u<0\}}| u^2.$
- $\tau = 1/2$ : Reduces to the standard expectation ( $E[Z]$ ).
- $\tau > 1/2$ : Risk-averse (weights upper tails of losses more heavily).
- $\tau < 1/2$ : Risk-seeking.
Reformulation of Information Measures: The authors define new information-theoretic quantities by applying the expectile operator to the underlying log-probability variables:
- Expectile Entropy: $H_\tau(X) = e_\tau[-\log f_X(X)]$ .
- Expectile KL Divergence: $D_\tau(P\|Q) = e_\tau^{(P)}[\log \frac{dP}{dQ}(X)]$ .
- Expectile Mutual Information: $I_\tau(X; Y) = D_\tau(P_{XY} \| P_X P_Y)$ .
- Expectile Cross-Entropy: $H_\tau(P, Q) = e_\tau[-\log q(X)]$ .
Theoretical Analysis: The paper utilizes properties of expectiles (translation invariance, positive homogeneity, monotonicity, and subadditivity for $\tau \geq 1/2$ ) to derive bounds, prove non-equivalence with Shannon measures, and analyze the failure of classical additivity and symmetry properties.
Game-Theoretic Extension: In multi-user settings, the framework induces a Mean-Field-Type Game (MFTG) where agents have heterogeneous risk-sensitivity indices ( $\tau_j$ ), leading to strategic interactions over information exchange.

Key Contributions and Results

1. Fundamental Non-Equivalence

The paper establishes that for any $\tau \neq 1/2$ , expectile-based information measures are not merely reparameterizations of Shannon's measures.

Theorem 1: There exist distributions where $I_\tau(X; Y) \neq I(X; Y)$ .
Sign of Divergence: While $D_\tau(P\|Q) \geq 0$ for $\tau \geq 1/2$ , the paper proves that for $\tau < 1/2$ (risk-seeking), the expectile KL divergence can be negative and may vanish for distinct distributions ( $P \neq Q$ ).
Sign of Mutual Information: Similarly, $I_\tau(X; Y)$ can be negative for $\tau < 1/2$ .

2. Failure of Classical Additivity and Symmetry

A central result (Theorem 2) demonstrates that the linear superposition laws of classical information theory break down under asymmetric risk sensitivity.

Non-Additivity: $I_\tau(X; Y) \neq H_\tau(X) - H_\tau(X|Y)$ for $\tau \neq 1/2$ .
Directional Asymmetry: The information gain is inherently directional: $H_\tau(X) - H_\tau(X|Y) \neq H_\tau(Y) - H_\tau(Y|X)$ .
Example: In a Gaussian channel with heteroskedastic noise, the forward and reverse information flows diverge significantly when $\tau \neq 0.5$ , creating an asymmetric vector field of information transfer.

3. Risk-Aware Channel Capacity

The paper analyzes the capacity of Additive White Gaussian Noise (AWGN) channels under risk-aware criteria.

Capacity Expansion: For risk-averse agents ( $\tau > 1/2$ ), the capacity region expands outward compared to the Shannon limit. This is driven by a positive "risk premium" ( $e_\tau(Q) > 0$ ) arising from the tail volatility of the information density.
Non-Uniqueness: Unlike the unique Gaussian capacity-achieving distribution in Shannon theory, the risk-aware capacity maximizer may not be unique if the optimal distribution is asymmetric, as the reflection of an optimal distribution is also optimal.
Multi-User Systems:
- Broadcast Channel (BC): The capacity region is defined via Successive Interference Cancellation (SIC), with boundaries expanding for $\tau > 0.5$ .
- Multiple Access Channel (MAC): The paper introduces Asymmetric Risk-Aware MAC where users have distinct $\tau_j$ . The capacity region exhibits complex geometric deformations (shearing, contraction, or expansion) depending on the combination of risk-seeking and risk-averse users. The joint sum-rate is bounded by the most conservative (risk-averse) user in the coalition.

4. Strategic Expectile Information Theory

The framework is extended to networks of Machine Intelligence (MI) agents.

Game Formulation: Agents with heterogeneous $\tau_j$ engage in a non-cooperative game (Expectile Information Rate Game).
Equilibrium: The paper proves the existence of a Risk-Aware Informational Nash Equilibrium using Glicksberg's extension of the Fan-Debreu theorem, provided utility functionals are upper semicontinuous and quasi-concave.
MIMO Systems: For M-user MIMO MACs, the capacity region is bounded by the minimization of risk-adjusted terms across sub-coalitions, forcing agents to dynamically tilt spatial eigenvectors to avoid tail-risk variance.

5. Multimodal LLMs and Superintelligence

The paper applies the theory to Large Language Models (LLMs) and multimodal systems.

Budget-Modality Separation: Theorem 6 proves that under risk-sensitive criteria ( $\tau \neq 1/2$ ), the capacity of a multimodal system is strictly greater than the sum of its unimodal (e.g., text-only) capacities. Text-only representations incur fundamental information loss under risk-sensitive evaluation.
Shannon vs. Superintelligence: The paper defines "Shannon Intelligence" as optimization under $\tau=0.5$ and "Superintelligence" as optimization over the full risk spectrum $\tau \in (0, 1)$ .
Theorem 4: An agent optimized solely for Shannon Intelligence is structurally static and "blind" to tail-risk volatility. It cannot trace the optimal adaptive trajectory required for Superintelligence, as it fails to account for the upper-conditional variance of the loss distribution.

Significance and Claims

The paper claims to establish a foundation for risk-aware communication, learning, and collective intelligence. Its primary significance lies in demonstrating that:

Shannon information is insufficient for quantifying the extreme risks that drive advanced machine intelligence and safety-critical autonomous systems.
Risk-sensitivity fundamentally alters information geometry, breaking classical additivity and symmetry, and inducing non-linear, strategic interactions in multi-agent systems.
Expectiles provide a mathematically rigorous bridge between coherent risk measures and information theory, enabling the quantification of tail risks and the design of systems that are robust to catastrophic failures or capable of exploiting rare high-reward events.

The work positions risk-aware information theory not just as a modification of existing metrics, but as a necessary evolution for systems operating in non-ergodic, adversarial, or safety-critical environments where the "average" case is an inadequate predictor of performance.