💻 computer science

Towards a Theory of Pragmatic Information

This paper introduces a quantitative definition of "pragmatic information" as the Kullback-Leibler divergence between prior and posterior probabilities of a decision-relevant variable, establishing it as a non-negative, additive measure analogous to free energy that bridges the gap between standard information theory and the actual meaning utilized in decision-making across diverse fields like finance, biology, and gambling.

Original authors: Edward D. Weinberger

Published 2026-08-28✓ Author reviewed ⓘ

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Edward D. Weinberger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of communication, there is a long-standing distinction between the transmission of symbols and the transmission of meaning. For decades, the standard science of information has focused entirely on the mechanics of sending a message: how accurately can a string of letters, numbers, or signals be copied from a sender to a receiver without error? This field, known as information theory, excels at measuring the volume of data and the efficiency of its encoding. However, it has historically remained silent on a question that feels obvious to any human listener: how much does that message actually matter? A string of random gibberish and a clear, life-saving instruction might contain the exact same amount of data, yet their impact on a person's life is worlds apart. The standard tools of the trade cannot tell the difference between a noise-filled radio broadcast and a crucial warning, because they measure the signal, not the significance.

This gap between data and meaning is the central problem addressed by a new theoretical framework proposed by Edward D. Weinberger. The paper seeks to quantify "pragmatic information," a concept that measures the specific amount of meaning in a message that is relevant to a particular decision. The core idea is that a message only becomes truly informative when it changes the mind of the receiver. If a person holds a certain belief about the world, and a new piece of information arrives that shifts that belief, the size of that shift represents the pragmatic value of the message. The author argues that this shift is not just a vague feeling of being informed, but a measurable quantity that can be calculated, much like the distance between two points on a map. This approach moves beyond asking how well a message was sent, to asking how effectively it altered the receiver's understanding of reality to guide a future action.

The paper builds its argument on a simple, concrete scenario: a person deciding whether to play a slot machine. Imagine a gambler standing before a machine with a single lever, known as a "one-armed bandit." The gambler knows that if they pull the lever, they might win a prize or win nothing, but they do not know the true odds of winning. Every time they play, the result—win or loss—provides a piece of data. Standard information theory would say that each play provides a fixed amount of information, regardless of how many times the machine has been played before. However, the author demonstrates that the practical value of that information changes dramatically over time. When the gambler has played only a few times, a new result tells them very little about the machine's true nature. But as they accumulate hundreds of results, they begin to form a reliable estimate of the odds. At that point, a single new result adds almost no new value to their decision-making process because they already know what to expect. The paper shows that the "pragmatic information" of a message is highest when it surprises the receiver by correcting a mistaken belief, and it drops to zero when the receiver already knows the answer or cannot process the new data.

To formalize this, the author defines the pragmatic information of a message as the degree to which it updates a person's probability estimates. If a person believes there is a fifty-fifty chance of an event occurring, and a message arrives that convinces them the chance is actually ninety percent, that message carries a high amount of pragmatic information. If the message arrives and the person's belief does not change at all, perhaps because they already knew the outcome or because they distrust the source, the pragmatic information is zero. The paper proves mathematically that this measure is always positive or zero; it never subtracts from a person's knowledge in a way that reduces their total capacity to understand. Furthermore, the author shows that for a series of messages, the total pragmatic information is simply the average of the individual updates. This means that if a person receives a stream of news, the total value they gain is determined by how much each piece of news shifts their specific beliefs about the world, not by the sheer volume of words spoken.

One of the most striking insights of the paper is the comparison between this new measure and a concept from physics known as free energy. In thermodynamics, free energy is the portion of a system's total energy that is actually available to do useful work, as opposed to energy that is wasted as heat. Similarly, the author proposes that pragmatic information is the "free information" available to do the useful work of making a decision. A message might contain a vast amount of raw data, but if that data does not help the receiver choose a better course of action, it is like heat energy that cannot power a machine. The paper suggests that just as free energy quantifies the useful work a system can perform, pragmatic information quantifies the useful change a message can induce in a decision-maker's mind. This distinction is crucial because it separates the quantity of a message from its utility. A short, sharp warning can have more pragmatic information than a long, detailed report if the warning is the only thing needed to make a critical choice.

The paper also tackles the difficult issue of disinformation and the value of information. It acknowledges that a message can change a person's mind without making that mind more accurate. If a gambler is tricked by a false signal into believing a machine is rigged when it is not, their belief has changed, and the paper's formula would still register a high amount of pragmatic information. The author clarifies that this measure captures the change in belief, not the truth of that belief. To handle this, the paper introduces a classification system where messages are sorted into those that are pragmatically useful, those that are irrelevant, and those that are disinformative. A message is disinformative if it leads the receiver to a worse decision, even if the receiver's internal state has shifted significantly. This allows the theory to account for the fact that being "informed" is not the same as being "correct," and that the true value of information lies in its ability to guide a decision toward a better outcome, not just to alter a belief.

The implications of this theory extend far beyond gambling machines. The author applies the framework to the efficient market hypothesis in finance, a famous idea suggesting that stock prices already reflect all available information, making it impossible to consistently beat the market. The paper reformulates this hypothesis to say that a market is efficient only if the information available to a specific investor is not pragmatically useful to them. This explains why some investors, like those with sophisticated computer algorithms, can find patterns and make profits while others, relying on human intuition, cannot. The information is the same for both, but the ability to process it and turn it into a decision differs. If an investor lacks the computational capacity to extract meaning from a complex stream of data, that data has zero pragmatic information for them, rendering the market efficient in their specific context. This reframing suggests that market efficiency is not a universal property of the market itself, but a relationship between the information and the specific capabilities of the person receiving it.

Finally, the paper touches on the biological realm, suggesting that this theory could help measure the rate of evolution. In this view, the environment acts as the message sender, and the population of organisms acts as the receiver. The "decisions" made by the population are which genetic traits survive and reproduce. The paper posits that the rate at which a species adapts is directly linked to how much pragmatic information the environment provides about which traits are best. If the environment changes slowly, the information provided is low, and evolution proceeds slowly. If the environment provides clear, high-value signals about survival, the population can adapt more quickly. However, the author notes that this process is limited by the "computational capacity" of the organism. Just as a human cannot process an infinite stream of financial data, a biological system has limits on how much information it can absorb and use to update its genetic strategy. This sets a theoretical ceiling on how fast evolution can proceed, a limit determined not just by the environment, but by the receiver's ability to understand the message.

The work presented is a theoretical construction, built on mathematical proofs and logical deductions rather than new experimental data. The author demonstrates that the definition of pragmatic information follows naturally from established principles of probability and information theory, showing that it behaves in ways that align with our intuitive understanding of meaning. The paper does not claim to have solved the mystery of consciousness or to have created a universal calculator for the value of every message in the universe. Instead, it offers a precise, quantitative tool for understanding how information becomes useful. It provides a way to measure the gap between data and decision, showing that the true power of a message lies not in its length or complexity, but in its ability to shift a mind toward a better action. By grounding the abstract concept of meaning in the concrete mechanics of decision-making, the paper offers a new lens through which to view communication, from the flip of a coin to the trading of stocks and the survival of species.

Technical Summary: Towards a Theory of Pragmatic Information

Problem Statement
Standard information theory, rooted in Shannon's framework, quantifies the technical transmission of symbols and the entropy of a message but remains silent on the "meaning" or "usefulness" of that information relative to a specific decision. While Shannon addresses the "technical problem" of transmission accuracy, it does not address the "semantics problem" (conveying meaning) or the "effectiveness problem" (changing conduct). Existing attempts to quantify meaning often rely on mutual information between a message and a state of the world, which fails to account for scenarios where a decision-maker's prior beliefs ( $q$ ) are incorrect and require correction, or where the utility of information is strictly tied to a specific decision context. The paper seeks to fill this gap by providing a quantitative definition of "pragmatic information"—the amount of meaning in a message relevant to a particular decision.

Methodology and Conceptual Framework
The paper establishes a framework involving a decision-maker ( $\Delta$ ) and a random variable ( $\omega$ ) representing the "state of the world" within a discrete sample space $\Omega$ .

Belief Update: $\Delta$ holds a prior probability distribution $q$ over $\omega$ . Upon receiving a message $m$ , $\Delta$ updates its beliefs to a posterior distribution $p_m$ .
Definition of Pragmatic Information: The pragmatic information of a single message $m$ , denoted $D_\Delta(p_m || q)$ , is defined as the Kullback-Leibler (KL) divergence between the posterior and prior distributions:
$D_\Delta(p_m || q) = \sum_i p_{i|m} \log \left( \frac{p_{i|m}}{q_i} \right)$
This definition assumes base-2 logarithms. The paper explicitly chooses the ordering $D(p_m || q)$ (posterior relative to prior) rather than the reverse, based on the assumption that the message provides an improvement or correction to the prior belief.
Ensemble Definition: For an ensemble of messages $\mathcal{M}$ sampled with probability $\phi_m$ , the pragmatic information $\Phi_\Delta(\mathcal{M}; \Omega)$ is defined as the expected value of the KL divergence over the ensemble:
$\Phi_\Delta(\mathcal{M}; \Omega) = \sum_{i,m} \phi_m p_{i|m} \log \left( \frac{p_{i|m}}{q_i} \right)$
The paper invokes Birkhoff's ergodic theorem to justify that the average pragmatic information accrued over a large number of trials converges to this expected value.

Key Contributions and Theoretical Results
The paper derives several properties and theorems to validate the proposed definition:

Interpretation as Coding Efficiency (The "Wrong Code Theorem"): The pragmatic information of a single message is proven to be equal to the expected difference in the length of the shortest binary encoding of $\omega$ under the prior distribution versus the posterior distribution. Specifically, $E[L_q(\omega) - L_{p_m}(\omega)] = D_\Delta(p_m || q)$ . This links pragmatic information directly to the reduction in coding length required to describe the outcome once the message is received.
Non-negativity and Additivity: Pragmatic information is non-negative ( $D \ge 0$ ), equaling zero only if the message does not change the decision-maker's beliefs. It is additive for independent decisions and "pragmatically independent" messages (where the joint pragmatic information equals the sum of individual pragmatic informations).
Relation to Mutual Information: The paper establishes a fundamental relationship between pragmatic information and mutual information $I(\mathcal{M}; \Omega)$ :
$\Phi_\Delta(\mathcal{M}; \Omega) = I(\mathcal{M}; \Omega) + D_\Delta(p || q)$
Here, $p$ is the marginal posterior distribution. This result shows that pragmatic information coincides with mutual information only in the special case where the prior $q$ equals the marginal posterior $p$ . In general, pragmatic information includes the mutual information plus the cost of updating the prior biases.
Analogy to Free Energy: The paper posits that pragmatic information is the information-theoretic analogue of Helmholtz free energy ( $F = U - TS$ ). Just as free energy represents the portion of total energy available to do useful work, pragmatic information represents the portion of total information actually used in making a decision, distinguishing it from "irrelevant" information.
Disinformation and Noise: The framework allows for the classification of messages into pragmatically useful, pragmatically irrelevant, and pragmatically disinformative sets. Crucially, the paper notes that disinformation (messages that change beliefs but lead to worse outcomes) still possesses positive pragmatic information because the definition measures the change in belief, not the value of the resulting decision.

Illustrative Applications
The paper applies the theory to three specific domains to demonstrate its utility:

The One-Armed Bandit: In a slot machine scenario with unknown payout probability $\pi$ , the paper calculates the pragmatic information gained from each trial. It demonstrates that as the number of trials $T$ increases, the pragmatic information per trial decreases toward zero, reflecting the diminishing returns of learning the payout probability. The information is highest when the decision (play vs. no play) is most uncertain (i.e., when $\pi \approx 0.5$ ).
Biological Evolution (Quasi-Species): The theory is applied to the quasi-species model, where the environment acts as the decision-maker $\Delta$ selecting the fittest replicator. The "messages" are environmental details affecting mutation and growth rates. The paper suggests that the "error catastrophe" in this model represents a limit on the computational capacity of the system to accrue pragmatic information.
Efficient Market Hypothesis (EMH): The paper reformulates the EMH. Instead of claiming prices reflect all information, it proposes that a market is efficient with respect to a specific participant if that participant cannot extract any pragmatically useful information from the available message ensemble. This allows for a market to be efficient for some participants (e.g., those with limited computational capacity) but not others (e.g., those with superior algorithms).

Significance and Claims
The paper claims to provide a rigorous, quantitative definition of meaning that is distinct from, yet related to, standard information theory. Its primary significance lies in:

Bridging the gap between technical information transmission and decision-making effectiveness.
Providing a mathematical justification for "meaning" based on the reduction of uncertainty relevant to a specific decision context (via KL divergence).
Highlighting the role of the receiver's computational capacity as a limiting factor in the accrual of pragmatic information, analogous to channel capacity in Shannon theory.
Offering a framework to distinguish between information that is merely transmitted and information that is "free" (useful for work/decision), particularly in complex systems like financial markets and biological evolution.

The author maintains a modest tone, noting that while the theory is general, it relies on the assumption that the decision-maker updates beliefs rationally based on the message, and that the "value" of the resulting decision (whether good or bad) is a separate metric from the pragmatic information itself.

More like this