🔢 mathematics

Towards a Theory of Pragmatic Information

This paper introduces a rigorous quantitative definition of "pragmatic information" as the Kullback-Leibler divergence between prior and posterior probabilities of a decision-relevant variable, establishing its properties as a non-negative, additive measure analogous to free energy and demonstrating its applicability to problems in gambling, biological evolution, and finance.

Original authors: Edward D. Weinberger

Published 2026-08-19

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Edward D. Weinberger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of how we understand the universe, there is a long-standing distinction between the raw data of a signal and the actual meaning we extract from it. For decades, the standard science of information has focused almost entirely on the technical side: how accurately a message can be sent, how many bits are required to transmit a string of symbols, and how to prevent errors during that transmission. This framework, known as information theory, treats a message as a sequence of symbols without asking whether those symbols actually tell the receiver anything new or useful. It is like measuring the weight of a letter without ever opening the envelope to see if it contains a love note, a bill, or a blank sheet of paper. While this technical approach has been incredibly successful for building communication networks, it leaves a gap when it comes to the human experience of receiving information. We often say a message is "informative" only if it changes our mind or helps us make a better choice, yet the standard mathematical tools of the field have no way to measure that change.

This gap is the focus of a new theoretical framework proposed by Edward D. Weinberger, a researcher at New York University. His work attempts to bridge the divide between the technical transmission of data and the practical utility of that data for a decision-maker. The core idea is simple yet profound: information only becomes meaningful when it alters the beliefs of the person or machine receiving it, specifically in a way that influences a decision. Weinberger calls this "pragmatic information." It is not about the length of the message or the complexity of the code, but about the shift in probability that occurs in the mind of the receiver. If a message arrives and the receiver's understanding of the world remains exactly the same, then no pragmatic information has been gained, regardless of how many words were spoken. Conversely, if a single word causes a receiver to completely revise their expectations about a future event, that word carries a high amount of pragmatic information.

To define this concept rigorously, the paper introduces a model where a decision-maker holds a set of initial beliefs about a situation, represented as a probability distribution. This is the "prior" view of the world. When a message arrives, these beliefs are updated to a new set of probabilities, known as the "posterior." The amount of pragmatic information is then calculated by measuring the distance between these two sets of beliefs. If the message causes a large shift, the distance is great, and the information is valuable. If the message causes no shift, the distance is zero, and the information is useless. This approach allows the theory to account for the fact that the same message can mean different things to different people. A message that is completely obvious to an expert might carry zero pragmatic information for them, while the same message could be a revelation to a novice, carrying a huge amount of pragmatic information for that person.

The paper demonstrates that this definition holds up under mathematical scrutiny by proving that pragmatic information is directly linked to the efficiency of coding. In simpler terms, the amount of pragmatic information a message provides is equal to the number of bits saved when a receiver updates their understanding of the world. If a receiver knows the outcome of an event with certainty, they can describe that event using the fewest possible bits. If their knowledge is vague, they need more bits to describe the possibilities. When a message clarifies the situation, it reduces the number of bits needed to describe the outcome. The theory shows that the reduction in the number of bits required to encode the world, caused by receiving a message, is exactly equal to the pragmatic information gained. This provides a concrete, measurable justification for the definition, grounding the abstract idea of "meaning" in the physical reality of data compression.

One of the most striking applications of this theory is found in the classic problem of the "one-armed bandit," or a slot machine with an unknown payout rate. Imagine a player trying to decide whether to keep playing a machine. Initially, they have no idea if the machine pays out often or rarely. As they play, they observe wins and losses, and with each outcome, they update their estimate of the machine's payout probability. The paper calculates exactly how much pragmatic information each new spin provides. The result is counterintuitive to some: as the player plays more and more, the amount of new information gained from each additional spin decreases. Early spins are highly informative because they drastically change the player's belief about the machine. Later spins, after hundreds of trials, add very little new knowledge because the player already has a very accurate estimate. The theory quantifies this diminishing return, showing that the value of information is not constant but depends entirely on what the receiver already knows.

The framework also addresses the nature of "disinformation" and "noise." In standard information theory, a false message is often treated the same as a true one if it is equally complex. However, in the realm of pragmatic information, the distinction is crucial. A message that changes a receiver's beliefs is still considered to carry pragmatic information, even if those new beliefs are wrong. The theory separates the amount of information from the value of that information. A message can be highly informative in the sense that it changes a mind, but if that change leads to a bad decision, it is "pragmatically disinformative." The paper suggests that we can categorize messages into three groups: those that are irrelevant and change nothing, those that are useful and lead to better decisions, and those that are disinformative and lead to worse decisions. This distinction is vital for understanding real-world scenarios where false information can be just as effective at changing behavior as true information.

The author extends these ideas to the financial world, offering a fresh perspective on the Efficient Market Hypothesis, a cornerstone of economic theory. The traditional view suggests that asset prices reflect all available information, making it impossible to consistently beat the market. The paper reformulates this idea by suggesting that a market is efficient only for those participants who cannot extract any pragmatic utility from the available data. If a piece of market news changes a trader's beliefs and helps them make a better trade, that information is pragmatically useful to them. The hypothesis, therefore, is not that information is useless to everyone, but that for any given participant, the information available to them is either irrelevant or already processed. This explains why some investors, like those with advanced algorithms, can find value in data that is useless to others. It is not that the market is magically efficient, but that the computational capacity of the average participant limits their ability to turn raw data into pragmatic information.

Finally, the paper touches on the biological implications of this theory, connecting it to the evolution of species. In the context of evolution, the "decision-maker" is the interaction between a population of organisms and their environment. The environment sends "messages" in the form of survival pressures, and the population updates its "beliefs" about which traits are advantageous. The theory suggests that the rate of evolution is limited by the amount of pragmatic information the population can extract from these environmental messages. Just as a human trader has a limit to how much data they can process, a biological species has a limit to how much information it can reliably accrue before making errors. This leads to the concept of an "error catastrophe," where the mutation rate is so high that the species can no longer maintain the information necessary to survive. The paper posits that the computational capacity of the receiver—whether it is a human brain, a computer, or a biological genome—is the fundamental constraint on how much meaning can be extracted from the world.

In conclusion, this work provides a rigorous mathematical foundation for the intuitive idea that information is only as good as the change it produces in the receiver. By defining pragmatic information as the shift in belief that leads to a decision, the theory unifies concepts from statistics, computer science, and economics. It moves beyond the technical question of how much data is sent to the more practical question of how much that data matters. The findings suggest that the value of information is not an inherent property of the message itself, but a relationship between the message and the mind that receives it. This insight has profound implications for how we understand everything from gambling and stock trading to the very process of life and evolution, reminding us that in a world of infinite data, the true currency is the ability to make sense of it.

Technical Summary: Towards a Theory of Pragmatic Information

Problem Statement
Standard information theory, rooted in Shannon's framework, addresses the "technical problem" of transmitting symbols accurately but remains silent on the "semantics problem" (how precisely symbols convey meaning) and the "effectiveness problem" (how effectively a message changes conduct). While a non-technical observer might deem a message "informative" if it allows for the extraction of meaning, standard entropy measures do not quantify this meaning. Existing attempts to define "pragmatic information" have often been qualitative or lacked a rigorous derivation from fundamental assumptions regarding what constitutes a measure of meaning. This paper seeks to fill this gap by providing a rigorously justifiable, quantitative definition of pragmatic information: the amount of meaning in a message relevant to a specific decision.

Methodology and Conceptual Framework
The paper establishes a conceptual framework involving a decision maker, $\Delta$ (which may be a machine, human, or organism), and a "state of the world" represented by a random variable $\omega$ with a discrete sample space $\Omega$ .

Belief Update: $\Delta$ holds a prior probability distribution $q$ over $\omega$ . Upon receiving a message $m$ , $\Delta$ updates its beliefs to a posterior distribution $p_m$ .
Definition: The pragmatic information of a single message $m$ , denoted $D_\Delta(p_m || q)$ , is defined as the Kullback-Leibler (KL) divergence between the posterior and prior distributions:
$D_\Delta(p_m || q) = \sum_i p_{i|m} \log \left( \frac{p_{i|m}}{q_i} \right)$
This definition assumes base-2 logarithms, measuring information in bits.
Ensemble Definition: For an ensemble of messages $\mathcal{M}$ sampled according to a distribution $\phi_m$ , the pragmatic information $\Phi_\Delta(\mathcal{M}; \Omega)$ is defined as the expected value of the KL divergence over the ensemble:
$\Phi_\Delta(\mathcal{M}; \Omega) = E[D(p_M || q)] = \sum_{i,m} \phi_m p_{i|m} \log \left( \frac{p_{i|m}}{q_i} \right)$

Key Contributions and Theoretical Results
The paper justifies the proposed definition through several theorems and properties:

The "Wrong Code" Theorem: The paper proves that $D_\Delta(p_m || q)$ represents the expected difference between the shortest binary encoding of $\omega$ under the prior distribution $q$ and the posterior distribution $p_m$ . Specifically, $E[L_q(\omega) - L_{p_m}(\omega)] = D_\Delta(p_m || q)$ . This links pragmatic information directly to the reduction in coding length (uncertainty) required to describe the outcome after receiving the message.
Properties of the Measure:
- Non-negativity: $D_\Delta(p_m || q) \geq 0$ , with equality if and only if the message does not change the decision maker's beliefs ( $p_m = q$ ).
- Additivity: For independent decisions and "pragmatically independent" messages, the pragmatic information of an ensemble is the sum of the pragmatic information of its components.
- Convexity: The KL divergence is convex with respect to the probability distributions.
- Upper Bound: The maximum pragmatic information is bounded by the maximum of $-\log q_i$ , achieved when a message is "pragmatically definitive" (i.e., it leads to certainty about a specific outcome).
Relation to Mutual Information: The paper derives a relationship between pragmatic information and mutual information $I(\mathcal{M}; \Omega)$ :
$\Phi_\Delta(\mathcal{M}; \Omega) = I(\mathcal{M}; \Omega) + D_\Delta(p || q)$
Here, $p$ is the marginal distribution of outcomes. This equation characterizes pragmatic information as "free information," analogous to Helmholtz free energy in thermodynamics. Just as free energy is the portion of total energy available for useful work, pragmatic information is the portion of total information actually used to update beliefs and make decisions. The term $D_\Delta(p || q)$ accounts for the cost of updating prior biases.
Disinformation and Noise: The framework distinguishes between "pragmatically useful" messages, "pragmatically irrelevant" messages, and "pragmatic disinformation." Crucially, the definition of pragmatic information is non-negative regardless of whether the message is true or false; it measures the change in belief, not the accuracy or value of that change. Disinformation is simply a subset of messages that change beliefs but may lead to suboptimal decisions.

Applications and Examples
The paper sketches three specific applications to demonstrate the utility of the theory:

The One-Armed Bandit (Slot Machine): The theory is applied to a scenario where a decision maker must decide whether to play a slot machine with an unknown payout probability $\pi$ . Using Laplace's Rule of Succession, the paper calculates the pragmatic information gained from each trial. It demonstrates that as the number of trials $T$ increases, the pragmatic information per trial decreases toward zero, reflecting the diminishing returns of additional data once the payout probability is well-estimated.
Biological Evolution (Quasi-Species): The paper references the quasi-species model, where the interaction between replicators and their environment acts as the decision maker $\Delta$ . The "messages" are environmental details determining growth and mutation rates. The theory suggests that the rate of biological evolution can be characterized by the rate at which the system aggregates meaningful (pragmatic) information, subject to an "error catastrophe" limit imposed by computational capacity.
Efficient Market Hypothesis (EMH): The paper reformulates the EMH. Instead of claiming prices reflect all information, it posits that a market is efficient with respect to a specific participant if none of the available information is "pragmatically useful" to that participant. This allows for markets to be efficient for some agents (e.g., those with limited computational capacity) but not others (e.g., those with superior algorithms), explaining the coexistence of market efficiency and successful active investors.

Significance and Claims
The paper claims to provide the first rigorous, quantitative definition of pragmatic information derived from natural assumptions about decision-making and coding. By equating pragmatic information with the expected reduction in coding length (via the KL divergence), the author argues that this measure captures the "meaning" of a message in the context of a specific decision.

The author emphasizes that pragmatic information is distinct from the value of information (e.g., financial gain), as it measures the magnitude of belief change rather than the quality of the resulting decision. However, they note that pragmatic information is a necessary precursor to value; if a message is pragmatically irrelevant (zero KL divergence), it cannot be pragmatically useful.

Finally, the paper highlights the critical role of the receiver's computational capacity. It argues that pragmatic information is contingent upon the decision maker's ability to process the message. If a receiver lacks the computational power to update its beliefs (i.e., $p_m = q$ regardless of the message), the pragmatic information is zero, regardless of the message's complexity or length. This suggests that the limits of information processing are as fundamental to the theory of pragmatic information as channel capacity is to Shannon's theory.

More like this