🧬 biology

Theory of Subjective Entropy: Cognition as compression of information

This paper proposes the Theory of Subjective Entropy, which formalizes randomness as a relationship between data and an agent's internal model, demonstrating through experiments that the Context-Tree Weighting algorithm's measure of compressibility predicts human sequence prediction accuracy better than traditional frequency-based models.

Original authors: Mikael Lindmark Moesgaard, Jakob Kargo Mortensen

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mikael Lindmark Moesgaard, Jakob Kargo Mortensen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine your brain is a super-smart detective trying to solve a mystery, but the mystery is just a stream of random events, like a coin flipping over and over again. For decades, scientists have looked at these events using a tool called "Shannon entropy," which basically asks: "Is this coin fair?" If the coin is fair, the answer is "yes, it's totally random," and the detective gives up, assuming there's no pattern to find. But what if the coin isn't the problem? What if the problem is the detective's notebook? This paper dives into a fascinating corner of science called Algorithmic Information Theory and Active Inference. Think of Algorithmic Information Theory as the study of how much "compression" you can do on a file—like zipping a folder to make it smaller. If you can zip it up tight, there's a pattern. If it stays huge, it's truly random. Active Inference is the idea that our brains are constantly trying to shrink their own "surprise" by predicting what happens next. We care about this because it changes how we see the world: maybe the universe isn't chaotic, but our internal models just aren't good enough to see the hidden order.

The authors, Mikael Lindmark Moesgaard and Jakob Kargo Mortensen, propose a new idea called the Theory of Subjective Entropy. Their big claim is that "randomness" isn't just a property of the coin or the source; it's a relationship between the sequence of events and the internal model of the person watching it. To test this, they ran two online experiments with 159 participants. The participants played a game where they had to guess the color of a square (blue or orange) that appeared on their screen. The squares followed a binary sequence (like 1s and 0s) that was generated to be globally random—meaning if you looked at the whole thing, it looked like a fair coin toss with no patterns. However, the researchers used a special math tool called the Context-Tree Weighting (CTW) algorithm to slice these sequences into tiny chunks. Some chunks were "sub-entropic," meaning they had hidden, compressible patterns that the CTW model could spot. Other chunks were "super-entropic," meaning they were truly messy and incompressible.

Here is the twist: even though the whole sequence was random, the participants didn't guess randomly. The study found that when the sequence fell into a "sub-entropic" region (a chunk with a hidden pattern), the participants were significantly better at predicting the next color. In fact, the more "compressible" a chunk was (the more the CTW model could shrink it), the higher the participants' accuracy. The paper suggests that humans are acting like these CTW models, constantly trying to find the shortest "zip file" for the data they see. When they find a pattern, they get better at guessing.

Crucially, the authors ruled out a few simpler explanations. They showed that the participants weren't just counting how many blues or oranges had appeared so far (a simple frequency count); they were actually detecting deeper, more complex structures. They also compared their results to a "fair coin" baseline and found that while the overall accuracy hovered near 50% (chance), the local accuracy in patterned regions was clearly higher. The paper suggests that this happens because our brains are driven by a principle called free energy minimization—basically, our brains hate being surprised, so they actively hunt for patterns to compress the information and make the world feel more predictable.

The study measured this with high precision. Across 89,520 trials, they identified over 22,000 "sub-entropic" regions and 22,000 "super-entropic" regions. They found that for every extra bit of "compression" (a measure of how much the pattern could be shrunk) in a sub-entropic region, the odds of a correct guess went up. In super-entropic regions, where no pattern existed, the participants' accuracy stayed near chance, and interestingly, they seemed to stop trying to find patterns as the messy regions got longer, suggesting they were smart enough to know when to give up.

So, what does this mean? It suggests that even in a world that looks totally random, our brains are constantly scanning for the "zip code" of reality. We don't just see noise; we see potential structure. If a sequence has a hidden pattern, even a tiny one, our internal models can latch onto it and improve our predictions. The paper concludes that "randomness" is subjective: a sequence might be random to a simple observer but full of order to a smarter one. This framework doesn't just explain how we guess colors on a screen; it offers a new way to think about how all cognitive systems process information, suggesting that the very act of thinking is the act of compressing the universe into something we can understand.

Technical Summary: Theory of Subjective Entropy

Problem Statement

Classical information theory, specifically Shannon entropy, defines randomness based on the abstract source distribution generating a sequence. However, this framework is insufficient for explaining how cognitive agents perceive and predict finite sequences, particularly when a sequence generated by a random source (e.g., a fair coin) contains locally compressible structures that an agent can exploit. The authors argue that apparent randomness is not an intrinsic property of the source alone but a relational property between the sequence and the agent's internal model. The central problem addressed is the lack of a formal, computable framework to quantify "subjective entropy"—the degree to which a sequence is compressible relative to an agent's specific internal model—and to explain how this relates to predictive accuracy in human cognition.

Methodology

Theoretical Framework

The paper unifies Algorithmic Information Theory (AIT), Classical Information Theory, and Active Inference.

Subjective Entropy: Defined as the model-relative code length (or algorithmic complexity) of a finite binary sequence. A sequence is "subjectively random" if it is incompressible under the agent's internal model.
Compression Threshold ( $\tau$ ): Set at 1 bit/bit (the entropy rate of a fair coin). Sequences with an average code length $K < 1$ are sub-entropic (compressible); those with $K \ge 1$ are super-entropic (incompressible).
The CTW Algorithm: The authors operationalize subjective entropy using the Context-Tree Weighting (CTW) algorithm. CTW serves as a computable approximation of Solomonoﬀ Induction, providing a Bayesian mixture over all context trees up to a finite depth $D$ . It incorporates Occam's Razor (penalizing deeper trees) and Epicurus' Principle (retaining all consistent models).
Entropic Regions: The authors introduce a novel unit of analysis: entropic regions. These are maximal contiguous subsequences where the trial-wise CTW log-loss ( $K_t$ ) remains consistently below or above the threshold $\tau$ . This allows for the identification of locally compressible structures within globally random sequences.
Active Inference Link: The framework posits that agents minimize variational free energy. Minimizing free energy corresponds to maximizing predictive accuracy (accuracy term) while controlling model complexity. In this context, compressing a sequence (reducing $K_{CTW}$ ) is equivalent to minimizing free energy, providing a normative explanation for why agents should exploit local structure.

Experimental Design

Two online experiments (A and B) were conducted with a combined sample of 159 participants (148 included in the final analysis) via the Prolific platform.

Task: Participants predicted the color (blue/1 or orange/0) of a square in a forced-choice binary sequence prediction task with feedback.
Stimuli: Sequences were generated to have near-maximal Shannon entropy (globally random) but were engineered to contain a balanced number of sub-entropic and super-entropic regions based on the CTW model.
- Experiment A used a generation procedure where complexities clustered near the threshold.
- Experiment B employed an $\epsilon$ -band around the threshold to ensure a symmetric distribution of sub- and super-entropic regions.
Analysis: The study utilized Generalized Linear Mixed-Effects Models (GLMM) with Bayesian and frequentist approaches.
- Trial-level analysis: Tested if per-trial CTW complexity ( $K_t$ ) predicted correctness, controlling for a simple Krichevsky-Trofimov (KT) estimator (0-order Markov model).
- Region-level analysis: Tested if the cumulative compressibility (Area, $A_r$ ), depth ( $d_r$ ), and length ( $L_r$ ) of entropic regions predicted accuracy and model-human agreement.

Key Contributions

Formalization of Subjective Entropy: The paper proposes a rigorous definition of randomness as a property of the relation between a sequence and an internal model, rather than the source itself. It operationalizes this using CTW complexity.
Entropic Regions: The introduction of "entropic regions" allows for the decomposition of globally random sequences into locally compressible (sub-entropic) and incompressible (super-entropic) segments. This provides a geometric interpretation of compressibility ( $Area = Length \times Depth$ ).
Mechanistic Explanation via Active Inference: The authors link algorithmic compression to the Free Energy Principle, arguing that exploiting sub-entropic structure is a rational strategy for minimizing free energy (maximizing predictive fit).
Universal Applicability: The framework is presented as a universal formal language for quantifying information processing in cognitive systems, applicable to any binary sequence regardless of the source, provided the sequence can be described in binary terms.

Results

Trial-Level Findings

Complexity Predicts Accuracy: Higher per-trial CTW complexity ( $K_t$ ) significantly predicted a lower probability of a correct response.
Beyond Frequency Tracking: Control models using the KT-estimator (which tracks only symbol frequencies) showed that while frequency tracking explains some variance, the CTW-specific gain (the additional compression provided by deeper context trees) significantly improved the prediction of trial-level correctness. This indicates humans utilize deeper structural patterns, not just simple counts.

Region-Level Findings

Sub-Entropic Advantage: In sub-entropic regions, greater cumulative compressibility (Area, $A_r$ ) and greater depth ( $d_r$ ) significantly predicted higher predictive accuracy. The effect of region length was also positive, suggesting that predictability compounds over longer spans of structure.
Super-Entropic Disadvantage: In super-entropic regions, the relationship was different. Longer super-entropic regions were associated with decreased model-human agreement. This suggests that as agents persist in a misleading (super-entropic) region, they may eventually revert to a baseline model (e.g., a fair coin guess) or shift models, reducing adherence to the CTW prediction rule.
Geometric Interpretation: The interaction between length and depth confirmed that the benefits of compressibility are most pronounced when deep structure is sustained over longer regions.

Model Comparison

The CTW-based models significantly outperformed both the simple KT-estimator and an intercept-only (chance-level) baseline in explaining the data, as evidenced by lower AIC/BIC scores and significant likelihood-ratio tests.

Significance and Claims

The paper claims that the Theory of Subjective Entropy offers a more informative approach to understanding sequence prediction than classical information theory alone.

Local vs. Global Randomness: The findings support the claim that even globally random sequences (high Shannon entropy) contain locally compressible structures that are not subjectively random to an agent with a sufficiently complex internal model.
Normative Basis: By anchoring the theory in Active Inference, the paper provides a normative explanation for why agents should and do exploit these local structures: doing so minimizes free energy.
Methodological Implications: The authors caution that in experimental psychology, uncontrolled variations in subjective entropy across trial sequences could lead to false positives. If an experiment uses sequences with hidden sub-entropic regions, participants may perform above chance not due to the experimental manipulation but due to the exploitability of the sequence structure.
Universal Language: The paper concludes modestly, positioning the theory as a "modest step" toward a universal account of cognition as the compression of information. It asserts that by formalizing concepts like patterns and structure through binary sequences and algorithmic complexity, cognitive psychology can be anchored in first principles.

The authors explicitly state that the framework is intended not merely as an account of sequence prediction but as a universal formal language for quantifying randomness and information processing in cognitive systems, provided these processes can be described in terms of binary sequences.