💬 NLP

Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding

This pre-registered study demonstrates that the K-way energy probe's performance relative to softmax is heavily dependent on cross-entropy training, which induces significantly larger logit norms and a scale-invariant ranking advantage, rather than on bidirectional inference dynamics or latent movement magnitude.

Original authors: Jon-Paul Cacioli

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jon-Paul Cacioli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Detective Story

Imagine you have a detective (the AI) trying to solve a mystery (identifying a picture as a cat or a dog). The detective has two ways of reporting their confidence:

The Official Report (Softmax): A standard, mathematically perfect summary of how sure they are.
The Internal Feeling (Energy Probe): A "gut feeling" score based on how much the detective's brain had to work to settle on an answer.

In a previous study, researchers found that the "Official Report" was always much better at predicting the truth than the "Internal Feeling." It was like the detective's gut feeling was consistently wrong, even when they were right.

This new paper asks: Why is the gut feeling so weak? Is it because the detective's brain works a specific way, or is it because of the specific rules (math) we used to train them?

The answer turns out to be: It's the training rules. Specifically, a rule called "Cross-Entropy" (CE) is doing all the heavy lifting, and it's actually tricking the comparison.

The Three Experiments (The "What Ifs")

The researchers ran three different versions of the detective training to see what changed.

1. The Standard Detective (Baseline)

The Setup: The detective is trained using the standard, strict rules (Cross-Entropy).
The Result: The "Official Report" is a superstar. The "Internal Feeling" is a weakling. The gap between them is huge.
The Analogy: Imagine a student who is taught to memorize answers perfectly. When asked a question, they shout the answer loudly and confidently. Their "confidence score" is huge. But their "gut feeling" (which is based on a different, quieter process) looks small in comparison.

2. The "No-CE" Detective (Removing the Rule)

The Setup: They trained the detective using a softer, simpler rule (MSE) that doesn't force the answers to be shouted as loudly.
The Result: The gap shrank by half! The "Internal Feeling" got much closer to the "Official Report."
The Analogy: Now the student answers calmly. Because they aren't shouting their confidence so loudly, the "Official Report" doesn't look like a giant compared to the "gut feeling." They are more on equal footing.

3. The "Two-Way" Detective (Bidirectional PC)

The Setup: They tried a fancy new training method where the detective's brain sends signals both forward and backward (like a conversation between two people) instead of just one way. They also removed the loud-shouting rule.
The Result: The "Internal Feeling" actually beat the "Official Report" slightly!
The Twist: The researchers thought this was because the "Two-Way" brain was super complex. But they checked, and the brain wasn't actually working that much harder than the standard one. The real reason it won was simply that they stopped shouting the answers so loudly.

The "Volume Knob" Discovery (Logit Scale)

Here is the most important finding, explained with a volume knob.

The researchers realized that the standard training rule (Cross-Entropy) acts like a volume knob turned up to 11.

It forces the AI to make its "confidence numbers" (logits) huge.
When you turn the volume up, the "Official Report" sounds incredibly clear and confident.
The "Internal Feeling" doesn't get turned up; it stays at a normal volume.
The Illusion: Because the Official Report is so loud, it looks like it's doing a better job at predicting the truth. But it's just louder.

The Temperature Test:
To prove this, the researchers took the "Loud" AI and turned the volume knob down (using something called "Temperature Scaling") until it matched the "Quiet" AI.

Result: When the volume was turned down, the "Official Report" lost its massive advantage. The gap between the two methods shrank by 66%.
The Takeaway: Two-thirds of the reason the "Official Report" looked so good was just because it was louder.

What About the Other 34%?

Even after turning the volume down, the "Official Report" was still slightly better than the "Internal Feeling."

Why? Because the standard training rule (Cross-Entropy) is a very smart teacher. It teaches the AI to organize its thoughts in a way that naturally aligns with being correct. The "Internal Feeling" method just doesn't get that specific training.
The Analogy: Even if you turn the volume down, the student who memorized the answers perfectly (Standard Training) is still slightly sharper than the student who just guessed based on vibes (Energy Probe).

The "Bidirectional" Red Herring

The researchers originally hoped that the "Two-Way" brain (Bidirectional PC) would prove that complex brain dynamics were the key to better confidence.

The Reality Check: They tested this, and the "Two-Way" brain didn't actually move its internal gears any more than the standard brain. It was just a "quiet" brain.
Conclusion: The fancy brain structure wasn't the hero. The hero was simply turning off the loud-shouting rule.

Summary: The "Load-Bearing" Wall

The title "Cross-Entropy Is Load-Bearing" means that the standard training rule is holding up the entire structure of the experiment.

If you remove it, the whole theory changes.
The "Internal Feeling" isn't actually broken; it just looked bad because it was being compared to a "Loud" baseline that was artificially inflated.

In plain English:
The AI's "gut feeling" wasn't as bad as we thought. We were just comparing it to a "confidence score" that was artificially pumped up by the training method. Once we turned down the volume, the gut feeling looked much more impressive. The study suggests that when we judge how well an AI knows what it knows, we shouldn't just look at the loudest number; we need to compare apples to apples (or quiet voices to quiet voices).

1. Problem Statement

The paper investigates the validity and scope of a theoretical decomposition proposed in Cacioli [2026]. That prior work suggested that on standard discriminative Predictive Coding (PC) networks, a K-way energy probe (a structural method for measuring confidence) approximately reduces to a monotone function of the log-softmax margin. Consequently, the probe was predicted to perform slightly worse than a standard softmax baseline.

However, this decomposition relied on five specific assumptions, most notably:

A1: The presence of a Cross-Entropy (CE) loss term at the output.
A3: Effectively feedforward inference dynamics.

The central question of this study is whether assumption A1 (CE at the output) is a "load-bearing" component of this decomposition. Specifically, does removing CE (and potentially changing inference dynamics to bidirectional) alter the relationship between the energy probe and the softmax baseline?

2. Methodology

The study is a pre-registered scope test conducted on the CIFAR-10 dataset using a matched 2.1M-parameter backbone (TinyConv). The experimental design compares three conditions across 10 random seeds:

Condition A (Baseline): Standard Discriminative PC trained with Cross-Entropy (CE) loss. This replicates the original Cacioli [2026] setup.
Condition B (Isolation of A1): Standard Discriminative PC trained with Mean Squared Error (MSE) loss instead of CE. This isolates the effect of removing the CE term while keeping inference dynamics feedforward.
Condition C (Bidirectional PC - bPC): Implements the bidirectional PC architecture (Oliviers et al., 2025) using bidirectional MSE. This removes CE and introduces genuine bidirectional (iterative) inference dynamics, violating assumptions A1, A3, and A5 simultaneously.

Key Metrics:

Primary Outcome ( $\Delta$ ): The difference in Type-2 AUROC (Area Under the ROC Curve for discriminating correct vs. incorrect predictions) between the K-way energy probe and the softmax baseline ( $\Delta = \text{AUROC}_{\text{probe}} - \text{AUROC}_{\text{softmax}}$ ).
Manipulation Check (H3): Verified if bPC produced significantly greater latent movement (iterative dynamics) than standard PC (target threshold: 10x).
Post-hoc Ablation: Temperature scaling was applied to CE-trained models to test if logit magnitude differences drove the results.

3. Key Results

A. Replication and Baseline (H2)

The study successfully replicated the negative result of Cacioli [2026] on standard PC with CE.

Result: The energy probe consistently underperformed the softmax baseline ( $\Delta = -0.082$ , $p < 10^{-6}$ ).
Conclusion: The decomposition holds for standard CE-trained PC networks.

B. The Effect of Removing CE (H4)

Replacing CE with MSE in a standard PC architecture significantly reduced the performance gap.

Result: The gap narrowed from $-0.082$ (CE) to $-0.037$ (MSE), a 54% reduction.
Implication: The CE loss function is a major empirically load-bearing component of the decomposition.

C. Bidirectional PC Performance (H1)

Contrary to the hypothesis that bidirectional dynamics would drive the probe's performance, the bPC condition showed the probe exceeding the softmax baseline.

Result: $\Delta = +0.008$ ( $p = 0.000027$ ). The probe outperformed softmax across all 10 seeds.
Crucial Caveat (Manipulation Check Failure): The pre-registered check for "strong dynamics" failed. bPC latent movement was only 1.6x that of standard PC (far below the 10x threshold). The generative chain contributed less than 1% of the total energy.
Interpretation: The improvement in bPC is not due to bidirectional dynamics. Instead, it is attributed to the removal of CE, which bPC shares with the MSE condition.

D. The Logit-Scale Mechanism (Post-hoc Analysis)

The study identified a systematic difference in output logit norms:

CE Training: Produced logit norms $\approx 11.5$ .
MSE/bPC Training: Produced logit norms $\approx 0.78$ (a 15x difference).
Mechanism: Large logits in CE training sharpen the softmax distribution, artificially inflating the softmax baseline's AUROC. The energy probe, operating on generative chain errors, is unaffected by this logit scaling.

Temperature Scaling Ablation:
By applying temperature scaling ( $T \approx 15$ ) to CE-trained models to match the logit norms of MSE models:

The probe-softmax gap decreased by 66% (from -0.080 to -0.027).
Decomposition of the Gap:
1. ~66% (Scale-dependent): Attributable to logit inflation caused by CE.
2. ~34% (Scale-invariant): A genuine ranking advantage where CE-trained representations produce logits better correlated with correctness than the energy probe, independent of scale.

4. Key Contributions

Scope Definition: Establishes that the "probe underperforms softmax" finding is specific to CE-trained discriminative PC. It does not generalize to MSE-trained or bidirectional PC architectures.
Mechanistic Insight: Identifies Cross-Entropy as the primary driver of the performance gap, not the architecture itself. CE induces logit inflation that unfairly advantages the softmax baseline in Type-2 discrimination tasks.
Refutation of Dynamics Hypothesis: Demonstrates that at the tested scale (TinyConv), bidirectional dynamics do not materially alter latent movement or probe performance; the observed effects are purely due to the loss function change.
Methodological Recommendation: Proposes that temperature-scaled softmax is a fairer baseline for comparing structural probes against standard classifiers on CE-trained networks, as raw softmax is biased by logit magnitude.

5. Significance

This paper refines the understanding of confidence measurement in neural networks. It suggests that the perceived "failure" of structural energy probes to beat softmax is largely an artifact of the training objective (CE) rather than a fundamental limitation of the probe's architecture.

For Predictive Coding: It clarifies that the theoretical decomposition linking energy margins to log-softmax breaks down when CE is removed, as the mathematical link between settled energy and log-softmax probability no longer holds.
For Metacognition Research: It highlights that how a network is trained to express confidence (via CE vs. MSE) matters as much as its architecture.
For Evaluation Standards: It argues for the use of temperature-scaled baselines to ensure fair comparisons between structural probes and standard confidence signals, preventing logit-scale inflation from masking the true efficacy of alternative confidence mechanisms.

In summary, Cross-Entropy is "load-bearing" for the specific decomposition observed in prior work; removing it allows the K-way energy probe to match or exceed softmax performance, primarily by eliminating the logit-scale bias that artificially boosts the softmax baseline.