Bayesian Hierarchical Models and the Maximum Entropy Principle

This paper demonstrates that when the prior given hyperparameters in a Bayesian hierarchical model is a canonical maximum entropy distribution, the resulting dependent marginal prior also possesses a maximum entropy property, but with respect to a constraint on the marginal distribution of a function of the unknown parameters, thereby clarifying the implicit information assumptions of such models.

The Gist

Here is an explanation of the paper using simple language and creative analogies.

The Big Idea: The "Hidden Chef" of Statistics

Imagine you are a chef trying to create a recipe for a giant pot of soup (the data). You have $n$ ingredients ($x_1, x_2, \dots, x_n$), but you don't know exactly what they are yet. You need to decide on a "prior belief"—a starting guess about what the soup might taste like before you actually taste it.

Usually, statisticians use a Maximum Entropy rule. Think of this as the "Principle of Least Assumption." It says: "Don't guess anything you don't have to. Just spread your possibilities as evenly as possible, unless you have a specific rule to follow."

For example, if you know the average temperature of the soup must be 50°C, the "Maximum Entropy" rule gives you a very specific, standard recipe (called a canonical distribution) that fits that rule perfectly.

The Problem:
Sometimes, we don't actually know the exact average temperature. We just have a feeling about it. Maybe we think the average is likely around 50, but it could be 40 or 60.

In the past, statisticians would say: "Okay, let's pretend we do know the average is 50. We'll use that standard recipe, but we'll add a 'secret sauce' (a hyperparameter) that we don't know the exact amount of. We'll guess a range for that secret sauce and mix it all together."

This creates a Hierarchical Model. It's like saying, "I don't know the exact temperature, so I'll assume the temperature is a random variable drawn from a distribution of temperatures."

The Confusion:
The author, Brendon Brewer, noticed something weird. When you mix all those different "standard recipes" together (because the temperature is unknown), the final result is a messy "mixture." It doesn't look like a clean, standard Maximum Entropy recipe anymore.

People thought: "Oh no! We lost the Maximum Entropy principle. This messy mixture isn't the most 'ignorant' or 'fair' guess anymore."

The Discovery:
Brewer says: "Wait a minute! You haven't lost the principle. You just changed the rule."

He proves that this messy mixture IS actually a Maximum Entropy distribution. It just isn't following the rule about the average temperature. Instead, it is following a rule about the shape of the distribution of the average temperature itself.

---

The Analogy: The "Indirect" Rule

Let's use a metaphor to make this crystal clear.

Scenario A: The Direct Rule (Standard MaxEnt)

Imagine you are a teacher grading a class of 100 students.

• The Rule: "The class average must be exactly 75."
• The Result: You assign grades to every student in a very specific, standard way (a bell curve) to ensure that average is hit. This is the "Canonical Distribution."

Scenario B: The Indirect Rule (The Hierarchical Model)

Now, imagine you don't know what the class average should be. You only know that the "ideal average" is a mystery.

• The Setup: You decide that the "ideal average" could be anywhere between 60 and 90, and you have a hunch about how likely each number is (maybe 75 is most likely, but 60 is possible).
• The Process: You imagine a thousand different classrooms, each with a different fixed average (60, 61, 62... 90). For each classroom, you assign grades using the "Standard Rule" from Scenario A. Then, you mix all those classrooms together into one giant pile of grades.
• The Result: The final pile of grades looks messy. It's not a perfect bell curve.

The Old View: "This messy pile is bad. It's not a clean Maximum Entropy solution."

Brewer's New View: "Actually, this messy pile IS the perfect Maximum Entropy solution, but for a different question."

Instead of asking, "What is the distribution of grades if the average is 75?"
You are now asking, "What is the distribution of grades if I have a specific uncertainty about what the average is?"

Brewer shows that the messy pile is the most "honest" (maximum entropy) way to distribute grades given that you have a specific belief about how the class average behaves.

The "Indicator" Trick (How it works mathematically)

The paper uses a clever mathematical trick to prove this.

Imagine you have a function $T$ (like the class average).

1. Standard MaxEnt says: "I know the average is 75." So, I adjust the grades to hit that target.
2. Hierarchical MaxEnt says: "I don't know the average, but I know the probability of the average being 60, 70, or 80."

Brewer proves that if you take the "Standard" recipe and mix it according to your beliefs about the average, the result is mathematically identical to taking the original "ignorant" recipe and applying a new rule: "Make sure the final grades look like they came from a specific distribution of averages."

It's like saying:

• Old Way: "I know the soup is 50°C."
• New Way: "I don't know the temperature, but I know the distribution of temperatures I'm willing to accept. Therefore, the soup must look like this specific mixture."

Why Does This Matter?

This is a huge deal for two reasons:

1. It Validates the "Messy" Models: Scientists often use these hierarchical models because they are practical. They are easier to build when you don't know the exact numbers. Before this paper, people worried they were breaking the "Maximum Entropy" rules. Now we know they are not breaking the rules; they are just applying the rules to a different, more complex constraint.
2. It Explains What We Are Assuming: When you build a hierarchical model, you aren't just guessing numbers. You are explicitly stating: "I believe the underlying parameters (like the average) follow this specific pattern." Brewer's work tells us exactly what information we are putting into the model.

Summary in One Sentence

Brendon Brewer proves that when you use a "hierarchical" model (where you guess the rules for your rules), you aren't abandoning the "Maximum Entropy" principle; you are simply applying it to the uncertainty of the rules themselves rather than the rules directly.

It's like realizing that while you can't predict the exact weather, you can predict the most "fair" distribution of weather patterns if you know how much you trust your weather forecast.

Technical Summary

Here is a detailed technical summary of the paper "Bayesian Hierarchical Models and the Maximum Entropy Principle" by Brendon J. Brewer.

1. Problem Statement

In Bayesian data analysis, Bayesian hierarchical models are frequently used to assign priors to unknown parameters ($x$) by introducing hyperparameters ($\alpha$). This approach is often justified by the principle of exchangeability, where the joint prior is constructed as a mixture of independent and identically distributed (i.i.d.) conditional distributions integrated over a prior on the hyperparameters:
$$p(x) = \int p(\alpha) \prod_{i=1}^n p(x_i | \alpha) d\alpha$$

However, a theoretical disconnect exists regarding the Maximum Entropy (MaxEnt) principle.

• The MaxEnt principle typically yields a canonical distribution (e.g., exponential family) when constraints are placed on the expected values of certain functions (e.g., $\langle T \rangle = \mu$).
• If the expected values (constraints) are unknown, a common practice is to treat the canonical distribution as conditional on unknown hyperparameters and integrate them out.
• The Conflict: The resulting marginal distribution $p(x)$ is a mixture of canonical distributions. Since a mixture of canonical distributions is generally not itself a canonical distribution, it was previously unclear whether this hierarchical marginal prior retains any Maximum Entropy interpretation. The paper seeks to resolve whether a MaxEnt justification exists for these hierarchical models and, if so, what the effective constraint is.

2. Methodology

The author employs a theoretical derivation to bridge the gap between hierarchical modeling and the MaxEnt principle. The methodology involves:

1. Generalizing MaxEnt Constraints: Moving beyond standard constraints on expected values (moments) to constraints on the marginal distribution of derived quantities ($T = f(x)$).
2. Deriving the General Form:
   • Starting with a base prior $\pi(x)$ and a derived quantity $T = f(x)$.
   • Imposing a constraint on the probability distribution of $T$ (e.g., specifying $P(T=t)$ or the shape of the distribution of $T$).
   • Showing that the MaxEnt solution for $x$ under this constraint takes the form:
     $$p(x) \propto \pi(x) g(f(x))$$
     where $g(\cdot)$ is a non-negative function determined by the desired marginal distribution of $T$.
3. Connecting to Hierarchical Models:
   • Considering a hierarchical model where the conditional prior $p(x|\lambda)$ is a canonical distribution:
     $$p(x|\lambda) \propto \pi(x) \exp\left(\sum \lambda_i f_i(x)\right)$$
   • Integrating out the Lagrange multipliers (hyperparameters) $\lambda$ with a prior $p(\lambda)$:
     $$p(x) = \int p(\lambda) p(x|\lambda) d\lambda$$
   • Demonstrating that because the integral depends on $x$ only through the sufficient statistics $\{f_i(x)\}$, the resulting marginal $p(x)$ matches the form $\pi(x)g(f_1(x), \dots, f_m(x))$.
4. Case Studies: The paper validates this theory using two specific examples:
   • Exponential Example: A uniform prior on $x$ leads to a narrow implied prior on the arithmetic mean $T$ via the Central Limit Theorem. By placing a log-uniform prior on the mean parameter $\mu$ in a hierarchical model, the author shows the resulting marginal is a MaxEnt distribution constrained on the marginal distribution of $T$.
   • Gaussian Example: A hierarchical model with unknown mean ($\mu$) and variance ($\sigma$) is shown to produce a marginal prior that is a MaxEnt distribution constrained on the marginal distributions of the sum ($T_1$) and sum of squares ($T_2$).

3. Key Contributions

• Reconciliation of Hierarchical Models and MaxEnt: The paper proves that the marginal prior resulting from a hierarchical model (where the conditional prior is canonical) is a Maximum Entropy distribution.
• Identification of the Effective Constraint: The paper identifies that the implicit constraint in such hierarchical models is not on the expected values of the parameters, but rather on the marginal distribution of the derived quantities (sufficient statistics) themselves.
• Theoretical Unification: It clarifies that the "Maximum Entropy on the Mean" approach and "Superstatistics" are specific instances of this broader principle, where the integration over hyperparameters effectively shapes the prior on the sufficient statistics.

4. Results

• Mathematical Equivalence: The author establishes that integrating out hyperparameters in a canonical hierarchical model is mathematically equivalent to applying the MaxEnt principle with a constraint on the distribution of the sufficient statistics $T = f(x)$.
• Form of the Distribution: The resulting marginal distribution is of the form $p(x) \propto \pi(x)g(T)$, where $g(T)$ is determined by the chosen prior on the hyperparameters.
• Practical Implications (Figures 1 & 2):
  • In the Exponential Example, a naive uniform prior on data leads to an unintended, narrow Gaussian prior on the mean. Using a hierarchical model with a log-uniform prior on the mean parameter corrects this, resulting in a log-uniform-like prior on the mean, which is a valid MaxEnt solution for that specific constraint.
  • In the Gaussian Example, the hierarchical model allows for "appropriate prior uncertainty" regarding the sum and sum of squares of the data, avoiding the artificial narrowing caused by fixed hyperparameters.

5. Significance

• Justification of Hierarchical Priors: This work provides a rigorous theoretical justification for using hierarchical models. It shows that they are not merely heuristic tools for exchangeability but are grounded in the Maximum Entropy principle, provided one interprets the constraint correctly.
• Clarification of Information Content: It sheds light on what information is actually being assumed when a hierarchical model is assigned. The model assumes a specific shape for the distribution of the derived quantities (e.g., the mean or variance of the group), rather than fixing their expected values.
• Guidance for Model Building: Practitioners can now understand that choosing a prior for hyperparameters is equivalent to choosing a constraint on the marginal distribution of sufficient statistics. This offers a principled way to select hyperparameter priors to achieve desired properties in the marginal prior of the data or parameters.
• Resolution of "Lost" Entropy: It resolves the apparent paradox that mixing canonical distributions destroys the MaxEnt property, demonstrating instead that the property is preserved under a different, more general type of constraint.