📊 statistics

Regularized Exponentially Tilted Empirical Likelihood for Bayesian Inference

This paper proposes a regularized exponentially tilted empirical likelihood method that overcomes the convex hull constraint in Bayesian inference by incorporating a continuous exponential family distribution via pseudo-data, thereby ensuring the posterior covers the full parameter space while maintaining desirable asymptotic properties and improved finite-sample performance.

Original authors: Eunseop Kim, Steven N. MacEachern, Mario Peruggia

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eunseop Kim, Steven N. MacEachern, Mario Peruggia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Fence" That Traps You

Imagine you are a detective trying to find a suspect's location (the parameter) based on a few clues (the data). In the world of statistics, there is a popular tool called Empirical Likelihood (EL) that helps detectives build a "map" of where the suspect is likely to be.

However, this tool has a weird, frustrating flaw called the Convex Hull Constraint.

Think of your clues as stones scattered on a beach. The "Convex Hull" is the shape of a rubber band stretched tight around all those stones.

The Rule: The tool only works if the suspect's location is inside the rubber band.
The Problem: If the stones are scattered in a way that leaves a big empty space in the middle (or if the suspect is actually outside the band), the tool simply says, "I can't calculate anything here." It assigns a zero probability to that area.

In Bayesian inference (a method where you combine your clues with your prior beliefs), this is a disaster.

You start with a belief that the suspect could be anywhere on the beach.
You look at the data.
The tool suddenly says, "The suspect cannot be in this huge empty zone!" and erases that part of your map.
Even if you get more data later, the tool might still struggle to "un-erase" that zone. It's like a GPS that says, "You are here," but then draws a wall around you and refuses to let you move, even if the road ahead is clear.

The Old Fix: "Fake Clues" (Pseudo-data)

Previous attempts to fix this involved adding fake clues (pseudo-data) to the mix. Imagine the rubber band is too small, so you sneak in a few extra stones to stretch the band wider until it covers the suspect.

The Issue: These fake stones had to be placed differently depending on where you thought the suspect was. If you moved your guess, you had to move the fake stones. This made the math messy, irregular, and not very "Bayesian" (it felt like cheating).

The New Solution: "The Invisible Safety Net" (RETEL)

The authors of this paper propose a new method called Regularized Exponentially Tilted Empirical Likelihood (RETEL). Instead of adding specific fake stones, they add an invisible safety net.

Here is how it works, using a metaphor:

1. The "Ghost" Distribution

Instead of adding fake stones, imagine you have a "Ghost" distribution—a smooth, continuous cloud of probability (like a fog) that exists everywhere, even where there are no real stones.

This fog represents a regularization term. It's a gentle, mathematical force that says, "Even if the real data doesn't reach this spot, don't worry, there's a tiny bit of probability here because of our safety net."

2. The Limit of Infinite Fake Data

The authors realized that if you add more and more fake stones (pseudo-data) until you have an infinite number of them, they stop looking like individual stones and start looking like that smooth fog.

By taking this "limit," they created a method that doesn't need to manually place fake stones for every guess. The "fog" is always there, ensuring the rubber band never snaps or leaves a hole.

3. The "Tilt"

The method uses a technique called Exponential Tilting. Imagine the rubber band is a piece of fabric. If the data pulls it one way, the fabric stretches. The "Ghost" fog ensures that even if the fabric is pulled tight, it never tears or hits a hard wall. It allows the probability to flow smoothly everywhere, ensuring the map never has "zero probability" zones.

Why This Matters (The Results)

The paper tests this new method against the old ones using computer simulations and real-world data (like income levels and ancient Egyptian lifespans).

Better Maps: The new method (RETEL) produces maps that look much more like the "true" answer, especially when you don't have a lot of data (small sample sizes).
No More Black Holes: It eliminates the "zero probability" zones. The suspect can be anywhere, and the math works smoothly.
Stability: It's more robust. If the data is weird or the sample size is tiny, the old methods might crash or give wild answers, but RETEL stays steady.
The "Unit Information" Concept: The authors treat their safety net as if it were just one extra piece of data. This is a clever way to say, "We are adding a tiny bit of help, but not so much that we overwhelm the real clues."

The Takeaway

Imagine you are trying to draw a map of a city based on a few street signs.

Old Method: If the signs don't cover a certain neighborhood, your map says "This neighborhood doesn't exist."
Old Fix: You draw fake signs to cover the gap, but you have to redraw them every time you move your pen.
New Method (RETEL): You have a faint, transparent grid over the whole city. Even if there are no signs in a neighborhood, the grid tells you, "It's probably here, just not 100% sure."

This new method makes Bayesian statistics more reliable, smoother, and less prone to breaking when data is scarce or messy. It turns a jagged, broken map into a complete, navigable one.

1. Problem Statement

The paper addresses a fundamental challenge in applying Empirical Likelihood (EL) and Exponentially Tilted Empirical Likelihood (ETEL) within a Bayesian framework.

The Convex Hull Constraint: In EL and ETEL, the likelihood is defined only when the zero vector lies within the convex hull of the estimating functions $\{g(X_i, \theta)\}$ . This restricts the posterior domain to a proper subset of the parameter space ( $\Theta_n = \{ \theta : 0 \in \text{Conv}_n(\theta) \}$ ).
Bayesian Inconsistency: This constraint causes the posterior probability to be exactly zero for certain parameter values, even if the prior assigns them positive probability. As sample size increases, these "zeroed-out" regions may reappear, violating the continuity and logical consistency expected in Bayesian inference.
Limitations of Existing Solutions: Previous methods, such as Adjusted Empirical Likelihood (AEL) and Adjusted ETEL (AETEL), attempt to fix this by adding a single pseudo-observation that depends on $\theta$ . While this satisfies the convex hull constraint, it introduces irregularities, parameter-dependent artifacts, and deviates from standard Bayesian principles by treating pseudo-data as actual observations.

2. Methodology

The authors propose a Regularized Exponentially Tilted Empirical Likelihood (RETEL) that eliminates the convex hull constraint through a novel regularization technique derived from a limiting procedure.

A. Weighted ETEL with Fractional Pseudo-Data (WETEL)

As an intermediate step, the authors introduce WETEL, which incorporates $m$ pseudo-data points with fractional weights.

Instead of adding one pseudo-observation (as in AEL), they add $m$ points.
They utilize an entropy balancing scheme (Hainmueller, 2012) to reweight the data.
The weights are assigned such that the $n$ observed data points have weight $1/(n+1)$ , and the $m$ pseudo-data points collectively share the remaining weight, effectively treating the pseudo-data as a single fractional unit.
This approach extends the convex hull but still relies on finite $m$ , meaning the constraint issue can persist if $m$ is not large enough or if the pseudo-data specification is poor.

B. Regularized ETEL (RETEL)

The core contribution is the limit of WETEL as $m \to \infty$ .

Limiting Procedure: The pseudo-data are modeled as samples from a continuous exponential family distribution (specifically a normal distribution $N(\mu, \Sigma)$ ). As $m \to \infty$ , the discrete sum of pseudo-data converges to the moment-generating function of this continuous distribution.
Regularization Term: The minimization problem for the Lagrange multiplier $\lambda$ changes from a finite sum to:
$\min_{\lambda} \left\{ \sum_{i=1}^n \exp(\lambda^\top g_i(\theta)) + \tau_n \exp\left(\lambda^\top \mu_{n,\theta} + \frac{1}{2}\lambda^\top \Sigma_{n,\theta}\lambda\right) \right\}$
The second term acts as a penalty derived from the Kullback–Leibler (KL) divergence between the empirical distribution and a mixture of the empirical distribution and the auxiliary continuous distribution.
Dual Interpretation: RETEL is shown to be the dual of minimizing the KL divergence between a mixture of the empirical data and the auxiliary distribution, subject to moment constraints. This ensures the optimization problem is strictly convex and has a unique, finite solution for all $\theta$ , regardless of whether the convex hull constraint is satisfied by the observed data.
Two Versions:
1. RETEL $_f$ (Full): Includes the probability mass of the auxiliary distribution in the likelihood ratio.
2. RETEL $_r$ (Reduced): Drops the auxiliary mass term from the likelihood ratio but retains its effect on the weights $p_i(\theta)$ .

3. Key Contributions

Resolution of the Convex Hull Constraint: RETEL removes the zero-probability regions in the posterior without relying on parameter-dependent pseudo-observations that vary irregularly. It ensures the posterior support is the entire parameter space.
Theoretical Guarantees:
- Wilks' Theorem: The authors prove that $-2 \log R_{RET}(\theta_0)$ converges in distribution to a $\chi^2_p$ distribution, preserving the asymptotic properties of standard EL.
- Bernstein–von Mises (BvM) Theorem: They establish that the posterior distribution of $n^{1/2}(\theta - \theta_0)$ converges to a normal distribution $N(0, \Omega)$ , ensuring valid Bayesian inference asymptotically.
Improved Finite Sample Performance: Unlike AEL/AETEL, which can suffer from boundedness issues or irregularity, RETEL provides stable estimates and credible intervals even in small samples.
Flexibility: The framework allows for the incorporation of prior information through the choice of the auxiliary distribution parameters ( $\mu_{n,\theta}, \Sigma_{n,\theta}$ ) and the regularization strength $\tau_n$ . It can also be extended to non-normal auxiliary distributions (e.g., Bernoulli).

4. Results

The paper validates RETEL through extensive simulations and real-data applications.

Simulation Studies

Posterior Coverage: In simulations with small sample sizes ( $n=5$ ), RETEL produces posterior intervals with coverage probabilities much closer to the nominal level (e.g., 95%) compared to ETEL and AETEL. ETEL often undercovers due to the convex hull constraint, while AETEL tends to overcover with excessively wide intervals.
Expected KL Divergence: RETEL preserves the monotonicity of information gain (Expected KL divergence increases with sample size), whereas ETEL and EL can exhibit non-monotonic behavior or inflated information values due to the "collapse" of the posterior domain in small samples.
High-Dimensional Confidence Regions: In settings where the number of parameters $p$ is large relative to $n$ ( $p/n$ ratios up to 1/2), RETEL maintains stable coverage rates (often >80-90%), while AETEL and AEL suffer severe undercoverage or become ineffective.

Real Data Applications

Median Income (Small Area Estimation): Applied to estimate 1989 median income for US states. RETEL provided more accurate estimates (lower average absolute deviation) and more reliable credible intervals compared to EL and ETEL in a hierarchical Bayesian model.
Roman-Era Egyptian Lifespan: Demonstrated an extension of RETEL using a non-normal auxiliary distribution (shifted Bernoulli) and hyperpriors on the regularization parameter $\tau$ . This allowed for a fully Bayesian treatment where the strength of regularization is inferred from the data, showing how the method adapts to specific data supports.

5. Significance

This paper provides a rigorous solution to a long-standing barrier in Bayesian nonparametric inference using empirical likelihood. By replacing the ad-hoc addition of pseudo-data with a principled regularization via KL divergence, the authors:

Restore the logical consistency of Bayesian updating (no arbitrary zeroing of parameter space).
Enable the use of standard MCMC sampling techniques without the need for complex, constrained sampling schemes.
Bridge the gap between frequentist empirical likelihood methods and Bayesian inference, offering a robust pseudo-likelihood that retains desirable asymptotic properties while significantly improving finite-sample behavior.

The proposed RETEL framework is presented as a superior alternative for Bayesian inference in semiparametric models, particularly in small-sample or high-dimensional settings where traditional EL/ETEL methods fail.