📊 statistics

Jeffreys-Type Penalized GEE for Correlated Binary Data with an Odds-Ratio Parameterization

This paper proposes a Jeffreys-prior penalized Generalized Estimating Equations (GEE) framework using an odds-ratio parameterization to ensure finite estimates and reliable inference for correlated binary data under separation, offering computationally efficient variants that outperform ordinary GEE in sparse or rare-event settings while maintaining performance in regular scenarios.

Original authors: Anestis Touloumis

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anestis Touloumis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery using data from a group of friends who are all related to each other (like a family or a team). Because they are related, their answers aren't independent; if one person says "yes," their sibling is likely to say "yes" too. In statistics, this is called correlated data.

To solve the mystery, you usually use a standard tool called GEE (Generalized Estimating Equations). Think of GEE as a reliable GPS that helps you navigate through the data to find the truth.

The Problem: The "Perfect Storm" (Separation)

Sometimes, the data gets tricky. Imagine a situation where a specific clue (like a specific symptom) perfectly predicts the outcome. For example, every single person with a certain gene gets sick, and no one without it gets sick. In statistics, this is called separation.

When this happens, the standard GPS (ordinary GEE) breaks down. It gets confused, spins in circles, and can't find a solution. It might scream, "I can't calculate this!" or give you a number so huge it's basically infinity. This is a disaster for researchers, especially when they have small groups of people or rare events.

The Old Fixes: Flawed Maps

Scientists tried to fix this by adding a "penalty" to the GPS to stop it from going to infinity. However, the old methods had two big flaws:

The Map Shrank: They used a way of measuring relationships (correlation) that would collapse to zero when the data got extreme. It was like trying to use a map that shrinks to a single dot right when you need to navigate a storm.
The Wrong Compass: They only worked for one specific type of question (the "logit" link), leaving other types of questions (like "probit" or "cauchit") unsolved.

The New Solution: A Better GPS (PGEE)

The author, Anestis Touloumis, proposes a new, upgraded GPS called PGEE (Penalized GEE). Here is how it works, using simple analogies:

1. The "Jeffreys" Safety Net (The Penalty)
Imagine the GPS has a safety net called the Jeffreys prior. If the GPS starts to drift toward infinity (because of the "perfect prediction" problem), this safety net gently pulls it back to a reasonable, finite number. It's like having a bungee cord attached to the GPS so it can never fall off the edge of the world.

2. The "Odds Ratio" Compass (The New Parameterization)
Instead of using the old, fragile way of measuring relationships, this new GPS uses Odds Ratios.

The Old Way: Measuring correlation was like trying to measure the distance between two moving cars while they are driving at light speed. If they get too close to the edge, the measurement breaks.
The New Way: The author suggests pooling all the data into big, sturdy tables (like combining all the puzzle pieces into a single, solid block). This creates a measurement that stays stable and finite, even when the data is messy or extreme. It's like using a solid steel ruler instead of a rubber band.

3. The "One-Step" Shortcut
Calculating this new GPS can be slow. So, the author also offers two shortcuts:

OPGEE: A "one-step" version that takes a quick guess and stops. It's like taking a quick look at the map instead of driving the whole route.
HPGEE: A "hybrid" version that mixes the old and new methods.
These are useful if the full calculation takes too long or gets stuck.

What Happened When They Tested It?

The author ran thousands of simulations (like test drives in a video game):

In the "Storm" (Separation): The old GPS crashed or gave nonsense. The new PGEE GPS kept driving, found the right answer, and didn't break. It correctly identified the important clues and ignored the irrelevant ones.
In "Calm Weather" (Regular Data): When there was no separation, the new GPS performed just as well as the old one. It didn't slow things down or make things worse.

Real-World Test: The Respiratory Illness Trial

The author tested this on real data from a respiratory illness study.

The Issue: The standard method failed completely. It couldn't converge because the data had "separation" (a specific group of patients all had the same outcome).
The Result: The new PGEE method worked perfectly. It found the answers, calculated the risks, and showed that the treatment worked, even though the old method gave up.

The Bottom Line

This paper introduces a new statistical tool that fixes a common crash in data analysis. It combines a safety net (to stop numbers from going to infinity) with a sturdy compass (to measure relationships without breaking). It works when the data is messy, rare, or extreme, but it's just as good as the old tools when the data is normal.

The author has also built this into a free software package (called geer in R) so other researchers can use it immediately.

Technical Summary: Jeffreys-Type Penalized GEE for Correlated Binary Data

Problem Statement
Generalized Estimating Equations (GEE) are the standard tool for population-averaged inference on correlated binary responses. However, ordinary GEE is numerically unstable in sparse settings, small samples, or when "separation" occurs (where a covariate or linear combination perfectly predicts the response). Under separation, ordinary GEE often fails to converge, yields infinite or extreme estimates, and produces unreliable Wald-type inference.

Existing Penalized GEE (PGEE) approaches attempt to mitigate these issues but suffer from two primary limitations:

Parameterization Collapse: Many methods rely on correlation-coefficient parameterizations for the working association. Under separation, fitted probabilities approach 0 or 1, causing the feasible range of the correlation coefficient to collapse toward zero (due to Fréchet bounds). This forces the working structure toward independence precisely when within-subject dependence is strongest, undermining efficiency.
Link Function Restrictions: Current penalized methods are largely restricted to the logit link and do not generally guarantee finite estimates under non-independence working structures.

Methodology
The paper proposes a Jeffreys-type Penalized GEE (PGEE) framework designed to ensure finite estimates and numerical stability under separation. The methodology integrates two core components:

Jeffreys-Type Penalty:
The authors adapt the Jeffreys-prior penalty (analogous to Firth's correction in GLMs) to the GEE framework. The penalty term is defined as $\Lambda_{PG} = \delta \log[\det(\Sigma_0)]$ , where $\Sigma_0$ is the GEE analog of the Fisher information matrix and $\delta \in (0, 1]$ controls the penalty strength. The default is $\delta = 1/2$ . This penalty stabilizes estimation by pushing fitted probabilities away from the boundaries (0 and 1).
- Theoretical Guarantee: Under a working independence structure, the PGEE equations reduce to penalized maximum likelihood score equations. The authors prove that for common link functions (logit, probit, complementary log-log, cauchit), the estimator is guaranteed to be finite provided the design matrix has full column rank and the link function satisfies a specific derivative condition.
- Non-Independence: While a formal finiteness guarantee is not established for non-independence structures, the penalty is shown to be $O_p(1)$ , ensuring asymptotic equivalence to ordinary GEE under standard regularity conditions.
Marginalized Odds-Ratio Parameterization:
To avoid the collapse of correlation coefficients, the working association is parameterized using marginalized odds ratios computed from pooled $2 \times 2$ contingency tables across subjects.
- Unlike iterative correlation estimators that depend on fitted marginal probabilities, these pooled estimates are computed directly from the data (with a Haldane continuity correction, $\zeta=0.5$ ).
- This approach ensures that association estimates remain finite and well-defined even under separation and sparsity, preventing the working structure from degenerating into independence.

Variants and Implementation
To reduce computational costs, the authors propose two approximations:

OPGEE (One-step PGEE): A single Fisher scoring iteration of the penalized equations starting from the penalized independence estimator.
HPGEE (Hybrid PGEE): A single Fisher scoring iteration of the ordinary GEE equations starting from the penalized independence estimator.

The methods are implemented in the R package geer via the function geewa_binary().

Key Contributions

Framework Development: A PGEE framework combining a Jeffreys-type penalty with marginalized odds-ratio parameterizations, specifically designed to handle separation in correlated binary data.
Finiteness Guarantees: Establishment of finite estimates under working independence for multiple link functions (logit, probit, cloglog, cauchit).
Robustness to Separation: Demonstration that the proposed method avoids the numerical failure modes (non-convergence, infinite estimates, correlation collapse) that plague ordinary GEE and existing penalized approaches under separation.
Practical Variants: Introduction of OPGEE and HPGEE to offer computationally efficient alternatives when full iteration is slow or fails.

Results

Simulation Studies:
- Separation Regimes: In scenarios of complete and quasi-complete separation, ordinary GEE and bias-reduced estimators (RBR) failed to converge in the majority of replications. In contrast, PGEE, OPGEE, and HPGEE achieved near-complete empirical convergence across all working structures (independence, exchangeable, unstructured). The methods correctly identified separating covariates with high power while maintaining nominal rejection rates for irrelevant covariates.
- Regular Regimes: In settings without separation, the penalized estimators performed comparably to ordinary GEE in terms of convergence, empirical coverage, and prediction error, while offering modest bias reduction in small samples.
Application (Respiratory Illness Trial):
- The method was applied to a trial (Clinic 2) where ordinary GEE failed to converge due to complete separation (all active-treatment females had good respiratory status).
- PGEE converged successfully, yielding finite estimates for a probit model. The results were stable across different working structures (independence, exchangeable, unstructured) and penalty strengths ( $\delta=0.1, 0.5$ ).
- Notably, implementations using correlation-coefficient parameterizations failed to produce admissible association estimates, whereas the odds-ratio parameterization remained stable.

Significance and Claims
The paper claims that the proposed PGEE framework provides a robust solution for correlated binary data analysis in challenging settings where ordinary GEE fails.

Numerical Stability: The combination of the Jeffreys penalty and odds-ratio parameterization prevents the "collapse" of the working association structure and ensures finite estimates under separation.
Flexibility: Unlike existing penalized GEE methods, this framework supports various link functions (not just logit) and non-independence working structures without sacrificing numerical stability.
Asymptotic Validity: The penalty is asymptotically negligible, meaning the proposed estimators share the same asymptotic distribution as ordinary GEE, allowing standard sandwich-based inference to be used without modification.
Practical Utility: The authors emphasize that the method is a tool for "existence and reliable estimation" in difficult regimes rather than a primary bias-reduction tool for regular settings, distinguishing it from other bias-corrected GEE approaches.

The paper concludes that while formal finiteness guarantees for non-independence structures remain an open theoretical problem, the empirical evidence suggests the method is highly effective in practice. Future work includes establishing formal finiteness conditions for non-independence structures, developing data-driven selection criteria for the penalty strength $\delta$ , and extending the framework to multinomial or ordinal outcomes.