📊 statistics

Integration of Individual Participant and Aggregate Data Under Dataset Shift: Summary Statistic Comparison and Scalable Computation

This paper demonstrates that incorporating outcome-stratified aggregate data significantly enhances estimation efficiency in integrated IPD-AD analyses under dataset shift, while also proposing a scalable, non-iterative constrained maximum likelihood framework to facilitate robust evidence synthesis.

Original authors: Ming-Yueh Huang, Jing Qin, Chiung-Yu Huang

Published 2026-03-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ming-Yueh Huang, Jing Qin, Chiung-Yu Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive jigsaw puzzle, but you don't have all the pieces. You have a box of Individual Participant Data (IPD)—these are the detailed, colorful pieces from one specific box that you own. They show exactly how the picture looks up close, but you only have a few hundred of them.

Then, you hear about a Library of Aggregate Data (AD). This library has thousands of other puzzle boxes, but they don't give you the pieces. Instead, they give you summary cards describing the boxes. Some cards say, "The average color of this box is blue." Others say, "In the top-left corner, 80% of the pieces are red."

The goal of this paper is to figure out the best way to combine your detailed pieces with these summary cards to solve the puzzle faster and more accurately.

Here is the breakdown of their findings, explained simply:

1. The Problem: Not All Summary Cards Are Created Equal

The researchers asked: Does it matter what kind of summary card the library gives us?

They tested three types of cards:

The "Average" Card: "The average age of people in this study is 40." (This is like looking at the whole box and guessing the average color).
The "Grouped by Age" Card: "For people under 30, the average income is X. For people over 30, it's Y." (This groups the data by who they are).
The "Grouped by Outcome" Card: "For people who earned a lot of money, the average age is X. For those who earned little, the average age is Y." (This groups the data by what happened to them).

The Big Discovery:
The paper found that the "Grouped by Outcome" cards are the superstars.

Analogy: Imagine you are trying to guess the weather.
- Grouped by Age: "People who are 20 say it's raining." (Not very helpful; age doesn't cause rain).
- Grouped by Outcome: "People who are holding umbrellas say it's raining." (This is a huge clue!).
Why it works: When you group people by the result (like income or disease status), you get a much clearer picture of the relationship between their characteristics and the outcome. The paper proves mathematically that using these "Outcome-Stratified" summaries makes your final answer much more precise (efficient) than using the other types.

The Catch: Most scientific reports only give you the "Grouped by Age" cards, especially when the result is a number (like income). They rarely give you the "Grouped by Outcome" cards. The authors are begging scientists to start sharing these specific cards because they are goldmines for accuracy.

2. The Complication: The "Different Worlds" Problem (Dataset Shift)

Sometimes, the library's data comes from a different time or place than your puzzle pieces.

Scenario: Your puzzle pieces are from New York in 2024. The library's summary cards are from New York in 1990.
The Shift: In 1990, there were fewer college graduates (Covariate Shift) or more people were unemployed (Prior Probability Shift). If you just mash the data together without adjusting, your puzzle will look wrong.

The Solution:
The authors built a "Universal Adapter." They created a mathematical framework that acts like a translator. It recognizes that the two datasets come from different "worlds" and adjusts the summary cards so they fit your specific puzzle pieces perfectly, even if the populations are different.

3. The Speed Bump: Too Many Cards = Too Slow

If you try to use every single summary card the library has (e.g., breaking income down into 100 tiny slices), the math gets incredibly heavy. It's like trying to solve the puzzle by looking at 10,000 tiny clues at once. The computer gets stuck, the numbers get messy, and the calculation takes forever.

The Fix:
The authors invented a Fast-Forward Button.

Old Way: A slow, iterative process where the computer guesses, checks, guesses again, and checks again until it's right.
New Way: A "One-Shot" algorithm. They figured out a mathematical shortcut that lets the computer calculate the perfect answer in a single step, without all the back-and-forth guessing. It's like having a GPS that calculates the route instantly instead of driving a mile, checking a map, and driving another mile.

4. Real-World Proof

They tested this on two real-life puzzles:

Income Data: They combined survey data from 1997 with summary stats from 1979. By using the "Outcome-Stratified" cards (grouping by income levels), they got a much clearer picture of how education and gender affect pay, with much less error than before.
House Prices in Japan: They combined individual house sales from 2019 with summary stats from 2018. Because house prices change over time (a "shift"), they used their "Universal Adapter." Again, the "Outcome-Stratified" cards gave them the most accurate estimates of how factors like distance to a train station or house age affect price.

The Bottom Line

This paper tells us three main things:

Stop ignoring the "Outcome" groups: If you are a researcher, don't just report averages. Report how your data looks when you split it by the result (e.g., "What do high earners look like?"). It makes everyone's future research better.
We can mix different eras: We can safely combine old data with new data, even if the populations have changed, using their new "Universal Adapter."
We can do it fast: We don't have to wait days for the computer to crunch the numbers; their new "One-Shot" method makes it instant and stable.

In short, they found a way to make our statistical "puzzles" bigger, clearer, and faster to solve by using the right kind of clues.

1. Problem Statement

The integration of Individual Participant Data (IPD) and Aggregate Data (AD) is a powerful strategy for synthesizing evidence across heterogeneous studies. However, two critical challenges limit the effectiveness of current methods:

Efficiency of AD Forms: Existing literature focuses heavily on integration algorithms but lacks a systematic evaluation of how different forms of AD (e.g., marginal means vs. stratified summaries) impact estimation efficiency. Specifically, it is unclear whether outcome-stratified summaries (common in case-control studies) offer superior efficiency for continuous outcomes compared to covariate-stratified summaries.
Dataset Shift and Scalability: IPD and AD often come from different populations, time periods, or study designs, leading to "dataset shift" (differences in joint distributions). Standard Constrained Maximum Likelihood Estimation (CMLE) frameworks struggle with this heterogeneity. Furthermore, incorporating rich AD (e.g., many stratified subgroups) creates high-dimensional constraint systems, leading to numerical instability and computational intractability due to the need for iterative Lagrange multiplier calculations.

2. Methodology

A. Unified Framework: Constrained Maximum Likelihood Estimation (CMLE)

The authors operate within a CMLE framework where AD is treated as estimates of population parameters satisfying unbiased estimating equations.

Model: Let $Y$ be the outcome and $X$ be covariates. The conditional density $f(y|x;\beta)$ is parametric, while the marginal distribution of $X$ is non-parametric (nuisance).
Constraints: AD provides constraints of the form $\sum p_i \Psi(X_i; \beta, \phi, \theta) = 0$ , where $\phi$ represents the AD summary statistics and $\theta$ represents nuisance parameters (e.g., shift parameters).
Handling Uncertainty: When AD sample sizes are finite, the uncertainty is modeled by augmenting the log-likelihood with a quadratic penalty term based on the normal approximation of the AD estimates.

B. Addressing Dataset Shift

The paper introduces a general biased-sampling formulation to handle two types of shift:

Covariate Shift: The marginal distribution of $X$ differs, but $f(y|x)$ remains constant.
Prior Probability Shift (Label Shift): The marginal distribution of $Y$ differs, but $f(x|y)$ remains constant.

Mechanism: They employ a density ratio model $w(x, y) = w_X(x)w_Y(y)$ to link the IPD and AD distributions.
Estimation: The shift functions ( $w_X, w_Y$ ) are modeled parametrically (e.g., exponential tilting) and estimated jointly with the main parameters $\beta$ via the constrained likelihood. This allows the derivation of unbiased estimating equations for AD under shift.

C. Fast Non-Iterative Algorithm

To overcome the computational burden of solving nested optimization problems (inner loop for Lagrange multipliers, outer loop for $\beta$ ), the authors propose a one-step, non-iterative estimator.

Derivation: Based on the asymptotic linear expansion of the CMLE and the unconstrained MLE.
Procedure:
1. Obtain initial estimators ( $\hat{\beta}_{MLE}$ , $\hat{\phi}$ , $\hat{\theta}$ ) using standard methods (e.g., GMM for nuisance parameters).
2. Compute the influence function and the Hessian/Jacobian matrices ( $J$ and $\Sigma$ ) at these initial values.
3. Update the estimator in a single step:
  $\hat{\beta}_{fast} = \hat{\beta}_{MLE} + \text{correction term involving } J^{-1} \text{ and the constraint residuals.}$
Advantage: This avoids Newton-Raphson iterations for Lagrange multipliers, significantly improving numerical stability and scalability for high-dimensional constraints.

D. Comparison of AD Types

The study compares three classes of AD summaries:

Marginal Means: Overall means of $Y$ or $X$ .
Covariate-Stratified: Means of $Y$ within subgroups defined by $X$ (e.g., $E[Y|X \in \Omega]$ ).
Outcome-Stratified: Means of $X$ within subgroups defined by $Y$ (e.g., $E[X|Y \in \text{interval}]$ ).

3. Key Contributions

Efficiency Hierarchy of AD: The paper establishes that outcome-stratified covariate summaries (e.g., mean covariates within outcome quartiles) provide substantially greater estimation efficiency than covariate-stratified outcome summaries or marginal means. This holds true even under dataset shift, particularly for continuous outcomes.
Extension to Dataset Shift: The authors successfully extend the CMLE framework to accommodate both covariate shift and prior probability shift simultaneously using a unified density ratio model, deriving valid estimating equations for AD under these conditions.
Scalable Computation: The development of the fast, non-iterative algorithm resolves the numerical instability associated with high-dimensional constraints, making the integration of rich AD feasible in practice.
Practical Recommendation: The findings strongly advocate for the routine reporting of outcome-stratified summaries (e.g., covariate means by outcome quartiles) in trial reports and public repositories, especially for continuous endpoints, to facilitate more efficient meta-analysis.

4. Results

Simulation Studies

No Shift: Outcome-stratified summaries (specifically using quartiles, $\hat{\phi}_{X|Y}^{quartile}$ ) consistently yielded the lowest standard errors and highest relative efficiency compared to marginal means and covariate-stratified summaries.
With Shift: Under both covariate and prior probability shift, the proposed CMLE with outcome-stratified summaries maintained high efficiency. Notably, marginal covariate means ( $\hat{\phi}_X$ ) provided little to no efficiency gain under prior probability shift due to the flatness of the moment conditions.
Algorithm Performance: The non-iterative estimator ( $\hat{\beta}_{fast}$ ) achieved asymptotic equivalence to the full iterative CMLE but with significantly reduced computational time and improved stability.

Empirical Applications

Income Data (NLSY): Analyzed the effect of demographics on income. Integrating outcome-stratified covariate means (quartiles) from an external cohort reduced standard errors by up to 61% (Relative Efficiency = 1.61) compared to the unconstrained MLE.
Housing Data (Japan): Analyzed real estate prices under prior probability shift (biased sampling of sold homes). Outcome-stratified summaries again provided the highest efficiency gains, reducing standard errors for key coefficients (e.g., house age, floor area) by 20–25% compared to standard MLE.

5. Significance

Methodological Advancement: This work bridges the gap between machine learning concepts (dataset shift) and biostatistical meta-analysis (IPD-AD integration), providing a rigorous theoretical framework for handling heterogeneity.
Practical Impact: By demonstrating that specific types of summary statistics (outcome-stratified) are far more valuable than others, the paper offers a concrete guideline for data curators and researchers. It argues that the current practice of omitting these summaries for continuous outcomes is a missed opportunity for evidence synthesis.
Computational Feasibility: The proposed algorithm makes it possible to integrate complex, high-dimensional aggregate data without the prohibitive computational costs previously associated with constrained likelihood methods.

In conclusion, the paper provides a robust, efficient, and scalable solution for integrating heterogeneous data sources, proving that the type of summary statistic reported is as critical as the integration algorithm itself.