📊 statistics

Semiparametric Efficient Fusion of Individual Data and Summary Statistics

This paper proposes a semiparametric framework and adaptive estimators to efficiently fuse individual internal data with external summary statistics for improved statistical inference, while providing robustness against bias when transportability assumptions fail.

Original authors: Wenjie Hu, Ruoyu Wang, Wei Li, Wang Miao

Published 2026-02-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenjie Hu, Ruoyu Wang, Wei Li, Wang Miao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about a specific group of people (let's call them the "Local Squad"). You have a detailed notebook of interviews with every single member of this squad. This is your Internal Data. It's gold, but maybe you don't have enough interviews to be 100% sure of the answer.

Now, imagine you hear rumors from a neighboring town (the "External Town"). You don't have access to their individual interview notebooks because of privacy rules, but you do have a few Summary Reports (like "The average age is 30" or "70% prefer coffee"). These are your External Summary Statistics.

The goal of this paper is to figure out how to combine your detailed Local Squad notebook with the External Town's summary reports to get the most accurate answer possible, without getting tricked.

Here is the breakdown of their solution, using simple analogies:

1. The Problem: The "Efficiency Paradox"

Usually, adding more information helps. But the authors found a weird trap. If you just blindly paste the External Town's numbers into your Local Squad's math, you might actually get a worse answer than if you ignored them entirely.

The Analogy: Imagine you are trying to guess the average height of your basketball team. You have measured every player perfectly. Then, a friend tells you, "I heard the average height of a random group of people in the next city is 6 feet."
- If you just average your team's height with that 6 feet, you might mess up your calculation if that friend's number is shaky or if the two cities are actually very different.
- The paper calls this the "Efficiency Paradox": Adding outside info can sometimes make your result less precise if you don't handle the "uncertainty" of that outside info correctly.

2. The Solution: The "Smart Fusion" (Efficient Estimator)

The authors created a new mathematical recipe (an estimator) that knows how to mix these two sources perfectly.

How it works: Instead of just averaging the numbers, this recipe weighs the External Summary Reports based on how "shaky" or uncertain they are.
- If the External report is very precise (like a high-quality survey), the recipe gives it a lot of weight.
- If the External report is shaky, the recipe gives it very little weight.
The Result: This method guarantees that you will never do worse than using just your Local Squad data. In fact, it usually gives you a much sharper, more precise answer. It's like having a super-powered magnifying glass that uses the outside rumors to sharpen the focus on your local evidence.

3. The Safety Net: The "Adaptive Fusion"

There is a big risk: What if the External Town is actually very different from your Local Squad? Maybe they eat different food or have different genetics. If you assume they are the same when they aren't, your combined answer will be biased (wrong).

The Analogy: Imagine the External Town is actually a group of professional basketball players, while your Local Squad is a group of jockeys. If you mix their height data without checking, your average will be nonsense.
The Fix: The authors built an "Adaptive" version of their recipe. This version acts like a smart filter.
- It looks at the data and asks: "Does this external number look like it belongs to my group?"
- If the answer is Yes, it uses the data to improve the result.
- If the answer is No (the groups are too different), it automatically ignores that specific piece of outside info to avoid bias.
- Crucially, this filter is smooth and continuous, meaning it doesn't suddenly snap from "use it" to "ignore it," which keeps the math stable.

4. The "Re-Bootstrap" Trick: Fixing the Confidence Intervals

Even with the smart filter, there's a tricky situation called "Moderate Heterogeneity." This is when the two groups are almost the same, but not quite. The math says the groups are different, but the difference is so small it's hard to tell with a small sample size.

The Problem: In these "gray area" cases, the standard way of calculating a "Confidence Interval" (a range where the true answer likely sits) can be too optimistic. It might say, "We are 95% sure the answer is between 5 and 10," when actually, the true answer is outside that range.
The Fix: The authors propose a "Re-Bootstrap" procedure.
- The Analogy: Imagine you are trying to guess the weight of a mystery box. You have a scale, but you aren't sure if the scale is slightly off. Instead of just trusting the scale once, you simulate thousands of different scenarios where the scale might be slightly off in different ways. You then take the "worst-case scenario" from all those simulations to draw your final line.
- This ensures that even if the two groups are slightly different, your final confidence interval remains honest and reliable, preventing you from making false claims.

5. Real-World Test: The Stomach Bug Study

The authors tested their methods on a real dataset about Helicobacter pylori (a stomach infection).

The Setup: They had a small study of patients taking a standard treatment plus Traditional Chinese Medicine (Internal Data) and a larger study of patients taking only the standard treatment (External Data).
The Goal: Did adding the Chinese medicine help?
The Outcome:
- Using only the internal data, the result was "maybe" (not statistically significant).
- Using the "Smart Fusion" to combine the data, the result became clear: Yes, the combination treatment works better.
- The "Adaptive" and "Re-Bootstrap" methods confirmed this result was robust, even if there were slight differences between the two hospital populations.

Summary

This paper provides a toolkit for statisticians to safely combine their own detailed data with outside summary reports.

Don't just mix them blindly (you might get worse results).
Use the "Smart Fusion" to weigh outside info correctly.
Use the "Adaptive Filter" to ignore outside info if the groups are too different.
Use the "Re-Bootstrap" to make sure your final confidence ranges are honest, even when the groups are slightly different.

The result is a way to get more accurate answers from less data, without falling into traps caused by bad assumptions.

Technical Summary: Semiparametric Efficient Fusion of Individual Data and Summary Statistics

Problem Statement
The paper addresses the challenge of improving statistical inference for a target population (internal study) by integrating individual-level data from that study with summary statistics from external studies. While external summary statistics offer potential efficiency gains, their improper use can lead to the "efficiency paradox," where the fused estimator is less efficient than the estimator based solely on internal data. Furthermore, if the external summary statistics are not transportable (i.e., the underlying parameters differ between populations due to heterogeneity or bias), naive integration introduces estimation bias. Existing methods often rely on parametric models or fail to account for the uncertainty of external summary statistics, limiting their applicability to general semiparametric or nonparametric inference problems.

Methodology
The authors develop a semiparametric framework to fuse internal individual data and external summary statistics under a weak transportability assumption.

Semiparametric Efficiency Bound: The paper extends classic semiparametric theory (typically designed for i.i.d. data) to a setting involving both individual data and summary statistics. Under the assumption that a functional $\beta$ is transportable (i.e., $\beta(P_0) = \beta(P_1)$ ), the authors derive the semiparametric efficiency bound for estimating a general functional $\tau$ of the internal data distribution. This bound is shown to be no larger than the bound using only internal data, theoretically guaranteeing potential efficiency gains.
Data-Fused Efficient Estimator ( $\hat{\tau}_e$ ): A closed-form estimator is proposed that achieves the derived efficiency bound. This estimator utilizes the efficient influence functions of the internal data for both the target parameter $\tau$ and the transportable parameter $\beta$ . It effectively corrects the internal estimator by incorporating the external summary statistics in an optimal way, accounting for the uncertainty of the external estimates. The estimator inherits the Neyman orthogonality property, allowing for the use of flexible machine learning methods (e.g., neural networks) to estimate nuisance parameters without compromising inference validity.
Adaptive Fusion Estimator ( $\hat{\tau}_{adf}$ ): To address scenarios where the transportability assumption fails for some components of the summary statistics, an adaptive estimator is proposed. This estimator uses a continuous weighting mechanism to down-weight or exclude untransportable components based on the observed discrepancy between internal and external estimates. It possesses an "oracle" property, meaning it is asymptotically equivalent to an estimator that knows exactly which components are transportable, while maintaining continuity with respect to the data to ensure numerical stability.
Re-bootstrap Procedure: Recognizing that finite-sample performance may suffer from undercoverage when heterogeneity is moderate (i.e., the bias is of the same order as the estimation error), the authors propose a re-bootstrap procedure. This method constructs confidence intervals by simulating bootstrap quantiles under multiple candidate values for the heterogeneity parameter, selecting the most conservative quantile to ensure robust coverage rates.

Key Contributions

Theoretical Extension: The paper establishes the first semiparametric efficiency bound for fusing individual data and summary statistics, resolving the efficiency paradox by optimizing the incorporation of external information.
General Applicability: Unlike previous works restricted to parametric models, the proposed framework applies to general semiparametric and nonparametric models, covering a wide range of inference problems such as average treatment effects and missing data analysis.
Robustness to Heterogeneity: The adaptive fusion estimator provides a solution for integrating summary statistics when transportability is uncertain, achieving oracle efficiency without requiring prior knowledge of which statistics are valid.
Computational Efficiency: The proposed estimators have closed-form expressions, making them computationally efficient provided the efficient influence functions are available.

Results

Simulations: Extensive simulations demonstrate that the data-fused efficient estimator ( $\hat{\tau}_e$ ) consistently outperforms estimators based solely on internal data and primitive fusion methods (which ignore external uncertainty), achieving lower Root Mean Squared Error (RMSE) and appropriate coverage probabilities.
Robustness: In scenarios with partially untransportable statistics, the adaptive fusion estimator ( $\hat{\tau}_{adf}$ ) performs comparably to the oracle estimator (which knows the transportable set) and significantly outperforms non-adaptive methods (like the Generalized Data Integration Method or naive fusion), which suffer from large bias and coverage failure.
Finite Sample Performance: The re-bootstrap procedure successfully mitigates undercoverage issues in finite samples under moderate heterogeneity, maintaining coverage rates close to the nominal level while keeping confidence interval widths reasonable.
Real Data Application: The methods were applied to a Helicobacter pylori infection dataset combining a randomized trial (internal) and a single-arm study (external). The fused estimators ( $\hat{\tau}_e$ and $\hat{\tau}_{adf}$ ) achieved smaller standard errors than the internal-only estimator, allowing the rejection of the null hypothesis regarding the efficacy of an additional traditional Chinese medicine treatment at the 0.1 significance level, whereas the internal-only analysis failed to do so.

Significance
The paper claims that its primary significance lies in providing a rigorous semiparametric framework that resolves the efficiency paradox in data fusion. By deriving the efficiency bound and constructing an estimator that attains it, the authors demonstrate that external summary statistics can be used to improve inference without sacrificing efficiency, provided they are integrated optimally. Furthermore, the adaptive approach offers a robust solution for real-world scenarios where population heterogeneity may invalidate the transportability of some summary statistics, ensuring that efficiency gains are not achieved at the cost of bias. The work bridges the gap between meta-analysis and individual-level data integration, offering a practical tool for integrative data analysis in the presence of privacy constraints or data access limitations.