📊 statistics

Bandwidth-free nonparametric density estimation for grouped data

This paper introduces a bandwidth-free, mean-adjusted log-concave (MALC) nonparametric method for estimating the density of univariate grouped data without relying on specific distributional assumptions, demonstrating its robustness and effectiveness through simulations across various scenarios.

Original authors: Furkan Danisman, Hanna Jankowski, Camila P. E. de Souza

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Furkan Danisman, Hanna Jankowski, Camila P. E. de Souza

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the police report has been shredded. You don't have a list of individual suspects' names or ages; instead, you only have a few scraps of paper that say, "Ten people were found in the 20-to-30 age range," and "Five people were in the 30-to-40 range." This is the world of grouped data. In science, medicine, and sociology, we often can't see the exact details of every single person or event because of privacy rules, missing records, or technical limits. We only see the "bins" or "buckets" where things landed.

To make sense of these buckets, statisticians usually try to guess the shape of the hidden crowd. A common tool for this is like a camera lens called a kernel estimator. But here's the catch: this lens needs a "focus knob" called a bandwidth. If you turn the knob too much, the picture gets blurry; too little, and it gets jagged and noisy. Finding the perfect setting is a headache that requires a lot of trial and error. This paper tackles a big question: Can we figure out the true shape of the hidden crowd from these buckets without needing to fiddle with that annoying focus knob?

The authors of this paper, Furkan Danisman, Hanna Jankowski, and Camila P. E. de Souza, say yes. They introduce a new method called MALC (Mean-Adjusted Log-Concave). Think of "log-concave" as a rule that says the crowd's shape must be a nice, smooth hill—like a bell curve or a gentle slide—rather than a jagged mountain range with many peaks. This rule covers most of the shapes we see in nature, from human heights to the time it takes for a lightbulb to burn out.

The clever trick in their method is a two-step dance. First, they play a game of "best guess" using a standard bell curve to figure out where the center (the mean) of the hidden crowd likely is, even though they only have the buckets. They call this "mean recovery." Once they have a solid guess for the center, they use a special mathematical tool to build a smooth, hill-shaped density that fits the bucket counts perfectly. The best part? This tool is bandwidth-free. It automatically adjusts itself, so you don't need to be a math wizard to tune it.

When the researchers tested this idea, they didn't just look at one type of data; they threw everything at it. They simulated millions of scenarios with different sample sizes (from small groups of 100 to massive crowds of 1,000,000) and different bucket widths. They compared their new MALC method against three other popular techniques that do require that tricky focus knob. The results were promising: in these simulations, MALC consistently produced the most accurate pictures of the hidden data, especially when the sample sizes were small or medium. It was less sensitive to how wide or narrow the buckets were, making it a very sturdy tool.

They even took their method for a real-world test using human mortality data from six countries, including Canada, the US, and Japan. The data was grouped by age (like "deaths between age 0 and 1," "1 and 2," etc.). The MALC method successfully reconstructed the smooth curves of death rates, showing clear differences between countries and between men and women. For instance, it confirmed that people in Japan and Norway tend to live longer than those in the US, all while producing smooth, easy-to-read graphs without any manual tuning.

The paper suggests that this approach is a robust way to handle messy, grouped data in fields like demography, survival analysis, and medical studies. While the method assumes the underlying shape is a single smooth hill (which might not work for data with multiple distinct peaks), it offers a powerful, automatic alternative to the old, finicky methods. It's like giving detectives a camera that focuses itself, letting them solve the mystery of the hidden crowd with just a few scraps of paper.

Technical Summary: Bandwidth-free Nonparametric Density Estimation for Grouped Data

Problem Statement
In many scientific and practical contexts—ranging from experimental reports with uncertainty to confidentiality constraints and non-detects—data is not available in a conventional individual-observation format. Instead, data is collected in a "grouped" form, where only the frequency of observations falling within specific intervals (bins) is known. The challenge lies in estimating the underlying ungrouped probability density function (PDF) from these grouped counts without prior knowledge of the distribution's parametric form.

Existing nonparametric approaches, such as kernel density estimators adapted for grouped data, suffer from a critical dependency on bandwidth selection, which significantly impacts the smoothness and accuracy of the estimate. Furthermore, standard nonparametric methods often lack identifiability when applied to grouped data; that is, multiple distinct underlying densities can produce the same grouped frequency distribution.

Methodology: Mean-Adjusted Log-Concave (MALC) Estimation
The authors propose a fully automatic, bandwidth-free nonparametric method called Mean-Adjusted Log-Concave (MALC) density estimation for univariate continuous grouped data. The method relies on the assumption that the underlying density is log-concave (i.e., $f(x) = \exp(\phi(x))$ where $\phi$ is concave), a class that encompasses many popular distributions (e.g., Gaussian, Gamma, Weibull) while remaining sufficiently broad to avoid restrictive parametric assumptions.

The MALC algorithm proceeds in three main stages:

Mean Recovery via EM: Since the true mean of the unobserved data is unknown, the authors first estimate it. They assume a Gaussian distribution for the purpose of estimation and apply the Expectation-Maximization (EM) algorithm to the grouped data to find the Maximum Likelihood Estimator (MLE) of the mean ( $\hat{\mu}_n$ ). Theoretical results demonstrate that under the Kullback-Leibler (KL) divergence, the Gaussian density closest to a true log-concave density $f_0$ shares the same mean. Consequently, even if the true distribution is not Gaussian, this step provides a highly accurate recovery of the population mean, with the error bounded by the grid width ( $\delta$ ).
Discrete Log-Concave Estimation: The observed empirical probabilities of the bins are smoothed to obtain a discrete log-concave probability mass function (PMF). This is achieved by maximizing a penalized log-likelihood over discretely concave functions, utilizing the logConDiscr R package. This step yields a unique discrete PMF ( $\hat{p}_n$ ) that is the closest log-concave approximation to the observed grouped frequencies in the KL sense.
Convolution and Ungrouping: To recover a continuous density, the method constructs a random variable $X_{n,\delta}$ defined as the convolution of the discrete log-concave variable (based on $\hat{p}_n$ ) and a continuous perturbation variable $Z_{n,\delta}$ supported on $[0, \delta)$ . The perturbation is parameterized (using a Beta distribution) such that the resulting convolution has a mean equal to the recovered mean $\hat{\mu}_n$ . Finally, the MLE of a continuous log-concave density is computed from samples of this convolution using the logcondens package. This step effectively "ungroups" the data by finding the continuous log-concave density closest to the convolved distribution.

Key Theoretical Contributions

Identifiability and Consistency: The paper establishes that while the problem of recovering a log-concave density from grouped data is generally non-identifiable (multiple densities can yield the same grouped probabilities), the proposed MALC estimator converges to a unique limit. As the sample size $n \to \infty$ and the grid width $\delta \to 0$ , the estimator is shown to be consistent with the true underlying density, provided the true density is log-concave.
Bandwidth-Free Nature: Unlike kernel-based methods, the MALC approach does not require the user to select a smoothing parameter or bandwidth, making it fully automatic.
Robustness: Theoretical and simulation results indicate that the log-concave assumption is robust; the method performs well even when the true underlying distribution is not strictly log-concave, particularly for small to medium sample sizes.

Results and Evaluation
The performance of MALC was evaluated through extensive simulations against three kernel-based competitors: Blower and Kelsall (2002), BinnedNP (Barreiro Ures et al., 2019a), and KernSmooth (Wand et al., 2025).

Simulation Study: Across various distributions (both log-concave like Gamma and Chi-square, and non-log-concave like Log-normal and Student's $t$ ) and varying sample sizes ( $10^2$ to $10^6$ ) and grid widths, MALC consistently achieved the lowest $L_2$ error. It demonstrated superior performance in small-sample regimes and exhibited greater stability (less sensitivity) to changes in grid width compared to kernel-based methods.
Real-World Application: The method was applied to grouped mortality data from six countries (Canada, USA, Japan, South Korea, Norway, Australia) spanning ages 0–110. The results successfully estimated age-specific mortality densities and hazard functions, revealing expected demographic patterns, such as higher life expectancy in Japan and Norway compared to the USA. The method also provided smooth, non-decreasing hazard function estimates, consistent with the properties of log-concave densities.

Significance and Claims
The authors claim that MALC offers a robust, automatic alternative to traditional grouped data analysis. Its primary significance lies in eliminating the need for bandwidth selection, a common source of instability and user-dependency in nonparametric estimation. The method is particularly valuable in scenarios where data is inherently grouped due to technical or confidentiality constraints, and where the underlying distribution is unknown or complex.

The paper modestly notes limitations: the current implementation is restricted to univariate data with uniform grid widths. Additionally, while the method assumes log-concavity, it acknowledges that the resulting estimator may not perfectly preserve the exact bin probabilities of the original grouped data (compatibility), though simulations suggest this trade-off yields favorable overall density estimates. Future work is identified as extending the method to non-uniform grids and multivariate settings.

Technical Summary: Bandwidth-free Nonparametric Density Estimation for Grouped Data

More like this