A simple tool for weighted averaging of inconsistent data sets

This paper proposes and validates a robust, general-purpose Bayesian method for averaging inconsistent data sets by treating input uncertainties as lower bounds to better handle outliers, demonstrating its effectiveness across various scientific domains and providing a freely available Python implementation.

The Gist

Imagine you are trying to guess the exact weight of a mysterious object. You ask ten different friends to weigh it.

• Friend A uses a high-tech digital scale and says: "It's 10.0 kg, give or take 0.1 kg."
• Friend B uses a bathroom scale and says: "It's 10.2 kg, give or take 0.5 kg."
• Friend C uses a very precise scale but accidentally bumped it, so they say: "It's 15.0 kg, give or take 0.1 kg."

The Problem:
If you just take the "standard" math approach (the one taught in most basic science classes), you would trust the precise scales (A and C) the most because they claim to be very sure. You would average their numbers. But because Friend C's scale was bumped (an "outlier"), your final answer would be pulled way off course, even though you thought you were being precise.

The standard math assumes that if a friend says "I'm sure," they are telling the truth. But in the real world, sometimes people (or machines) are overconfident, or there are hidden errors they didn't account for.

The Solution in This Paper:
The authors, M. Trassinelli and M. Maxton, propose a new, smarter way to average these numbers. They call it the "Conservative" or "Jeffreys' Weighted Average."

Here is how it works, using simple analogies:

1. The "Pessimist's Safety Net"

The standard method assumes the uncertainty a friend gives (e.g., "±0.1 kg") is the exact truth.
The new method says: "Let's assume that number is just a minimum safety net."

It treats the reported uncertainty as a lower bound. It assumes the real uncertainty could be bigger. It's like saying, "You claim your scale is accurate to 0.1 kg, but I'm going to bet that if you bumped it or the floor was uneven, the real error might be 0.5 kg or even 1.0 kg."

2. The "Fat Tails" (The Umbrella Analogy)

In standard math, the "bell curve" (the shape of probability) is very skinny. If a data point is far away from the average, the math screams, "That's impossible! Throw it out!"

The new method uses a distribution with "fat tails."

• Imagine a standard bell curve is a tight umbrella. If a raindrop (a data point) falls outside the edge, it's not covered.
• The new method is a giant, floppy beach umbrella. It covers a much wider area. If a data point is far away (an outlier), the umbrella still covers it, but it doesn't panic. It says, "Okay, that's a weird drop, but it's possible. I won't let it drag my whole average to the side."

This makes the final average robust. It ignores the "weird" numbers just enough so they don't ruin the result, but it doesn't ignore them completely either.

3. The "Group of Experts" Test

The authors tested this new method on three real-world scenarios:

• The Simulation: They created fake data with a "bumped scale" (an outlier). The old method got confused and gave a wrong answer. The new method saw the weird number, shrugged, and gave the correct answer.
• Gravity (The Newton Constant): Scientists have been trying to measure the force of gravity for decades, but different labs get different results. The "official" average often has to be manually adjusted by experts to make sense. The new method automatically handled the messy data and arrived at a result that matched the most trusted modern values, without needing a human to manually "fix" the math.
• Particle Physics (The Proton Radius): This is a famous mystery in physics. Some experiments say the proton is small; others say it's big. The data is split into two distinct groups (bimodal).
  • The old method tries to force these two groups into one single average number, which is misleading.
  • The new method is honest. It shows you a graph with two humps. It says, "We can't just give you one number. The data is split. Here are the two possibilities." This is a much more honest way to present the truth.

4. The Tool

The authors didn't just write a theory; they built a free Python tool (a computer program) that anyone can use. It does the hard math (which is too complex to do by hand) automatically.

The Bottom Line

The standard way of averaging data is like driving a car with a very sensitive steering wheel: a tiny bump in the road sends you off the lane.

The method in this paper is like driving a tank. It's heavier and a bit slower to calculate, but it can drive over rocks (outliers) and uneven ground (inconsistent data) without losing its way. It admits, "I don't know everything, so I'll be a little more cautious," and in doing so, it often gives a more reliable answer than the "perfect" math we usually use.

In short: When data is messy and scientists disagree, don't just average the numbers. Use this "pessimistic" approach that assumes everyone might be a little more wrong than they think, and you'll get a result that is much harder to fool.

Technical Summary

1. Problem Statement

The standard method for combining independent measurements of the same quantity is the inverse-variance weighted average (standard weighted average). While analytically simple and universally accepted, this method fails when data sets are inconsistent (i.e., the scatter of data points exceeds their reported uncertainties).

• Limitations of Standard Methods: The standard approach assumes reported uncertainties ($\sigma_i$) are exact. If data is inconsistent due to uncontrolled systematic errors or biases, the standard method yields a final uncertainty that depends only on the input $\sigma_i$, ignoring the actual data spread. This leads to misleadingly precise results.
• Limitations of Existing Alternatives: Common fixes include the Birge ratio, which scales all uncertainties by a common factor to force the reduced $\chi^2$ to unity. However, this assumes a common scaling factor for all measurements, which is inappropriate for inter-laboratory comparisons where systematic errors differ between groups. Other methods involving random biases or complex Bayesian models often introduce too many assumptions or parameters, reducing their generality and ease of use.

2. Methodology

The authors propose a robust alternative based on Bayesian statistics, specifically the method originally formulated by Sivia (1996) and Sivia & Skilling (2006).

Core Assumptions

• Pessimistic Framework: The reported uncertainty $\sigma_i$ for each datum is treated not as the true uncertainty, but as a lower bound of the true, unknown uncertainty $\sigma'_i$.
• Marginalization: The true uncertainty $\sigma'_i$ is unknown. The probability distribution for each data point is obtained by marginalizing over $\sigma'_i$ using a prior probability distribution $p(\sigma'_i)$.
• No Common Scaling: Unlike the Birge ratio, this method does not assume a common scaling factor or random bias for the entire dataset. Each datum is treated independently regarding its potential underestimation of uncertainty.

Two Specific Approaches

The paper derives and compares two specific implementations of this framework:

1. The Conservative Approach:

   • Uses a prior $p(\sigma'_i) \propto 1/(\sigma'_i)^2$.
   • Resulting Likelihood: The likelihood function for a single datum becomes a Gaussian with "heavy tails" proportional to $1/(x_i - \mu)^2$.
   • Effect: This distribution is more tolerant of outliers than a Gaussian but less extreme than the Jeffreys approach.

2. The Jeffreys' Prior Approach (Proposed as the primary solution):

   • Uses the non-informative Jeffreys' prior $p(\sigma'_i) \propto 1/\sigma'_i$.
   • To avoid divergence, the authors consider the limit where the upper bound of the uncertainty ($\sigma_{max}$) approaches infinity.
   • Resulting Likelihood: The likelihood function has tails that decrease as $1/(x_i - \mu)$, which is even slower than the conservative approach.
   • Advantage: This provides maximum robustness against outliers and inconsistencies without requiring arbitrary parameters like $\sigma_{max}$.

Implementation

• Unlike the standard weighted average, these methods do not yield a closed-form analytical solution for the mean ($\hat{\mu}$) or its uncertainty ($\hat{\sigma}_{\mu}$).
• The authors provide a Python library (bayesian_average) that uses numerical optimization to find the maximum of the likelihood function and calculate the uncertainty.
• The method naturally handles multimodality and asymmetry in the final probability distribution, which standard methods cannot do.

3. Key Contributions

1. Robust Averaging Tool: A practical, freely available Python implementation of the Sivia/Skilling Bayesian averaging method, making it accessible to non-experts in metrology.
2. Theoretical Justification: A clear derivation showing that treating reported uncertainties as lower bounds leads to likelihood functions with heavy tails, naturally down-weighting outliers without arbitrary rejection criteria.
3. Comparative Analysis: A comprehensive benchmarking of the proposed method against standard weighted averages, Birge ratio corrections, and other Bayesian models across synthetic and real-world data.

4. Results and Applications

The authors tested the method on three distinct categories of data:

A. Synthetic Data

• Consistent Data: The Jeffreys' method produced results consistent with the standard average but with slightly larger (more realistic) uncertainties.
• Inconsistent Data (Random Bias): The standard method underestimated the uncertainty significantly. The Jeffreys' method correctly expanded the uncertainty to reflect the data spread (approx. 6x larger than the standard method in the test case).
• Outliers: When a $5\sigma$ outlier was introduced, the standard average shifted drastically toward the outlier. The Jeffreys' method remained stable, with the outlier only causing a slight asymmetry in the likelihood tails rather than shifting the mean.

B. Newtonian Gravitational Constant ($G$)

• Applied to CODATA compilations (1969–2022).
• 1998 Case: A specific dataset contained a highly precise measurement that was later found to have a large systematic error (an outlier).
  • The standard method (with Birge ratio) required an expansion factor of 37 to accommodate the outlier.
  • The Jeffreys' method naturally handled the outlier, recovering a mean value close to the modern CODATA recommendation with a plausible uncertainty, without needing ad-hoc scaling factors.

C. Particle Properties (Particle Data Group - PDG)

• Tested on various particle properties (e.g., neutron lifetime, kaon mass, proton charge radius).
• General Agreement: For most properties, the Jeffreys' average agreed well with PDG recommended values, though with slightly larger uncertainties (factor of ~2.2 max).
• Detection of Complex Distributions: The method successfully identified cases where the final probability distribution was multimodal or highly asymmetric (e.g., the proton charge radius).
  • In the case of the proton radius, the distribution showed two distinct peaks (bimodality). The authors argue that in such cases, a single "weighted average" is physically meaningless, and the full probability distribution should be reported. This aligns with PDG recommendations but is more easily diagnosed using this method's output.

5. Significance and Conclusion

• Generality: The method requires minimal assumptions (only that $\sigma_i$ is a lower bound) and avoids complex modeling of systematic errors.
• Robustness: It is highly resistant to outliers and inconsistent data, making it superior to the standard inverse-variance method for inter-laboratory comparisons.
• Transparency: By outputting the full likelihood distribution, the method reveals issues (like multimodality) that standard averages hide. This allows researchers to make informed decisions rather than relying on a single number that might be misleading.
• Practicality: The provided Python library bridges the gap between complex Bayesian theory and daily data analysis, offering a "drop-in" replacement for standard averaging in scenarios where data inconsistency is suspected.

Final Verdict: The paper advocates for the Jeffreys' weighted average as a robust, general-purpose tool for scientific data analysis, particularly when dealing with inconsistent datasets, outliers, or inter-laboratory comparisons where systematic errors are difficult to quantify. It emphasizes that while the method is powerful, the resulting probability distribution should be inspected for multimodality before reporting a single mean value.