🤖 machine learning

The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World

This paper argues that the assumption of true data-generating probability distributions in the social world is a harmful fiction that obscures critical choices and goals in machine learning, proposing instead a framework focused directly on relevant populations that preserves classical learning theory while avoiding these misleading abstractions.

Original authors: Benedikt Höltgen, Robert C. Williamson

Published 2026-04-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Benedikt Höltgen, Robert C. Williamson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Idea: The "Magic Dice" Myth

Imagine you are playing a game with a die. If you roll it a thousand times, you can predict with high confidence that a "6" will come up about 16% of the time. In the world of gambling or physics, we assume there is a true, hidden rule (a probability distribution) that governs the die. The die doesn't care about you; it just follows the laws of physics.

Machine Learning (ML) researchers often treat people like that die.

They assume that if they collect enough data about people (their income, age, location), there is a hidden, perfect "rulebook" or "map" that dictates exactly how those people will behave. They believe their job is to find this map, which they call a Data-Generating Distribution. They think, "If we just get enough data, we will discover the true probability that a specific person will commit a crime or default on a loan."

The authors of this paper say: "Stop pretending."

They argue that in the social world (dealing with humans), there is no such thing as a hidden rulebook. Humans are not dice. We are complex, changing, and influenced by our environment. Assuming a perfect, pre-existing probability map exists is not just wrong; it's dangerous because it hides the fact that the people building these models are making huge, subjective choices.

Analogy 1: The "Recipe" vs. The "Ingredient List"

The Old Way (The Gen-D Framework):
Imagine you are trying to predict how a cake will taste. The old way of thinking says, "There is a perfect, magical recipe for 'Cake' that exists in the universe. Our job is to taste enough cakes to figure out that exact recipe. Once we find it, we can bake the perfect cake every time."

The New Way (The Authors' View):
The authors say, "There is no magical recipe. There is only the specific batch of ingredients you have right now and the specific oven you are using."

If you change the flour brand, the cake changes.
If you change the oven temperature, the cake changes.
If you change the baker, the cake changes.

In Machine Learning, the "ingredients" are the data we choose to collect. The "baker" is the person designing the model. The "recipe" isn't a hidden truth waiting to be found; it is constructed by the people making the model.

Analogy 2: The "Weather Forecast" vs. The "Human Mood"

The Weather (Where the old idea works):
If you want to know if it will rain tomorrow, you look at atmospheric pressure. The atmosphere doesn't care about your feelings. It follows physical laws. You can say, "There is a 70% chance of rain," and that number is based on a stable physical reality.

The Human Mood (Where the old idea fails):
Now, imagine trying to predict if a specific person will be happy tomorrow.

If you ask them today, they might say "Yes."
If you ask them after a bad day at work, they might say "No."
If you ask them after a promotion, they might say "Yes."

There is no single "True Probability of Happiness" for that person. The number changes based on the context, the time, and the questions you ask.

The paper argues that when we use AI to predict human behavior (like who will get a job or who might re-offend), we are trying to predict a mood, not the weather. But we are using math designed for weather. This leads us to believe our predictions are "objective facts" when they are actually just guesses based on a specific, arbitrary snapshot of data.

Why Does Pretending Matter? (The "Costs")

The authors say that pretending these "perfect maps" exist causes three big problems:

1. It Hides the Choices We Make

When we say, "The model found the true probability," it sounds like the computer did the work and we just watched.

Reality: The computer didn't find a truth. The engineers chose which data to feed it. They chose which features to look at (e.g., "Should we look at zip code?").
The Cost: By pretending the model is just "discovering" a truth, we stop asking, "Why did you choose these specific variables?" This lets bad or biased choices slide under the rug.

2. It Creates a False Sense of Objectivity

If you tell a judge, "The algorithm says this person has a 30% chance of re-offending," it sounds scientific and neutral.

Reality: That 30% isn't a law of nature. It's a number calculated based on a specific group of people from a specific year. If you changed the year or the group slightly, the number might jump to 60% or drop to 10%.
The Cost: People trust the number too much. They think the algorithm is "fair" because it's "math," but the math is built on shaky ground.

3. It Confuses "Generalizing" with "Guessing"

In school, we learn that if we see a pattern in a sample, we can apply it to the whole world.

The Problem: In the social world, the "whole world" is constantly changing. A model trained on data from 2017 might fail completely in 2024 because society changed, not because the model was "wrong" mathematically.
The Cost: We keep building models that work for yesterday's world but fail in today's world, blaming the "noise" in the data rather than realizing the "map" we are using is outdated.

The Solution: Stop Looking for the "Ghost"

The authors suggest we stop trying to find the "Ghost in the Machine" (the true distribution). Instead, we should:

Admit we are building, not discovering: Acknowledge that every model is a construction, like a house built with specific bricks.
Focus on the specific population: Instead of saying "This model works for everyone," say "This model works for this specific group of people in this specific context."
Be honest about choices: When we build these systems, we need to openly discuss: "We chose to look at these variables because..." rather than "The data spoke for itself."

The Bottom Line

The paper is a warning:
We are treating human beings like dice rolls. We think there is a hidden, perfect number that describes a person's future. But there isn't. There is only the story we tell about them based on the data we chose to collect.

If we keep pretending there is a "True Probability," we risk building systems that feel objective and fair but are actually just reflecting the biases and choices of the people who built them. We need to stop looking for the magic map and start taking responsibility for the territory we are drawing.

1. Problem Statement

The paper challenges the foundational assumption in Machine Learning (ML) and Statistics that observed data is sampled from a true, underlying data-generating probability distribution (Gen-D).

The Status Quo: Standard ML theory (including fairness research) relies on the premise that data points $\{(x_i, y_i)\}$ are independent and identically distributed (i.i.d.) samples from a fixed joint distribution $P(X, Y)$ . This framework underpins error guarantees, generalization bounds, and the concept of a "Bayes-optimal predictor."
The Core Issue: The authors argue that while this assumption is a useful idealization for stable physical systems (e.g., gambling, physics), it is a fiction in social settings involving human behavior.
Consequences: Pretending such distributions exist obscures the specific modeling choices made by researchers, creates a false sense of objectivity, and leads to misleading conclusions regarding fairness, generalization, and the nature of prediction.

2. Methodology and Theoretical Framework

The authors employ a mix of philosophical argumentation, historical analysis, and rigorous mathematical re-derivation of learning theory to deconstruct the Gen-D framework.

Philosophical Critique: They argue that probabilities in social science are constructed (based on choices of population, timeframes, and attributes) rather than discovered. They cite the "reference class problem," noting that the probability of an event depends entirely on how one defines similarity (the abstraction level), which is a modeling choice, not a physical constant.
Mathematical Alternative (Finite Populations): Instead of assuming an infinite, abstract distribution, the authors propose a framework based on finite populations.
- They adapt classical learning theory (specifically Vapnik's work) to bound generalization error without invoking a true distribution.
- Key Mathematical Tool: They utilize sampling without replacement (hypergeometric setting) rather than sampling with replacement (binomial setting).
- Lemma 1 & 2 (Symmetrisation): They prove modified versions of the Symmetrisation Lemma for finite urns. These lemmas bound the difference between the error on a training set and the error on the remaining population (or the whole population) using a counting measure ( $\mu$ ) rather than a probability measure.
- Theorem 3 (Generalization Bound): They derive a generalization bound for a hypothesis class $\Lambda$ with VC dimension $h$ on a finite set of $l+k$ points. The bound shows that for large sample sizes, the training error will be close to the error on the rest of the finite population for the vast majority of possible samples, without assuming a true underlying distribution.

3. Key Contributions

A. Refutation of True Distributions in Social Settings

The authors demonstrate that "true" conditional probabilities $P(Y|X)$ do not exist because:

Population Dependence: The probability depends heavily on the definition of the population (e.g., time frame, geographic scope). Changing the training population (e.g., 1 year vs. 4 years of data) significantly alters the "optimal" predictions, proving there is no single stable distribution.
Sparsity and Volatility: In high-dimensional spaces, specific attribute combinations may have very few data points. Calculating a "true probability" based on these few points (e.g., 2 defaults out of 3) is volatile and not generalizable.
Sampling Without Replacement: In social contexts, we do not sample people with replacement. Once a person is observed or acted upon, they are not the same "random variable" in the future. The Gen-D framework fails to account for this.

B. Re-framing Learning Theory

The paper proves that learning and generalization do not require a Gen-D assumption.

By treating the dataset as a finite population, one can derive rigorous bounds on how well a model trained on a subset will perform on the rest of that specific population.
This shifts the goal from "approximating a metaphysical truth" to "finding a decision rule that minimizes loss on a defined population."

C. Exposing the "Illusion of Objectivity"

The authors argue that the Gen-D framework creates a dangerous illusion:

Masking Choices: It suggests that models are merely "estimating" a pre-existing truth, thereby hiding the fact that the model is a construction based on specific choices of input space (abstraction), loss functions, and data collection.
Fairness vs. Accuracy: The framework implies that fairness constraints are deviations from the "optimal" (Bayes) solution. However, empirical evidence shows that less discriminative models often exist with equal accuracy. The Gen-D framework obscures this by assuming a single "correct" model exists to be approximated.
Model Multiplicity: The existence of many models with similar average performance but different individual predictions (the Rashomon effect) is treated as a nuisance in Gen-D. In the finite population view, it is a fundamental conceptual reality: there is no single "true" predictor.

D. Critique of Abstraction and Variable Construction

The paper highlights that the choice of the input space $\mathcal{X}$ (how to represent individuals) is a critical, often unexamined, modeling choice.

The Gen-D framework treats $\mathcal{X}$ as given.
In reality, the choice of attributes (e.g., including or excluding race/gender) fundamentally changes the "similarity" metric and the resulting predictions.
Pretending a distribution exists suggests that adding more data will eventually reveal the "true" probability, ignoring that the abstraction itself dictates the outcome.

4. Results and Evidence

Empirical Demonstration (ACS Income): The authors analyzed US Census data (ACS Income) across four large states.
- Result: Models trained on 1 year of data vs. 4 years of data produced significantly different predictions for the same individuals (approx. 10% of predictions changed by >10%).
- Implication: If a "true distribution" existed, the optimal predictor should be stable regardless of the training window size (assuming the distribution is stationary). The volatility proves the distribution is not a fixed entity but a construct of the data window.
Theoretical Bounds: The derived generalization bounds (Theorem 3) confirm that learning is possible and bounded in finite populations without invoking i.i.d. sampling from an infinite distribution.

5. Significance and Implications

For Fairness and Ethics: The paper argues that the Gen-D framework is harmful in social settings because it allows practitioners to claim their models are "objective" or "following real chances." By rejecting this, the authors argue for explicit accountability: every modeling choice (abstraction, loss function, population definition) must be justified, as there is no "true" distribution to fall back on.
For Machine Learning Theory: It calls for a shift in how learning is taught and theorized. While Gen-D is acceptable for stable physical systems, social ML should be framed around finite populations and decision-making rather than distribution estimation.
For Practice:
- Researchers should stop framing goals as "approximating the Bayes-optimal classifier."
- Evaluation should focus on performance on specific, defined populations and worst-case scenarios rather than average risk over an abstract distribution.
- The "march towards truth" (getting closer to the true distribution) is a fallacy; the goal is to build useful models for specific contexts.

Conclusion

Höltgen and Williamson conclude that the assumption of data-generating distributions in social ML is a "costly fiction." It obscures the constructed nature of algorithms, fosters a false sense of objectivity, and hinders the development of truly fair and accountable systems. They advocate for a parsimonious framework based on finite populations, which forces researchers to confront the specific choices they make and the actual scope of their predictions.