🤖 machine learning

Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool

This paper presents an open-source unsupervised bias detection tool that successfully identified disparities in a large-scale Dutch government algorithm used to assess student risks, demonstrating its effectiveness for auditing high-stakes decision-making systems even when protected demographic data is unavailable due to privacy constraints.

Original authors: Floris Holstege, Mackenzie Jorgensen, Kirtan Padh, Jurriaan Parie, Krsto Prorokovic, Joel Persson, Lukas Snoek

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Floris Holstege, Mackenzie Jorgensen, Kirtan Padh, Jurriaan Parie, Krsto Prorokovic, Joel Persson, Lukas Snoek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head of a massive school district. You have a limited budget for scholarships, and you need to make sure students who claim to live away from home actually do. To do this, you hire a robot assistant (an algorithm) to scan thousands of student files and flag anyone who might be lying.

The robot is told to look at three simple things:

What kind of school they go to (Vocational vs. University).
How old they are.
How far their home address is from their parents' house.

If the robot thinks a student is "High Risk" of lying, a human investigator shows up to check their mail and ask questions.

The Problem:
For over a decade, this robot was flagging certain students for investigation far more often than others. Specifically, it was targeting students with a non-European migration background. But here's the catch: The robot didn't know their race or background. It was just looking at the three simple rules above.

It turns out, students with non-European backgrounds were statistically more likely to go to vocational schools, be slightly older or younger in specific ways, and live closer to their parents' homes. Because the robot was so strict about these three factors, it accidentally created a "proxy" for race. It was like a bouncer at a club who says, "I'm not judging your race, I'm just checking if you're wearing red shoes," but it just so happens that almost everyone from a specific neighborhood wears red shoes. The result? Indirect discrimination.

The Challenge for Auditors:
Now, imagine you are an auditor hired to check if this robot is fair. You want to see if it's treating different racial groups unfairly. But there's a problem: Privacy laws. The school district (and the law) says you cannot look at the students' race or background data. You are blindfolded. How do you check for bias if you can't see the groups you're worried about?

The Solution: The "Unsupervised" Detective
This paper introduces a clever new tool, like a detective who doesn't need to know the suspects' names to find the criminal. Instead of looking for "Race," the tool looks for patterns in the data itself.

Think of it like sorting a giant bag of mixed marbles. You don't know which marbles are red, blue, or green (because you can't see the labels). But you notice that some marbles are heavy, some are smooth, and some are rough. The tool uses a smart sorting machine (an Unsupervised Clustering Algorithm) to group the marbles based on these physical traits.

Once the marbles are sorted into piles, the detective checks: "Do the marbles in Pile A get rejected by the bouncer much more often than the marbles in Pile B?"

What They Found:
When the researchers used this tool on the Dutch student data:

The tool automatically sorted the students into three groups based on their school type, age, and distance from home.
It found that Group 3 was getting flagged as "High Risk" 50% of the time, while Group 1 was only flagged 3% of the time.
Even though the tool didn't know who was who, the researchers later checked the "hidden" demographic data (which they had access to for this study only) and discovered that Group 3 was overwhelmingly made up of students with non-European backgrounds.

The tool successfully found the "unfair group" without ever being told who they were. It found the "red shoes" without knowing they were red.

Why This Matters:

It works without spying: This method allows auditors to check for bias even when privacy laws prevent them from seeing sensitive data like race or religion.
It's open source: The authors didn't just write a paper; they built a free, easy-to-use tool (like a downloadable app) that anyone can use to check their own algorithms for hidden bias.
It's a starting point: The tool doesn't prove discrimination on its own. It's like a smoke alarm. It goes off and says, "Hey, something weird is happening in this group! A human expert needs to come investigate."

The Takeaway:
Algorithms aren't always "neutral" just because they don't use race as a variable. They can learn to discriminate by using other clues that act as stand-ins. This paper shows us how to use a "blind" detective tool to find those hidden patterns, ensuring that our automated systems don't accidentally become unfair gatekeepers.

1. Problem Statement

The paper addresses the critical challenge of detecting algorithmic bias in high-stakes public sector decision-making when demographic data (protected attributes) is unavailable.

Context: In the Netherlands, the Dutch Executive Agency for Education (DUO) used a rule-based risk profiling algorithm (2012–2023) to determine which students were likely to be improperly receiving "living-away-from-home" grants.
The Issue: The algorithm was found to indirectly discriminate against students with a non-European migration background. However, due to privacy legislation (GDPR), organizations and external auditors often lack access to individual-level demographic data (e.g., race, ethnicity, migration background) required for traditional "supervised" bias detection.
The Gap: Existing bias mitigation and detection methods typically require pre-defined protected groups. There is a need for unsupervised methods that can identify potentially unfair treatment clusters without prior knowledge of demographic labels.

2. Methodology

The authors employed an unsupervised bias detection tool based on Hierarchical Bias-Aware Clustering (HBAC) to re-audit the DUO algorithm.

A. The Algorithm (HBAC)

The core methodology is an iterative clustering algorithm designed to isolate groups with the highest variance in a specific bias metric.

Initialization: All data points start in a single cluster.
Iterative Splitting: The algorithm identifies the cluster with the highest standard deviation of the chosen bias metric and splits it into two sub-clusters.
Clustering Technique: The paper utilizes k-modes (for categorical data) and k-means (for continuous data) to perform the splits.
Termination: The process continues until a predefined number of iterations is reached or a minimum sample size per cluster is met.
Statistical Validation: To prevent false positives (overfitting), the authors introduced methodological improvements:
- Sample Splitting: The dataset is split; 80% is used to train the clustering, and 20% is reserved for testing.
- Hypothesis Testing: A null hypothesis ( $H_0$ ) is tested to see if the bias metric difference between a specific cluster and the rest of the data is statistically significant.
- Correction: A Bonferroni correction is applied to account for multiple hypothesis testing across different clusters.

B. The Case Study (DUO Audit)

Dataset: 298,882 students (focusing on 2014 and 2019 data).
Input Features: The algorithm used three neutral features:
1. Type of education (MBO vocational, HBO higher professional, WO university).
2. Age (categorized ranges).
3. Distance to parents' address (in km).
Bias Metric: Demographic Parity. The metric measured the probability of a student being classified as "high risk" (categories 1 & 2). The goal was to detect if this probability varied significantly across the unsupervised clusters.
Ground Truth (for Validation only): The authors did not use individual demographic data for the clustering. Instead, they used aggregated statistics from Statistics Netherlands (CBS) regarding migration backgrounds post-hoc to validate whether the identified clusters aligned with known disadvantaged groups.

3. Key Results

The unsupervised tool successfully identified clusters that corresponded to the known discriminatory outcomes of the DUO algorithm.

Cluster Identification: The tool identified three distinct clusters in the 2014 dataset.
- Cluster 3 (Highest Bias): 50% of students were classified as "high risk."
- Cluster 1 (Lowest Bias): Only 3% were classified as "high risk."
- Statistical Significance: The difference in risk classification rates between clusters was statistically significant ( $p < 10^{-16}$ ).
Characteristics of the Biased Cluster (Cluster 3):
- Enrolled in vocational education (MBO).
- Younger (15–18) or older (25–50) age groups.
- Living close to their parents' address (0–10km).
Alignment with Demographics:
- Using aggregated CBS data, the authors estimated that 41% of students in Cluster 3 had a non-European migration background, compared to only 10% in Cluster 1.
- The tool successfully identified that the "neutral" features (vocational education, proximity to parents) acted as proxies for non-European migration status, confirming the indirect discrimination found in previous supervised audits.
Replication: Similar results were observed in the 2019 dataset, where the tool identified two clusters with a stark disparity in risk classification (46% vs. 17%) and migration background estimates (36% vs. 14%).

4. Key Contributions

Real-World Large-Scale Audit: The paper provides a comprehensive audit of a government algorithm affecting over 250,000 students, demonstrating how unsupervised learning can detect indirect discrimination in high-stakes policy contexts without access to sensitive individual data.
Open-Source Tool: The authors released an open-source Python library and a web application for the unsupervised bias detection tool. This makes the technology accessible to non-technical auditors and organizations.
Methodological Improvements: The paper refines the original HBAC algorithm by:
- Integrating k-modes for categorical variables.
- Implementing sample splitting and Bonferroni correction to rigorously control false positive rates in statistical testing.
- Providing a framework for assigning new data points to clusters using centroid distances.

5. Significance and Implications

Privacy-Preserving Auditing: The study proves that effective bias auditing is possible even when organizations cannot legally access protected demographic data. This is crucial for compliance with GDPR and the EU AI Act.
Proxy Detection: It demonstrates that unsupervised clustering can uncover "proxy discrimination," where seemingly neutral rules (like distance to parents) inadvertently target protected groups.
Regulatory Impact: The work supports the growing regulatory requirement for algorithmic auditing (e.g., EU AI Act, NYC AEDT law) by providing a scalable, repeatable methodology for internal and external auditors.
Human-in-the-Loop: The authors emphasize that the tool serves as a starting point for deliberation. It flags potential issues, which human experts must then investigate to determine if the disparity constitutes illegal discrimination under specific legal frameworks.

In conclusion, the paper validates that unsupervised learning is a powerful, necessary tool for auditing algorithmic fairness in the modern privacy-constrained landscape, successfully identifying the same discriminatory patterns in the Dutch student grant system that were previously found only through supervised methods.