🤖 machine learning

Peoples Water Data: Enabling Reliable Field Data Generation and Microbial Contamination Screening in Household Drinking Water

This study presents a two-stage machine-learning framework that leverages low-cost physicochemical and contextual indicators to predict *E. coli* contamination in household drinking water in Chennai, India, thereby enabling scalable, reliable, and AI-supported microbial screening in resource-constrained settings.

Original authors: Suzan Kagan, Shira Spigelman, Sankar Sudhir, Thalappil Pradeep, Hadas Mamane

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Suzan Kagan, Shira Spigelman, Sankar Sudhir, Thalappil Pradeep, Hadas Mamane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a few bad apples in a massive orchard, but you don't have the time or money to taste every single one. You need a way to spot the orchard sections that are most likely to have the bad apples so you can focus your tasting efforts there.

This paper is about building a smart "bad apple detector" for drinking water in homes in Chennai, India.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The Invisible Danger

Drinking water can look, smell, and taste perfectly clean, but still be full of invisible germs (like E. coli) that make people sick. Usually, checking for these germs requires sending water to a fancy lab, which is expensive and slow. In many neighborhoods, people just don't have access to these labs.

The researchers wanted a way to say, "Hey, this bucket of water looks suspicious. Let's test it in the lab immediately," without having to test every bucket.

2. The Team: Students as "Water Detectives"

Instead of hiring expensive scientists to go door-to-door, the project trained students to be "Water Detectives."

The Mission: These students went into neighborhoods with simple, low-cost kits to measure things like how cloudy the water is, how salty it tastes (conductivity), and how acidic it is (pH).
The AI Coach: To make sure the students did a good job, the team built an AI Coach. Think of this like a GPS for data. If a student forgot to take a photo, entered a weird number, or took too long, the AI Coach would immediately beep and say, "Wait a minute, something looks wrong here!" This helped catch mistakes while they were happening, not weeks later.

3. The Secret Sauce: The Two-Stage "Sniffer"

The researchers didn't just build one computer model; they built a two-step detective team.

Step 1: The "General Smell" Detector (Total Coliforms)
First, the computer looks at the water's physical traits (cloudiness, saltiness, etc.) and asks: "Does this water smell like it has any bacteria at all?"
- Analogy: Imagine a security guard at a club checking if someone looks like they might be up to no good. They aren't looking for a specific crime yet, just a general "suspicious vibe."
Step 2: The "Specific Criminal" Detector (E. coli)
The computer takes the answer from Step 1 and combines it with more clues (like: "Do they have young children in the house?" "How long has the water been sitting in a bucket?" "Is the container clean?"). Then it asks: "Given that it smells suspicious, is it specifically the dangerous E. coli germ?"
- Analogy: Now the security guard has a specific list of known criminals. If the person looked suspicious in Step 1, the guard checks the list to see if they are the specific criminal they are looking for.

Why two steps? Because finding any bacteria is a huge clue that the dangerous bacteria might be there too. By using the first step as a "hint" for the second step, the computer gets much smarter.

4. The Results: Catching the Bad Guys

The system was tested on over 2,000 water samples.

The Goal: The researchers cared more about not missing a sick person (catching the bad water) than about accidentally flagging clean water. They wanted to catch 9 out of 10 bad samples, even if it meant checking a few clean ones by mistake.
The Outcome: The two-step system was very good at this. It successfully identified almost 92% of the contaminated water.
The "Magic" Feature: The computer learned that things like cloudy water and how people store their water (e.g., leaving the bucket open near the cooking area) were huge red flags. It also learned that knowing the neighborhood location helped predict risks.

5. Why This Matters

This isn't just a computer game; it's a life-saving tool.

For the Future: Instead of testing 1,000 buckets of water in a lab (which costs a fortune), a community can use this AI tool to test the water's "vibe" first. Then, they only send the top 100 most suspicious buckets to the lab.
The Big Picture: It turns a complex scientific problem into a simple checklist. It empowers regular people (students and families) to protect their health using cheap tools and smart math.

In a nutshell: The researchers built a digital filter that uses simple clues (cloudy water, storage habits) and a two-step thinking process to find dangerous germs in drinking water, ensuring that limited lab resources are used exactly where they are needed most.

1. Problem Statement

Unsafe drinking water remains a critical global public health issue, particularly in low-resource regions where routine microbiological surveillance is limited. While Escherichia coli (E. coli) is the standard indicator for fecal contamination, laboratory-based testing is often inaccessible at scale.

The Gap: Existing predictive models often focus on source water, irrigation, or centralized municipal supplies. They frequently fail to account for Point-of-Use (POU) contamination, where water quality deteriorates due to storage conditions, handling practices, and infrastructure issues within the household.
The Challenge: There is a need for a scalable, low-cost screening tool that can prioritize which household samples require expensive, confirmatory laboratory testing, specifically in decentralized urban environments like Chennai, India.

2. Methodology

A. Data Collection & The "People's Water Data" (PWD) Initiative

Source: Data was collected from 3,023 household samples in Chennai, India, via the PWD initiative.
AI-Supported Field Implementation: A novel operational layer was introduced where student teams were supported by:
- AI Field Assistant: A knowledge-base tool guiding students on sampling, testing, and troubleshooting.
- Real-Time Quality Control (QC): An automated system screening submissions for protocol deviations (e.g., GPS errors, missing photos, implausible values) in real-time.
Variables:
- Target: Binary presence of Total Coliforms and E. coli (measured via AquaGenx field kits).
- Predictors: Physicochemical parameters (Turbidity, TDS, EC, pH, ORP, Hardness, Alkalinity), water source characteristics, storage/handling practices, and demographic indicators.
Data Cleaning: After harmonization, technical cleaning, and outlier removal, 2,207 valid samples remained. RO-treated samples were excluded to avoid confounding variables.

B. Statistical Analysis & Modeling Framework

Key Insight: A strong statistical association was found between Total Coliforms and E. coli (Odds Ratio = 14.28, $p < 0.0001$ ).
Two-Stage Machine Learning Pipeline:
- Stage 1: Train classifiers (Random Forest, HistGradientBoosting, XGBoost, LightGBM) to predict the probability of Total Coliform presence.
- Stage 2: Use the out-of-fold (OOF) predicted probabilities from Stage 1 as an auxiliary feature (PTC) to train models predicting E. coli presence.
Validation Strategy:
- Stratified 5-fold cross-validation with nested validation for hyperparameter tuning and early stopping.
- Metric Selection: Prioritized the $F_2$ score (weighting Recall > Precision) to minimize false negatives in a public health context, alongside ROC-AUC and PR-AUC.
- Hypothesis Testing: Used stratified paired bootstrap with False Discovery Rate (FDR) correction to compare model performance.

3. Key Contributions

POU Focus: Unlike previous studies focusing on source water, this framework targets the household level, integrating behavioral and storage variables which are critical for POU contamination.
Two-Stage "Model-as-Feature" Architecture: Leveraging the strong correlation between Total Coliforms and E. coli to create a more robust predictor for E. coli.
AI-Integrated Field Workflow: Demonstrated a novel approach where AI supports data generation upstream (via student guidance and real-time QC), not just downstream for analysis. This improved data reliability and protocol adherence in a decentralized setting.
Contextual Integration: Successfully combined low-cost physicochemical sensors with demographic and behavioral data, showing that contextual variables significantly improve predictive power over physical metrics alone.

4. Results

Performance Metrics

The selected pipeline (LightGBM for Stage 1 $\to$ HistGradientBoosting for Stage 2) achieved the following on the held-out test set (442 samples):

ROC-AUC: 0.768
PR-AUC: 0.872 (indicating strong performance on the imbalanced dataset)
Recall: 0.919 (High sensitivity, crucial for screening)
Precision: 0.758
$F_2$ Score: 0.881
Brier Score: 0.174 (indicating good probability calibration)

Comparative Analysis

Impact of PTC Feature: Adding the Stage 1 probability (PTC) to Stage 2 improved the ROC-AUC from 0.738 to 0.818 and significantly stabilized the decision threshold (reducing standard deviation from 0.141 to 0.028).
Feature Importance (SHAP Analysis):
- Stage 1: Dominated by physicochemical indicators (TDS, pH, Conductivity, ORP, Turbidity).
- Stage 2 (without PTC): Relied heavily on spatial cues (Latitude/Longitude) and physical metrics.
- Stage 2 (with PTC): The PTC feature became the dominant predictor, simplifying the feature structure and confirming that Total Coliform probability is the strongest signal for E. coli.
Subgroup Analysis: The model maintained high recall across different household treatment groups (e.g., Boiling, Household treatment), demonstrating robustness across varying user behaviors.

5. Significance and Conclusion

Scalable Decision Support: The framework provides a practical tool for resource-constrained environments. It allows field teams to screen thousands of households using low-cost sensors and surveys, flagging only the highest-risk samples for confirmatory laboratory testing.
Public Health Impact: By prioritizing Recall (minimizing missed contaminated cases), the model aligns with public health safety objectives, even if it generates some false positives (which trigger further testing).
Methodological Innovation: The study proves that integrating AI into the data collection phase (via real-time QC and student guidance) is as critical as the modeling phase for generating reliable scientific datasets in decentralized monitoring.
Transferability: While calibrated for Chennai, the framework is designed to be adaptable to other regions through local recalibration of thresholds and inclusion of region-specific contextual variables.

Limitations: The model is specific to Chennai and requires external validation. It lacks certain variables like free residual chlorine and water temperature. It is a screening tool, not a replacement for confirmatory lab analysis.