🤖 machine learning

Data-driven Lake Water Quality Forecasting for Time Series with Missing Data using Machine Learning

This paper presents a data-driven framework for forecasting Secchi Disk Depth in Maine lakes with missing time-series data, demonstrating that a simple ridge regression model using multiple imputation can achieve near-optimal accuracy with minimal training samples and a single predictor, thereby establishing a practical strategy for efficient lake monitoring.

Original authors: Rishit Chatterjee, Tahiya Chowdhury

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rishit Chatterjee, Tahiya Chowdhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for a specific lake, but your notebook is full of holes. Sometimes you forgot to write anything down because it was too cold (ice cover), sometimes you couldn't get there because of a storm, and sometimes you just made a mistake. This is exactly the problem scientists face when trying to monitor lake water quality. They have decades of data, but it's messy, full of gaps, and irregular.

This paper is like a guidebook for how to get the best possible forecast from that messy notebook without wasting time or money. Here is how they did it, explained simply:

1. The Problem: The "Broken Notebook"

Lakes are vital for drinking water and nature, but they are getting sick (algal blooms). To stop this, we need to know how clear the water is. Scientists use a tool called a "Secchi Disk" (a white plate lowered into the water) to measure clarity.

However, the data collected by volunteers is like a diary where the writer only shows up when the weather is perfect. There are huge gaps. If you try to predict the future based on a broken diary, your guesses will be bad.

2. The Fix: "Filling in the Blanks" (Imputation)

Before they could predict anything, the researchers had to fix the holes in the data. They used a smart statistical trick called MICE (Multiple Imputation by Chained Equations).

Think of this like a very smart puzzle solver. If a page is missing a temperature reading, the solver looks at the water depth, the oxygen levels, and the season to guess what that temperature probably was. They didn't just guess randomly; they used the relationships between different measurements to fill in the gaps so the "diary" was complete again.

3. The Experiment: How Much Data Do We Actually Need?

The researchers asked a simple question: "Do we need to read the entire 30-year diary to make a good prediction, or can we just read the last few pages?"

They tested this by training their computer models on different amounts of recent history.

The Result: They found that you don't need the whole 30 years. Once you have about 176 recent samples (roughly a few years of data), adding more old history doesn't really help the prediction get much better. It's like studying for a test: after you've reviewed the last few chapters thoroughly, re-reading the first chapter from three years ago doesn't help you pass the test any more.

4. The Shortcut: Which Measurements Matter?

Next, they asked: "Do we need to measure everything, or just a few things?"
They had 13 different things they could measure (like temperature, oxygen, phosphorus, etc.). Measuring all of them is hard and expensive.

The Result: They found that a tiny group of just four specific measurements was enough to get almost the same accuracy as measuring all 13.
The Ultimate Shortcut: Even better, for most lakes, they only needed one single measurement (specifically, the amount of oxygen in the water, called "Oxic") combined with a short history of recent data to get a prediction that was within 5% of the "perfect" prediction.

5. The "Feasibility Rule": The Magic Formula

The researchers combined these findings into a simple rule for lake managers. Instead of guessing, they now have a formula that says:

"To get a reliable forecast, you only need to measure one specific thing (like oxygen) and look at the last 64 recent samples."

This is a huge deal because it means volunteers and scientists can stop wasting time measuring 13 different things every time they visit a lake. They can focus on the one or two things that actually matter, saving time and money while still getting accurate warnings about bad water quality.

Summary

Think of this research as finding the "cheat code" for lake monitoring.

Before: You needed a perfect, complete diary and had to measure everything to get a good guess.
Now: You can fill in the missing pages with a smart algorithm, and you only need to look at the most recent few pages and measure just one or two things to get a prediction that is almost as good as the perfect version.

This makes it much easier for regular people and small organizations to keep an eye on lake health, even when they can't visit every single day or measure every single chemical.

Technical Summary: Data-driven Lake Water Quality Forecasting for Time Series with Missing Data using Machine Learning

Problem Statement
Volunteer-led lake monitoring programs generate irregular, seasonally intermittent time series data characterized by significant gaps due to ice cover, weather constraints, and human error. These missing data points complicate the forecasting of Secchi Disk Depth (SDD)—a critical metric for water clarity and early warning of harmful algal blooms (HABs). Existing approaches face several limitations: remote sensing is often unreliable for small, shallow, or cloud-prone inland lakes; in-situ studies often rely on small datasets with limited generalizability; and standard missing-data handling (e.g., listwise deletion) degrades predictive performance. Furthermore, current forecasting evaluations often use scale-dependent metrics (like MAE) that hinder cross-lake comparability, and few studies provide a unified strategy to simultaneously determine the minimal required training history and the essential feature set for accurate prediction.

Methodology
The study utilizes a dataset of three decades of in-situ records from Maine lakes, focusing on a subset of 30 lakes with the highest data density. The methodology proceeds through the following stages:

Data Preprocessing and Imputation:
- Exclusions: Chlorophyll-a was excluded to prevent target leakage (as it is a direct proxy for SDD), and "SDD-to-bottom" observations were removed as they are non-determinable.
- Imputation: To handle missingness, the authors employ Multiple Imputation by Chained Equations (MICE). This method models each variable with missing data conditionally on the others, assuming data is Missing At Random (MAR). MICE was selected over deep generative networks due to comparable performance with lower computational cost. SDD itself is treated as the target and is never imputed.
Forecasting Protocol:
- Backward-Expanding Strategy: Unlike traditional methods using the earliest available data, this study uses a "backward" protocol. For each lake, the most recent 5 years of data serve as the test set. The training set consists of the most recent $n$ samples from the pre-test history. This approach prioritizes recent climate and weather patterns.
- Model Selection: Six candidate models were evaluated (ARIMA, SARIMA, TSMixer, Transformer, Random Forest, and Ridge Regression). Ridge Regression demonstrated the best performance in terms of Mean Absolute Error (MAE), normalized MAE (nMAE), and $R^2$ , and was selected as the primary forecaster.
Evaluation Metrics:
- Normalized MAE (nMAE): To enable cross-lake comparison across varying depths and baseline clarity, the study uses nMAE, defined as the average absolute error divided by the average SDD of the test block.
- Performance Threshold: A "minimal" configuration is defined as one where the nMAE is within 5% of the performance achieved using the full historical dataset and all features.
Feature Selection and Joint Optimization:
- Feature Ranking: Features were ranked using Mean Decrease in Impurity (MDI) from a full-feature Ridge model.
- Greedy Forward Selection: The top- $k$ features were tested to find the minimal subset ( $k^*$ ) meeting the 5% tolerance.
- Joint Feasibility Function: A unified rule was introduced to identify the minimal pair $(\hat{n}, \hat{k})$ —the shortest recent history length and the fewest predictors—required to meet the accuracy target. The selection prioritizes minimizing the sample size ( $n$ ) first, then the feature count ( $k$ ).

Key Results

Minimal Sample Size: The study found that predictive performance improves rapidly with the first 50–100 samples and exhibits diminishing returns thereafter. On average, a lake requires approximately 176 training samples (from the most recent pre-test history) to reach within 5% of the full-history accuracy.
Minimal Feature Set: A compact subset of four features—TPEC (Surface Temperature), OXIC (Dissolved Oxygen), SCHMIDT (Schmidt Stability), and CONDUCT (Conductivity)—achieved performance within the 5% tolerance of the 13-feature baseline.
Joint Configuration: When applying the joint feasibility rule, the median minimal configuration across the 30 lakes was 64 recent samples and 1 predictor.
- Specifically, for lakes where a single predictor sufficed (20 out of 30 lakes), OXIC (Dissolved Oxygen) was selected 90% of the time, and TPEC 10% of the time.
- Ten lakes did not meet the 5% tolerance with reduced data and reverted to the full-data/full-feature baseline.

Significance and Claims
The paper claims to provide a practical, data-driven framework for optimizing lake monitoring efforts. By unifying the determination of recent-history length and feature selection under a fixed accuracy target, the study offers a "joint feasibility strategy" that yields simple, efficient rules for setting sampling priorities.

The authors assert that this approach allows researchers and monitoring programs to:

Reduce Resource Burden: Identify that a small number of recent measurements (median ~64) and a single, high-value predictor (often Dissolved Oxygen) can suffice for reliable forecasting, reducing the need for extensive, costly, and labor-intensive sampling campaigns.
Enable Targeted Monitoring: Shift focus from broad, infrequent sampling to targeted, high-frequency monitoring of specific, critical variables.
Support Climate Resilience: Provide a scalable method for early detection of water-quality degradation, aiding in the anticipation of climate-exacerbated algal bloom events.

The study acknowledges limitations, including the reliance on the Missing-At-Random assumption for imputation, the potential bias of MDI feature importance, and the specific design choices regarding the 5% tolerance and lexicographic selection rules. The authors note that these findings are preliminary and based on a subset of 30 lakes, with plans to expand the study to the full database and incorporate physics-informed machine learning in future work.