📄 earth_science

OTTER-NYC: A Minute-Scale Multimodal Dataset for Urban Flood Event Dynamics and Forecasting

This paper introduces OTTER-NYC, a high-quality, minute-scale multimodal dataset for New York City that integrates flood depth, precipitation, terrain, and drainage infrastructure data through a reproducible quality control pipeline to enable detailed analysis of urban flood dynamics and data-driven forecasting.

Original authors: Seongwon Jin, Jiho Park, Juheon Kim, Jehong Bang, Seunghwan An, Jibum Kim

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seongwon Jin, Jiho Park, Juheon Kim, Jehong Bang, Seunghwan An, Jibum Kim

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Why Do We Need This?

Imagine trying to predict a flash flood in a city like New York. It's like trying to catch a lightning bolt in a jar. Rain can start, water can rise to dangerous levels, and drain away again all within 30 minutes.

Most existing weather data is like a slow-motion movie; it updates every hour. By the time that hourly update says "it's raining," the flood might have already peaked and started to recede. To catch these fast-moving urban floods, we need a high-speed camera that updates every single minute.

The authors created OTTER-NYC, a massive, high-speed dataset designed specifically to teach computers how to predict these rapid city floods.

The Ingredients: What's in the Dataset?

Think of building a perfect flood prediction model like baking a very specific cake. You need the right ingredients mixed together at the right time. OTTER-NYC combines four distinct "ingredients" into one neat package:

The "Water Level" (FloodNet): This is the main ingredient. It comes from 293 ultrasonic sensors installed on street corners across NYC. They act like smart rulers, measuring exactly how deep the water is every minute.
The "Rain Gauge" (ASOS): This is the rainfall data. The authors took data from 5 major weather stations (like at airports) and mathematically "spread" it out to cover every single street corner where a sensor is located.
The "Terrain Map" (DEM): This is a super-detailed 3D map of the ground. It tells the computer which streets are low-lying (like a bowl) and which are sloped (like a slide). Water behaves differently in a bowl than on a slide.
The "Drainage System" (Catch Basins): This is a map of all the storm drains and gutters in the city. It tells the computer where the water should be going.

The Magic Trick: The paper's main achievement is aligning these four things. Usually, rain data is at the airport, flood data is on the street, and drain maps are in a different file. The authors stitched them all together so that for every single minute, the computer knows: How much rain fell here? How deep is the water here? How low is the ground here? And how close is the nearest drain?

The Problem: The Sensors Get "Confused"

The raw data from these street sensors is messy. Imagine a child playing with a toy that records water levels. Sometimes the toy glitches, drops the ball, or thinks the water is gone when it's actually still there.

False Zeros: The sensor might say "0 inches of water" for a few seconds even though the street is flooded, just because of a communication glitch.
Spikes: The sensor might suddenly jump from 0 to 10 feet in one second, which is physically impossible.
Too Much "Nothing": Most of the time, the streets are dry. The data is mostly zeros. If you feed this raw data to a computer, the computer gets lazy and just learns to say "No flood" all the time because that's what happens 99% of the time.

The Solution: The "Three-Stage Filter"

To fix this mess, the authors built a Quality Control (QC) Pipeline. Think of this as a three-step sieve or a filter that cleans the data before anyone uses it.

Stage 1: The "Find the Action" Filter.
The computer looks for any time the water level was not zero. It cuts out all the dry days and only keeps the chunks of time where something interesting happened.
Stage 2: The "Glue" Filter.
Sometimes the sensor glitches and says "0" for a minute in the middle of a flood. This stage uses math to "glue" those broken pieces back together, filling in the gaps so the flood looks like one continuous event, not a broken video.
Stage 3: The "Reality Check" Filter.
This is the most important step. The computer asks: "Did it actually rain before the water rose?"
- If the water jumped up but there was no rain in the last 3 hours, the computer assumes the sensor is broken and throws that data away.
- If the water rose after it rained, the computer keeps it.

The Result: They started with nearly 200 million data points. After this cleaning process, they ended up with 31,000 high-quality, verified flood events that are perfect for training AI models.

Did It Work? (The Test Drive)

To prove the dataset is good, the authors tried to use it to predict floods. They taught four different types of AI (like different students) to look at the past 60 minutes of data and guess the water level 10, 30, or 60 minutes in the future.

The Winner: The AI models that used a special learning method called Distribution Focal Loss (DFL) performed the best.
- Analogy: Imagine a student taking a test. A standard student (MSE) tries to guess the exact number. The DFL student guesses the range of possibilities. Because floods are unpredictable and can get very deep very fast, guessing the range (the probability) turned out to be much more accurate than guessing a single number.
The Performance: The best models could predict flood depths with high accuracy, even 60 minutes into the future. This proves the dataset is clean and useful enough to train smart systems.

Summary

OTTER-NYC is a "cleaned, high-definition, minute-by-minute" recipe book for New York City floods. It takes messy sensor data, filters out the errors, matches it with rain and street maps, and creates a perfect training ground for computers to learn how to predict flash floods before they happen.

Where to find it: The authors have made the data and the cleaning code available for anyone to download and use.

Technical Summary: OTTER-NYC Dataset for Urban Flood Event Dynamics and Forecasting

Problem Statement

Existing hydrological and flood datasets are predominantly designed for watershed- or global-scale analyses, often providing complementary products like flood extent or streamflow rather than direct, high-resolution observations. A critical gap exists in datasets that combine minute-scale, per-sensor flood depth observations with precipitation, terrain, and drainage infrastructure data at the urban scale.

Current datasets typically suffer from:

Coarse Temporal Resolution: Most are hourly or coarser, missing peak flood depths that form and recede within tens of minutes, a characteristic of urban flash floods.
Lack of Multimodality: They often focus on single modalities, failing to integrate the complex topographic and drainage infrastructure information that heavily influences urban flood peak depth and duration.
Data Quality Issues: IoT-based sensor networks (like FloodNet) contain noise, including communication interruptions, ultrasonic measurement errors, and "false-zero" observations where flood depth is recorded as zero despite the presence of water.
Class Imbalance: Non-flooded conditions (zero depth) dominate the data distribution, which can induce bias in deep learning models, causing them to overfit to non-flood patterns and underperform during actual flood events.

Methodology

The authors present OTTER-NYC, a high-quality, minute-scale multimodal dataset for New York City (NYC). The construction process involved three main stages:

1. Data Collection and Integration

The dataset integrates four heterogeneous data sources aligned to the FloodNet sensor network (up to 293 sensors) over a period from October 2020 to August 2025:

Flood Depth: Ultrasonic sensor observations (1-min resolution) from FloodNet.
Precipitation: Automated Surface Observing System (ASOS) data from five nearby stations, interpolated to sensor locations using Inverse Distance Weighting (IDW).
Terrain: Digital Elevation Model (DEM) derived attributes (elevation, slope, aspect) at 1-ft resolution.
Drainage Infrastructure: NYC Citywide Catch Basins (NYC CCB) data, used to derive catch basin density and nearest basin distance for each sensor.

2. Spatiotemporal Preprocessing

Temporal Alignment: All dynamic variables were standardized to a uniform 1-minute UTC+0 grid. FloodNet data was resampled by averaging within bins, while ASOS precipitation was treated as interval-specific values.
Spatial Alignment: Precipitation was interpolated using Haversine distance. Terrain and infrastructure variables were extracted or computed based on sensor coordinates (transformed to NY State Plane for DEM processing).
Variable Encoding: Static variables (e.g., aspect) were encoded to handle circularity and missing values on flat terrain.

3. Three-Stage Quality Control (QC) Pipeline

To transform raw, noisy observations into a physically meaningful event-based dataset, a reproducible three-stage QC pipeline was implemented:

Stage 1 (Segment Extraction): Identified continuous non-zero segments. Boundary zeros were inserted to facilitate the detection of false-zero errors in the next stage.
Stage 2 (Temporal Integration & Gap Interpolation): Merged adjacent segments separated by short gaps (≤1 min) and interpolated false-zero values within segments using time-weighted linear interpolation. This restored the continuity of flood events.
Stage 3 (Quality Filtering & Rainfall Validation):
- Hardware Threshold: Removed segments with flood depths exceeding the sensor installation height (3,000 mm).
- Laplace Threshold: Applied a change-rate filter based on a Laplace distribution fitted to flood depth changes ( $\Delta d$ ). Segments with changes exceeding $\tau = 87.88$ mm (3.5 $\sigma$ ) were removed to eliminate physically implausible spikes.
- Rainfall Validation: Retained only segments where at least one time step of precipitation > 0 occurred within the 3 hours preceding the flood peak. This ensured hydrometeorological plausibility.

Key Contributions

OTTER-NYC Dataset: A multimodal dataset containing approximately 31,085 valid urban flood events across 264 sensors, featuring minute-scale temporal resolution. It includes dynamic variables (flood depth, precipitation) and static variables (terrain, drainage infrastructure).
Reproducible QC Pipeline: A three-stage pipeline that effectively removes sensor noise, corrects false-zero observations, and filters non-physical signals, resulting in a dataset with improved signal stability and hydrometeorological plausibility.
Benchmarking Results: The paper provides technical validation through urban flood forecasting experiments using four deep learning models (ANN, GCN, LSTM, GRU) and two loss functions (MSE, Distribution Focal Loss).

Results and Technical Validation

Data Quality Improvement: The QC pipeline significantly improved data quality metrics:
- Maximum flood depth gradient decreased from 8,103 mm/min to 87 mm/min.
- Zero-inflation ratio dropped from 29.06% to 9.34%.
- The fraction of segments with non-zero pre-peak rainfall increased from 21.4% to 100.0%.
Forecasting Performance:
- Loss Function: Models trained with Distribution Focal Loss (DFL) consistently outperformed those trained with Mean Squared Error (MSE) across all architectures and forecast horizons (10, 30, and 60 minutes). DFL reduced RMSE from 11.85 mm to 8.65 mm on average and improved Nash-Sutcliffe Efficiency (NSE) from 0.686 to 0.841.
- Model Architecture: Recurrent architectures (LSTM and GRU) demonstrated the most stable performance. The best overall results were achieved by LSTM-DFL and GRU-DFL, with the LSTM-DFL model achieving an NSE of 0.889 at the 60-minute forecast horizon.
- Graph Models: Graph Convolutional Networks (GCN) showed lower performance than recurrent models, suggesting that simple distance-based graphs may not fully capture hydrological connectivity in urban settings.

Significance and Claims

The paper claims that OTTER-NYC successfully addresses the lack of high-resolution, multimodal datasets for urban flood dynamics. By providing a dataset that captures the abrupt and compound characteristics of urban floods (flash-flood behavior) and integrating critical infrastructure data, it supports:

Data-Driven Forecasting: Enabling the training of deep learning models for minute-scale urban flood nowcasting and early warning systems.
Event Dynamics Analysis: Facilitating the study of flood formation mechanisms under different rainfall conditions (e.g., widespread cyclonic storms vs. localized intense rainfall).
Methodological Transferability: The proposed QC pipeline offers a framework for cleaning and refining data from other low-cost IoT-based ultrasonic sensor networks.

The authors emphasize that the dataset is a reusable resource for future studies on urban flood forecasting and risk management, specifically highlighting its utility in developing models that require short forecast horizons and high spatiotemporal resolution.