3W Dataset 2.0.0: a realistic and public dataset with rare undesirable real events in oil wells

This paper introduces version 2.0.0 of the 3W Dataset, a publicly available, expert-labeled multivariate time series resource containing rare undesirable real events in oil wells, designed to advance AI-driven early detection methodologies and mitigate economic, environmental, and safety risks in the industry.

The Gist

Imagine the oil industry as a massive, complex orchestra. Each oil well is a musician playing a specific instrument. Usually, they play a smooth, predictable melody (normal operation). But sometimes, a musician hits a wrong note, the instrument jams, or the sheet music gets torn. These are the "undesirable events"—like a valve closing unexpectedly or a blockage forming in a pipe. If the conductor (the oil company) doesn't notice these mistakes immediately, the whole orchestra could crash, leading to wasted money, environmental spills, or even injury.

This paper introduces a new, upgraded "sheet music library" called the 3W Dataset 2.0.0. It's a public collection of recordings (data) that helps computers learn to spot these mistakes before the orchestra crashes.

Here is a breakdown of what this paper claims, using simple analogies:

1. What is this Dataset?

Think of the dataset as a giant library of time-traveling recordings.

• The Recording: Instead of audio, it records 27 different "sensors" (like pressure, temperature, and flow rates) from oil wells, ticking away every single second.
• The Label: Every recording comes with a "sticker" from an expert human. The sticker says: "This part was normal," "This part was a sudden valve closure," or "This part was a blockage forming."
• The Goal: The goal is to teach Artificial Intelligence (AI) to read these stickers and learn the patterns so it can spot a problem in a new recording without needing a human to look at it first.

2. The Three Types of "Musicians" (Data Sources)

The paper explains that they didn't just grab recordings from real life; they used three different methods to build this library, each with its own flavor:

• Real Life (The Live Concert): These are actual recordings from real oil wells owned by Petrobras (a Brazilian oil giant).
  • The Catch: Real life is messy. Sometimes the microphone (sensor) stops working, or the tape gets stuck (frozen data). The authors intentionally kept these messes in the data. Why? Because they want to train AI to be tough enough to handle a real, messy concert hall, not just a perfect studio.
• Simulated (The Rehearsal): Some problems (like a specific type of pipe blockage) are so rare in real life that they almost never happen. To get enough examples, the team used a super-computer simulator (OLGA) to "rehearse" these disasters.
  • The Catch: These are perfect, clean recordings. No static, no missing notes. They are great for teaching the AI what a "perfect" disaster looks like.
• Hand-Drawn (The Sketch): Some problems are so weird that even the super-computer can't simulate them accurately. So, human experts took a pen and paper and drew what the sensor readings should look like during these rare events.
  • The Catch: These are like a musician's sketch of a song. They capture the essence and the shape of the problem, even if they aren't a real recording.

3. What's New in Version 2.0.0?

The first version of this library came out in 2019. This paper announces Version 2.0.0, which is like a major expansion pack for a video game. Here is what changed:

• More Wells: They doubled the number of real oil wells they recorded (from 21 to 42).
• More Sensors: They added 20 new "microphones" (variables) to the recordings, giving a much clearer picture of what's happening.
• New Problems: They added a new type of disaster to the list: "Hydrate in Service Line" (a specific type of ice-like blockage).
• Better Labels: They added a new type of "sticker" called a State Label. Before, the stickers just said "Normal" or "Broken." Now, they also say what the well was doing at that moment (e.g., "We are flushing it with diesel," "We are shutting it down," or "We are restarting"). This helps the AI understand the context, not just the noise.
• Better Format: They switched from old, clunky file formats (CSV) to a modern, high-speed format called Parquet, which is like switching from a floppy disk to a solid-state drive.

4. Why Does This Matter?

The paper claims that having this specific, high-quality library allows researchers and companies to:

• Train Better AI: Because the data includes "messy" real-world problems, the AI trained on it won't get confused when it encounters real oil wells.
• Detect Problems Early: The AI can learn the subtle "tremors" in the data that happen before a disaster strikes, allowing operators to fix it early.
• Share Knowledge: Because this is a public dataset, anyone (students, startups, other oil companies) can download it and try to build better detection tools.

5. What the Paper Does Not Claim

• It does not claim that this AI is currently running in every oil well in the world. It is a tool for research and development.
• It does not claim to have solved the problem of oil spills or accidents. It claims to provide the data necessary to build solutions that might prevent them.
• It does not discuss medical uses or other industries, even though the technology (time-series analysis) could theoretically be used elsewhere. The paper focuses strictly on oil wells.

In short: This paper is an invitation to the world to use a massive, upgraded, and very realistic library of oil well "soundtracks" to teach computers how to be better detectives, spotting trouble in oil wells before it becomes a catastrophe.

Technical Summary

1. Problem Statement

In the oil and gas industry, undesirable events in oil wells (e.g., equipment failures, flow instabilities, hydrate formation) lead to significant economic losses, environmental accidents, and human casualties. While Artificial Intelligence (AI) and Machine Learning (ML) offer promising solutions for the Early Detection of these events through Abnormal Event Management (AEM), their success is heavily dependent on the availability of high-quality, labeled, and realistic datasets.

Prior to this work, there was a critical lack of public datasets containing real-world industrial data with rare, undesirable events. Existing datasets often lacked the complexity of real operations (e.g., missing values, sensor freezes, noise) or were not publicly accessible, hindering the development of robust, generalizable detection models.

2. Methodology

The paper describes the evolution and structure of the 3W Dataset 2.0.0, a Multivariate Time Series (MTS) dataset managed by Petrobras. The dataset is constructed using three distinct data generation methods to ensure a balance between realism, volume, and coverage of rare events:

A. Data Sources and Generation Methods

1. Real Instances (Real-world Data):

   • Source: Extracted from Petrobras' Plant Information Management Systems (PIMS), specifically AVEVA PI System environments.
   • Characteristics: These instances preserve the "messiness" of real industrial data, including frozen variables, missing values, and outliers, which are left untreated to challenge algorithms to handle real-world conditions.
   • Sampling: 1 Hz frequency (1-second intervals).
   • Labeling: Performed by Petrobras experts using a proprietary web tool, validated by an expert committee, and finalized by a curator.

2. Simulated Instances (Physics-based Simulation):

   • Source: Generated using OLGA, a dynamic multiphase flow simulator widely used in the industry.
   • Characteristics: Data is perfectly periodic with no missing values or noise. Used to generate data for rare events that are difficult to capture frequently in real operations.
   • Labeling: Fully automated based on simulation parameters.

3. Hand-Drawn Instances (Expert Knowledge):

   • Source: Created by Petrobras experts using a proprietary image-processing tool. Experts manually drew time-series graphs representing specific event signatures.
   • Characteristics: Digitized from paper charts. Used for events that are both rare and difficult to simulate accurately.
   • Labeling: Derived directly from expert markings on the original graphs.

B. Data Structure and Nomenclature

• Format: The dataset is organized into Apache Parquet files (column-oriented, compressed with Brotli) for efficient storage and retrieval, replacing the CSV format of version 1.0.0.
• Variables: Each instance contains 27 variables representing physical quantities (pressure, temperature, flow rate, valve states) across the well production system (subsea Christmas tree, umbilical, etc.).
• Labels: The dataset introduces a dual-labeling system:
  • Class Labels: Indicate the nature of the event (e.g., Normal Operation, Abrupt Increase of BSW, Hydrate formation). Codes range from 0 (Normal) to 9 (Specific Events) and 101–109 (Transient conditions).
  • State Labels: Indicate the operational status of the well (e.g., Open, Shut-In, Flushing Diesel/Gas, Bullheading, Depressurization).

C. Scope

The dataset covers satellite-type offshore oil-producing wells operating without a manifold. It includes data from 42 different wells spanning from 2011 to 2023.

3. Key Contributions

The primary contribution of this paper is the release and detailed documentation of 3W Dataset 2.0.0, which represents a significant upgrade over version 1.0.0. Key advancements include:

• Expanded Event Coverage: Introduction of a new event type, "Hydrate in Service Line," and increased instances for existing rare events.
• Increased Data Volume:
  • Total instances increased from ~1,984 to 2,228.
  • Real instances increased by 94; Simulated instances increased by 150.
  • The number of covered real wells doubled (from 21 to 42).
• Enhanced Variable Set: Discontinued one variable and added 20 new variables, bringing the total to 27, providing a more comprehensive view of the production system.
• New Labeling Dimension: The addition of State Labels allows researchers to correlate specific operational procedures (e.g., "Bullheading") with variable behaviors, enabling more nuanced model training.
• Infrastructure:
  • Migration to Parquet format for better performance.
  • Development of the 3W Toolkit (Python-based) to facilitate data loading, visualization, and metric calculation.
  • Establishment of a 3W Community and an Open Lab module to foster collaborative research.

4. Results and Statistics

The paper provides a statistical overview of the dataset's composition and quality:

• Distribution: The dataset contains 1,119 real instances, 1,089 simulated instances, and 20 hand-drawn instances.
• Realism Metrics: The real data subset exhibits significant "real-world" challenges:
  • 65.90% of variable observations are missing.
  • 9.77% of variables are frozen.
  • 5.26% of observations are unlabeled.
• Community Impact: The paper highlights the dataset's adoption, citing over 100 published works (including theses, journal articles, and conference papers) that utilize the 3W Dataset, demonstrating its growing role as a foundational benchmark for Fault Detection and Diagnosis (FDD) in the oil industry.

5. Significance

The 3W Dataset 2.0.0 is significant for several reasons:

1. Bridging the Data Gap: It addresses the scarcity of public, labeled industrial time-series data, specifically for rare and critical failure modes in oil wells.
2. Realism for Robustness: By intentionally preserving data imperfections (missing values, frozen sensors), it forces the development of ML models that are robust enough to be deployed in actual industrial settings, rather than just working on clean, synthetic data.
3. Transfer Learning Potential: The large volume and diversity of the data make it an ideal "pre-training" resource for Transfer Learning frameworks, allowing models trained on this dataset to be fine-tuned for specific, data-scarce target tasks.
4. Open Science and Collaboration: Managed under a CC BY 4.0 license and hosted on GitHub/Figshare, it promotes open innovation, allowing researchers, startups, and operators to collaborate on improving Early Detection systems, ultimately enhancing safety and reducing economic losses in the energy sector.

In conclusion, the paper presents a mature, community-driven, and technically rigorous dataset that serves as a critical enabler for advancing AI-driven safety and efficiency in the oil and gas industry.