🤖 machine learning

Synthetic Flight Data Generation Using Generative Models

This study demonstrates that generative models, specifically the Tabular Variational Autoencoder (TVAE) and Gaussian Copula (GC), can effectively generate high-quality synthetic flight data that preserves statistical properties and enables predictive modeling of flight delays with accuracy comparable to real-world data, offering a scalable solution to data scarcity and confidentiality challenges in aviation research.

Original authors: Karim Aly, Alexei Sharpanskykh

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Karim Aly, Alexei Sharpanskykh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent a new, perfect recipe for a dish that is very hard to make. The problem? You only have a few old, secret recipe books, and many of the pages are torn out or locked away because they contain confidential information. Plus, the most interesting parts of the dish (like a rare spice that only appears once in a thousand meals) are almost impossible to find in your existing books.

This is exactly the situation facing airline researchers today. They want to use Artificial Intelligence (AI) to predict flight delays, cancellations, and diversions, but they are stuck because:

Real data is scarce: There aren't enough records of rare events (like a plane being diverted due to a storm).
Privacy is tight: Airlines and governments won't share their real flight logs because they are confidential.

The Solution: The "Fake" Recipe Book
This paper is about a team of researchers at TU Delft who decided to build a "synthetic" (fake) recipe book. Instead of stealing real secrets, they used advanced AI to learn the patterns of real flights and then write thousands of brand-new, fake flight records that look and feel exactly like the real ones, but contain no actual private information.

They tested two different "AI Chefs" to see which one could write the best fake recipes:

The Two AI Chefs

The Gaussian Copula (GC) Chef:
- Style: This chef is a statistical genius. It looks at the ingredients and calculates the exact mathematical probability of how they should mix.
- Pros: It creates fake data that is incredibly realistic. If you tried to tell the difference between its fake flights and real flights, you'd be hard-pressed. It's like a master forger who can replicate a painting so perfectly that even an art expert can't spot the difference.
- Cons: It's slow and expensive. It can only cook a small batch at a time. If you ask it to cook for a whole stadium, it crashes the kitchen.
The TVAE (Tabular Variational Autoencoder) Chef:
- Style: This chef is a deep learning wizard. It uses a neural network (a brain-like structure) to "dream" up new flights based on what it has seen before.
- Pros: It is fast and scalable. It can cook a massive banquet for thousands of people in the time it takes the GC chef to make a single appetizer. It can handle huge datasets easily.
- Cons: It's a bit finicky. If you don't give it the right ingredients (features) or the right format, it gets confused and might forget to include important details, like "delayed departures."

The Great Taste Test (The Experiments)

The researchers set up a four-stage competition to see which chef was better:

Diversity Check: Did the fake flights cover all the different scenarios (rain, snow, busy airports, empty skies)?
- Result: The TVAE chef needed help to get this right. At first, it forgot to create "delayed" flights. But once the researchers gave it better instructions, it started creating a diverse mix. The GC chef got this right immediately.
Statistical Similarity: Do the fake numbers match the real numbers? (e.g., Is the average delay 15 minutes in both?)
- Result: The GC Chef won. Its fake data was statistically almost identical to the real data.
The "Spot the Fake" Test (Fidelity): They trained human-like AI detectives to try to find the fake flights.
- Result: The GC Chef's fake flights were so good that the detectives got confused and failed to spot them. The TVAE's flights were slightly easier to spot, but still very good.
The Utility Test (The Most Important Part): This is the "Does it work?" test. They trained a prediction model using only the fake data and then tested it on real flights.
- Result: This is where the plot twist happened.
  - The GC Chef made beautiful, perfect fake data, but because it could only make a small amount of it, the prediction model didn't learn enough patterns. It was like trying to learn to drive by only practicing in a tiny parking lot.
  - The TVAE Chef made slightly less "perfect" fake data, but it made so much of it (52,000 flights vs. 2,000). Because the model had so much practice, it learned the patterns of flight delays incredibly well. The TVAE model predicted real delays just as accurately as a model trained on real data.

The Big Takeaway

The paper concludes that synthetic data is a game-changer for aviation.

The GC Chef is great for making small, perfect samples to study specific details.
The TVAE Chef is the winner for training AI models because it can generate massive amounts of data, allowing the AI to learn complex patterns that are rare in the real world.

In simple terms: You don't need a million real flight records to train an AI to predict delays. You just need a smart AI (like TVAE) to generate a million fake records that are "good enough." This solves the privacy problem and the data scarcity problem, letting researchers build better tools to keep our skies running smoothly.

The Future: The researchers plan to keep refining these AI chefs, making them faster and even better at creating realistic scenarios (like extreme weather) that haven't happened yet, helping airlines prepare for the future.

1. Problem Statement

The aviation industry faces significant challenges in developing robust machine learning (ML) models due to data scarcity and confidentiality.

Data Limitations: Access to comprehensive, high-quality flight data (e.g., schedules, delays, diversions) is restricted by privacy regulations, commercial competition, and proprietary barriers.
Class Imbalance: Critical events like flight cancellations, diversions, and severe delays are rare in historical datasets, making them difficult to predict accurately using standard ML techniques.
Need for Synthetic Data: There is a critical need for Synthetic Data Generation (SDG) to augment historical datasets, balance class distributions, and provide a privacy-preserving alternative for training predictive models without compromising real-world data security.

2. Methodology

The study proposes a four-stage evaluation framework to assess the quality of synthetic flight data generated by two distinct generative models: Tabular Variational Autoencoder (TVAE) and Gaussian Copula (GC).

A. Data Preparation

Source: The study utilized the "TranStats Database for Airline On-Time Performance" (BTS), focusing on U.S. domestic flights in New York State during January 2023.
Dataset: Approximately 61,000 flights with 109 features, reduced to 30 key features (categorical, numerical, and datetime).
Preprocessing:
- Time features were standardized to Coordinated Universal Time (UTC).
- Missing values were handled by splitting columns into a value column and a binary "missing" indicator column.
- Feature Engineering: To preserve complex dependencies (e.g., ensuring departure time aligns with arrival time), the dataset was organized into three distinct DataFrames (df_utc_ts, df_utc_d, df_utc_d_2) with varying feature combinations (timestamps vs. durations).

B. Generative Models

Gaussian Copula (GC): A statistical model that decomposes multivariate distributions into marginal distributions and dependency structures. It uses Kernel Density Estimation (KDE) for marginals and a Gaussian Copula function to model correlations.
Tabular Variational Autoencoder (TVAE): A deep learning model adapted for tabular data. It uses an encoder to map input data to a latent probabilistic distribution and a decoder to reconstruct the data, optimizing the Evidence Lower Bound (ELBO).

C. Experimental Design

Five experiments were conducted to test different feature sets and model combinations:

Experiments 1–3: TVAE trained on different feature sets (df_utc_ts, df_utc_d, df_utc_d_2).
Experiments 4–5: GC trained on df_utc_ts and df_utc_d_2.
Scale Constraint: TVAE was trained on the full dataset (~61k rows), while GC was limited to 5,000 rows due to computational memory constraints.

D. Four-Stage Evaluation Framework

Diversity: Assessed via Principal Component Analysis (PCA) and class balance checks to ensure synthetic data captures the variability and distribution of rare events (e.g., delays) found in real data.
Statistical Similarity: Measured using Total Variation Distance (TVD) for categorical data and Kolmogorov-Smirnov (KS) tests for numerical data. Correlation and Contingency Similarity metrics evaluated pairwise relationships.
Fidelity: Evaluated by training seven classifiers (e.g., Random Forest, SVM, Logistic Regression) to distinguish between real and synthetic data. Lower accuracy indicates higher fidelity (i.e., the models cannot tell them apart).
Utility: The most critical stage. Regression models were trained on synthetic data and tested on real data (TSTR scenario) to predict flight arrival delays. Performance was compared against a baseline trained on real data (TRTR) using MAE, RMSE, and $R^2$ .

3. Key Contributions

Comprehensive Framework: Established a rigorous, four-stage benchmark specifically for evaluating synthetic flight data, moving beyond simple statistical similarity to include predictive utility.
Model Comparison: Provided a direct comparison between a deep learning approach (TVAE) and a statistical approach (GC) in the context of air transportation.
Feature Sensitivity Analysis: Demonstrated that the representation of time features (timestamps vs. durations) significantly impacts the ability of generative models to learn operational dependencies (e.g., taxi-in time vs. arrival airport).
Scalability vs. Fidelity Trade-off: Identified a critical trade-off where high-fidelity statistical models (GC) may fail in utility due to scalability limitations, while scalable models (TVAE) offer practical utility despite slightly lower statistical fidelity.

4. Key Results

Statistical Similarity & Fidelity:
- GC outperformed TVAE in statistical similarity (89.06% vs. 81.47%) and fidelity (lower classifier accuracy in distinguishing real vs. synthetic).
- GC effectively preserved marginal distributions and bivariate relationships even with complex datetime inputs.
Diversity:
- TVAE initially struggled with "posterior collapse" when using datetime formats, failing to capture delay clusters. Performance improved significantly when switching to duration-based features (df_utc_d_2) and explicitly including "Taxi In Time."
- GC maintained diversity across all configurations but was limited by the small sample size (5,000 rows).
Predictive Utility (The Critical Finding):
- TVAE (Experiment 3): Models trained on TVAE synthetic data achieved comparable or superior accuracy to real-data baselines when predicting flight delays (e.g., $R^2$ of 0.79 vs. 0.76 baseline).
- GC (Experiment 5): Despite higher statistical fidelity, GC-generated data resulted in poor predictive performance ( $R^2$ dropped to 0.66).
- Reason: The computational limitation restricted GC to a small dataset (~2,400 cleaned rows). This small sample size failed to capture the full variability of flight routes and patterns, leading to underfitting in downstream tasks. TVAE, trained on ~52,000 rows, captured the necessary complexity.

5. Significance and Conclusion

Viability of Synthetic Data: The study proves that synthetic flight data can effectively replace real data for training predictive models, provided the generative process is scalable enough to capture the full distribution of flight patterns.
Scalability is Paramount: For aviation applications involving complex, high-dimensional data, the ability to generate large-scale datasets (TVAE) is more valuable than the marginal statistical gains of smaller, high-fidelity datasets (GC).
Future Directions: The authors suggest leveraging increased computational power to scale GC models, refining TVAE to prevent posterior collapse, and exploring the generation of entirely new, plausible flight routes (diversions/cancellations) that do not exist in historical records.

Final Takeaway: Synthetic data generation is a viable solution for aviation data scarcity. However, the choice of model must balance statistical fidelity with computational scalability; a model that generates a small, perfect dataset is less useful than one that generates a large, "good enough" dataset capable of training robust predictive models.