🤖 machine learning

Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

This paper proposes and demonstrates an optimized Transformer neural network architecture implemented on a PYNQ-Z2 FPGA to enable efficient, low-latency real-time outlier detection in financial time series data.

Original authors: Ilia Sobakinskikh, Paul Alexander Bilokon

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ilia Sobakinskikh, Paul Alexander Bilokon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to spot a thief in a crowded city. The city is a river of numbers, constantly flowing with the prices of stocks, the temperature of servers, or the number of people in a taxi line. Most of the time, this river flows smoothly, but occasionally, a giant rock—a "glitch" or an "outlier"—jumps into the stream, causing a splash that could ruin everything downstream. In the world of finance and data science, these splashes are dangerous; they can trick computers into making bad decisions. For a long time, catching these splashes was slow, like trying to find a needle in a haystack by hand. But recently, scientists have built a super-smart detective called a "Transformer." Think of a Transformer as a genius librarian who can read a million books at once and instantly remember how a story from page one connects to page one thousand. This librarian is amazing at spotting weird patterns, but there's a catch: the librarian is so heavy and slow that they can't run fast enough to catch the thief in real-time.

Enter the FPGA, which is like a custom-built, super-fast race car engine that you can reconfigure on the fly. The big question this paper tackles is: Can we take our genius, heavy librarian (the Transformer) and teach them to drive this race car (the FPGA) so they can spot the bad data instantly? The authors, Ilia Sobakinskikh and Paul Alexander Bilokon, set out to see if they could shrink this giant brain down to fit inside a small, reconfigurable chip without losing its smarts. They wanted to prove that you don't need a massive supercomputer to find these financial glitches; you can do it on a tiny board, fast enough to stop a disaster before it happens.

The Race to Catch the Glitch

The authors started by looking at the problem of "anomaly detection." In simple terms, this is just finding the weird numbers in a list. They focused on "point anomalies," which are like sudden spikes in a stock price—think of a stock jumping from $100 to $101 in a split second when it shouldn't. To catch these, they used a specific type of AI called a Transformer. Unlike older AI models that read data one step at a time (like reading a book word by word), Transformers look at the whole picture at once, understanding how a price today relates to a price from weeks ago.

However, running these Transformers is usually slow and expensive. The authors decided to try running them on an FPGA, specifically a board called the PYNQ-Z2. To make this work, they had to be very clever with how they programmed the chip. They treated the math inside the Transformer like a factory assembly line. Instead of waiting for one calculation to finish before starting the next, they used a technique called "pipelining." Imagine a car wash where the first car is still being soaped up while the second car is already under the rinse cycle; this keeps the line moving fast. They also used "unrolling," which is like having ten workers do the same task at the exact same time instead of one worker doing it ten times in a row.

The Results: Speed vs. Smarts

The team tested two versions of their AI detective. The first was the "Vanilla Transformer," the standard, heavy version. The second was a "Linear Transformer," a lighter, faster version that skips some of the heavy math. They ran these models on the FPGA and compared them to a standard computer processor (CPU).

The results were a tale of two different trade-offs. The standard Transformer, when optimized for the FPGA, became incredibly fast. It could process data in just 37.14 microseconds (that's 0.000037 seconds!). Without these special optimizations, the same task took 347.45 microseconds, which is ten times slower. The Linear Transformer was even faster, clocking in at 29.86 microseconds.

But speed came with a price tag. To get that speed, the chip had to work much harder. The optimized Transformer used 90% of the available "LUTs" (the tiny logic switches inside the chip) and 30% of the "FFs" (memory bits), whereas the slow, unoptimized version only used 10% and 2% respectively. It's like upgrading a bicycle to a motorcycle: you go much faster, but you need a much bigger engine and more fuel.

Did the Detective Still Get the Job Done?

Speed is great, but is the detective still smart? The authors tested their models on real-world data, including New York City taxi demand, server performance logs, and high-frequency stock prices. They compared their AI against a simple "Linear Regression" model, which is like a detective who only looks at the most recent number and guesses based on that.

The results showed that the Transformers were generally much better at spotting the anomalies than the simple model. On the KPI dataset (server logs), the standard Transformer achieved an accuracy of 0.98 on the training data and 0.97 on the validation data, with an F1 score (a balance of catching bad stuff and not crying wolf) of 0.71 and 0.76 respectively. The Linear Transformer was slightly less accurate but still strong. However, on the stock market data (FI2010), the models struggled a bit more, with F1 scores dropping to 0.06 and 0.09, suggesting that while they are fast, they aren't perfect at spotting every single glitch in every type of data.

Interestingly, the authors found that some features usually considered essential for Transformers, like "positional encoding" (a way to tell the AI the order of the numbers), actually made the training unstable and worse in their specific setup. They had to turn these features off to get the models to work properly on the chip.

The Bottom Line

This paper doesn't claim to have solved the mystery of anomaly detection forever. Instead, it suggests that we can successfully shrink these powerful AI models down to fit on small, fast chips. The authors showed that by using clever programming tricks like pipelining and unrolling, you can make a Transformer run 10 times faster on an FPGA. While this speed comes with a heavy cost in terms of the chip's resources, it proves that real-time, high-speed anomaly detection is possible without needing a massive data center. It's a proof of concept that the future of catching financial glitches might not be in a giant server room, but in a tiny, reconfigurable chip racing through the data at lightning speed.

Technical Summary: Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

Problem Statement
Financial time-series data, such as asset prices, are increasingly voluminous but frequently contaminated by errors and outliers. These anomalies render downstream processing tasks ineffective or harmful to decision-making. While deep learning models, particularly Transformers, have demonstrated superior performance in capturing long-range dependencies for anomaly detection, their computational complexity poses a challenge for real-time applications. There is a critical need for data-cleaning methods that balance high accuracy with low inference latency. This work addresses the challenge of deploying Transformer architectures on Field-Programmable Gate Arrays (FPGAs) to achieve real-time anomaly detection with minimized latency.

Methodology
The study proposes the implementation and optimization of two Transformer-based architectures on a PYNQ-Z2 FPGA board: a "Vanilla" Transformer Encoder and a Linear Attention Transformer.

Model Architectures:
- Vanilla Transformer: Utilizes the standard encoder architecture with multi-head self-attention and feed-forward networks. Positional encodings and layer normalization were experimentally disabled to improve training stability and performance on the specific datasets used.
- Linear Transformer: Implements linear attention mechanisms (using a feature map $\phi(x) = \text{elu}(x) + 1$ ) to reduce computational complexity from $O(N^2)$ to $O(N)$ by eliminating the softmax operation.
FPGA Optimization Strategy:
The authors utilized High-Level Synthesis (HLS) with C++ to bridge the gap between algorithmic description and hardware implementation. Key optimizations applied to the matrix multiplications and loops within the Transformer blocks include:
- Loop Pipelining: Applied to inner loops to improve throughput, allowing new iterations to begin before previous ones complete.
- Loop Unrolling: Used for operations like ReLU activation to execute iterations in parallel.
- Array Partitioning: Employed (cyclic, block, or complete) to enable parallel memory access and avoid bottlenecks, though this increases resource utilization (LUTs and FFs).
- Numerical Stability: A max-subtraction technique was applied to the softmax function to prevent overflow.
- Data Types: Single-precision float was selected over double to fit the model within the FPGA's resource constraints without sacrificing accuracy.
Experimental Setup:
- Datasets: The models were evaluated on the Numenta Anomaly Benchmark (NAB) using NYC taxi demand data, the KPI Anomaly Detection Dataset (KPI AIOps), and the FI2010 high-frequency limit order book dataset (with synthetic outliers injected).
- Baselines: Performance was compared against a simple Linear Regression model on handcrafted features.
- Training: Models were trained using the Adam optimizer with learning rate finding and gradient clipping to mitigate gradient explosion. Class imbalance was addressed by weighting positive (anomaly) samples five times more than negative samples in the binary cross-entropy loss function.

Key Contributions
The paper outlines four primary contributions:

The successful implementation of a Vanilla Transformer architecture on an FPGA board.
The implementation of a Linear Attention Transformer on the same FPGA hardware.
A detailed analysis of the trade-off between latency and resource utilization for these architectures under specific HLS optimizations.
A comparative analysis of the accuracy of the proposed architectures against a linear baseline on anomaly detection tasks across multiple datasets.

Results

Inference Performance: The optimized FPGA implementations significantly reduced latency compared to non-optimized versions.
- The Vanilla Transformer achieved a latency of 3,714 clock cycles (~~37.14 $\mu$ s) with optimizations, compared to 34,745 cycles (~~347.45 $\mu$ s) without. This required 90% LUT utilization and 30% FF utilization.
- The Linear Transformer achieved a latency of 2,986 clock cycles (~~29.86 $\mu$ s) with optimizations, compared to 68,894 cycles (~~688.94 $\mu$ s) without. This required 99% LUT utilization and 43% FF utilization.
Accuracy:
- On the KPI dataset, the Transformer models significantly outperformed the Linear Regression baseline in F1 score (0.71–0.76 vs. 0.12–0.25).
- On the NAB dataset, the Transformer achieved an F1 of 0.23–0.47, while the Linear Regression baseline struggled with an F1 of 0.16–0.33.
- On the FI2010 dataset, the Transformer models showed higher accuracy (0.80–0.84) than the baseline (0.14–0.16), though precision and recall remained low across all models, indicating the difficulty of the specific anomaly injection task.
- Generally, the Vanilla Transformer and Linear Transformer showed comparable performance, with the Vanilla model often achieving slightly higher precision on the KPI dataset.

Significance and Claims
The authors modestly claim that the primary significance of this work is demonstrating that Transformer architectures can be efficiently implemented on FPGAs to minimize latency for real-time anomaly detection, rather than proving the superiority of Transformers over all other anomaly detection methods. The work highlights that:

Transformers can achieve "good out of the box performance" on anomaly detection with minimal tuning.
FPGA reconfigurability allows for significant acceleration of Transformer computations, making real-time processing feasible.
There is a tangible trade-off between resource utilization (LUTs/FFs) and latency, where aggressive optimization (pipelining and partitioning) drastically reduces inference time at the cost of higher resource consumption.

The paper concludes that while specialized architectures exist, the vanilla and linear Transformer architectures are viable candidates for FPGA deployment in financial time-series cleaning, offering a balance of performance and speed. Future work is suggested but not detailed beyond general ideas.

The Race to Catch the Glitch

The Results: Speed vs. Smarts

Did the Detective Still Get the Job Done?

The Bottom Line

Technical Summary: Optimizing Transformer Neural Network for Real-Time Outlier Detection on FPGAs

More like this