🤖 machine learning

Fast Exact Nearest-Neighbor Learning for High-Frequency Financial Time Series

This paper demonstrates that a Mojo-based implementation of an exact SIMD k-d tree significantly outperforms existing scikit-learn methods in speed and scalability for high-frequency financial time series, enabling real-time nearest-neighbor learning and improved derivative pricing models without sacrificing accuracy.

Original authors: Henry Han, Diane Li

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Henry Han, Diane Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Needle in a Haystack" Dilemma

Imagine you are a financial trader. Every second, you need to make a decision based on the current market. To do this, you look at your "memory bank"—a massive history of how the market behaved in the past. You want to find the exact moments in history that look most like today to predict what happens next.

The problem is that this "memory bank" is growing huge (millions of data points).

The Old Way (Python/Scikit-learn): Imagine trying to find a specific book in a library by walking down every single aisle, checking every single book, one by one. It's accurate, but it's incredibly slow. As the library gets bigger, you get slower.
The "Fast" Way (C++): Imagine hiring a team of super-fast runners to do the same search. They are fast, but they speak a different language than your researchers. You have to translate your ideas into their language, which is slow, expensive, and prone to errors.

The Solution: Mojo

The authors introduce Mojo, a new programming language that is like a "super-charged Python." It speaks the same language as the researchers (easy to write) but runs with the speed of the super-fast runners.

They used Mojo to build a smarter way to search this financial history. Instead of checking every single book (data point), they built a smart filing system (a "k-d tree") that helps them skip huge sections of the library that definitely don't contain the answer.

How They Made It Fast (The Three Tricks)

The paper explains that they didn't just use a smart filing system; they optimized it in three specific ways to make it fly:

The "Smart Split" (Variance-Based Splitting):
- Analogy: Imagine sorting a messy pile of clothes. Instead of just splitting them by "shirts vs. pants," you look at the pile and ask, "What feature separates these items the most?" Maybe you split them by "color" first because that creates the cleanest groups.
- In the paper: The algorithm looks at the financial data and finds the specific feature (like volatility or price momentum) that varies the most. It splits the data there first, creating tighter, easier-to-search groups.
The "Flat Floor" (Contiguous Flat-Buffer Storage):
- Analogy: Imagine your books are stored in a library where some are in a box, some are on a shelf, and some are in a basement, and you have to run back and forth to get them. That's slow. Now, imagine all the books are lined up perfectly in a single, long row on one shelf. You can grab them in one smooth motion.
- In the paper: They stored the data in one continuous block of memory. This allows the computer's "prefetcher" (a part of the brain that guesses what you need next) to grab data efficiently without wasting time jumping around.
The "Super-Reader" (SIMD Vectorization):
- Analogy: Imagine you are reading a list of numbers. A normal person reads one number at a time. A "Super-Reader" (SIMD) can read eight numbers at once and do the math on all of them in a single blink.
- In the paper: They programmed the computer to compare eight financial data points simultaneously. This makes the actual math of comparing "today" to "yesterday" incredibly fast.

The Results: Speed vs. Accuracy

The team tested this on real financial data (stocks, ETFs, and currency) on two types of computer chips (Intel x86 and Apple M3).

The Speed:
- On standard computers (x86), their new method was 17 to 21 times faster than the standard Python tool (scikit-learn).
- On Apple computers (ARM64), it was 28 to 43 times faster than the standard tool.
- Crucial Point: They didn't just guess the answer. They found the exact same answer as the slow method, just much faster.
The "Why" (The ARM64 Surprise):
- On Apple chips, the standard "brute force" method (checking everything) was surprisingly slow because the chip's "Super-Reader" (SIMD) was narrower than the code expected. However, because the authors' "Smart Filing System" (k-d tree) skipped so many unnecessary checks, it didn't matter. It was still the fastest method by a huge margin.

The Real-World Win: Better Predictions

The paper didn't just stop at speed. They showed that being faster allows you to do more work.

They trained a model to predict "Implied Volatility" (a measure of risk for stock options).
Because their system was so fast, they could train the model on 10 times more data than the standard Python system could handle in the same amount of time.
The Result: By using more data, the model became 8% more accurate. This proves that speed isn't just about waiting less; it's about learning better.

Summary

The paper argues that to handle the massive amounts of data in modern finance, we can't just use slow, easy tools (Python) or hard, fast tools (C++). We need a middle ground.

Mojo provides that middle ground. By combining a smart search algorithm, a neat way of storing data, and a "super-reading" math engine, they created a system that is:

Exact: It doesn't guess; it finds the real answer.
Fast: It is 17x to 43x faster than current standard tools.
Scalable: It gets even more powerful as the amount of data grows, allowing financial models to learn from much larger histories and make better predictions.

Technical Summary: Fast Exact Nearest-Neighbor Learning for High-Frequency Financial Time Series

Problem Statement
Financial AI is entering a scaling era where models must ingest massive historical corpora (equities, ETFs, FX, options) to support real-time trading, risk management, and derivative pricing. A critical bottleneck exists in the current stack: while Python-based tools (NumPy, scikit-learn) offer flexibility, they suffer from interpreter overhead, memory-bandwidth saturation, and poor cache locality when querying hundreds of thousands of historical states. Conversely, C++ and FPGA solutions offer speed but create a "research-to-production" gap due to the need for separate codebases and manual optimization. The specific challenge addressed is exact nearest-neighbor (KNN) inference for high-frequency time series, where the linear cost of brute-force search ( $O(n \cdot d)$ ) becomes prohibitive for real-time latency budgets, yet approximate methods (like HNSW) sacrifice the exactness required for certain financial applications.

Methodology
The authors propose a Mojo SIMD k-d tree designed specifically for financial time-series corpora. Mojo, a compiled systems language with Python-compatible syntax, enables the co-design of algorithm, memory layout, and hardware without leaving the Python ecosystem. The methodology integrates three core optimizations:

Variance-Based Splitting: Unlike standard median-based splits, the tree selects the split axis based on the dimension with the maximum variance ( $j^* = \arg\max \text{Var}(\{x_{i,j}\})$ ). This concentrates early pruning on the most discriminative financial features (returns, volatility, momentum), creating tighter subregions.
Contiguous Flat-Buffer Storage: Training vectors are stored in a row-major flat buffer ( $X \in \mathbb{R}^{n \times d}$ ) indexed by a permuted array. Leaf nodes are defined by integer offsets, allowing candidates to be read as contiguous memory blocks. This eliminates pointer-chasing overhead and ensures cache-line alignment.
Compile-Time Vectorized Kernels: Distance computations utilize SIMD (Single Instruction, Multiple Data) vectorization. The kernel is specialized at compile-time (comptime) for specific dimensions (e.g., $d=16$ or $d=24$ ) and SIMD widths, generating fully unrolled, scalar-free code. The squared Euclidean distance is decomposed into parallel register-width blocks.

The system was evaluated on eight financial datasets (U.S. equities, ETFs, FX) across two hardware architectures: x86 (Intel/AMD) and ARM64 (Apple M3). Comparisons were made against scikit-learn's brute-force and k-d tree implementations, as well as Mojo's own SIMD brute-force approach.

Key Contributions

Mojo SIMD k-d Tree Implementation: A novel implementation combining variance-based pruning, flat-buffer memory layout, and compile-time vectorized distance kernels for exact KNN on large financial datasets.
Theoretical Runtime Analysis: A proof demonstrating that under standard pruning assumptions, the Mojo k-d tree asymptotically dominates both Mojo SIMD brute force and scikit-learn's k-d tree in the fixed-stock, large- $n$ , moderate-dimensional regime.
Finance-Hardware Co-Design: A standardized feature engineering approach where OHLCV-derived features are aligned with SIMD widths (16 and 24 float32 values mapping to 2 and 3 register widths, respectively), preserving financial interpretability while maximizing throughput.
Empirical Validation: Comprehensive benchmarks showing significant speedups across diverse asset classes and hardware platforms, with statistical significance confirmed via Wilcoxon signed-rank tests.
Scalability Demonstration: A secondary experiment using Extra Trees for implied-volatility pricing, showing that the throughput gains allow training on 10 $\times$ more data, directly improving prediction accuracy.

Results

Speedup on x86 ( $d=16$ ): The Mojo k-d tree achieved a 17.5–21.6 $\times$ speedup over scikit-learn's k-d tree and a 1.2–1.3 $\times$ speedup over scikit-learn's brute force across datasets ranging from 143K to 277K samples.
Speedup on ARM64 ( $d=24$ ): The method achieved 28.1–43.5 $\times$ speedup over scikit-learn's brute force on Apple M3. Notably, on ARM64, the Mojo SIMD brute-force kernel was 780–1549 $\times$ slower than the k-d tree due to a mismatch between the compile-time SIMD width (optimized for x86 256-bit registers) and the native 128-bit ARM registers. However, the k-d tree's algorithmic pruning ( $O(n^{1-1/d})$ ) rendered it immune to this hardware-specific penalty, maintaining massive gains.
Scaling Exponents: Empirical fitting of the runtime complexity $T(n) = Cn^\alpha$ yielded an exponent of 1.755 for the Mojo k-d tree (query only), compared to 1.885 for brute force and 2.320 for scikit-learn's k-d tree. This confirms sub-quadratic scaling that widens as dataset size increases.
Prediction Quality: In the Extra Trees experiment, training on 200,000 samples (10 $\times$ the Python baseline limit) reduced the Put Implied Volatility (IV) RMSE by 8.0% (from 0.361 to 0.333). Separating calls and puts further reduced RMSE by 27–31%.
Statistical Significance: A Wilcoxon signed-rank test confirmed the Mojo k-d tree is strictly faster than scikit-learn's k-d tree on all eight datasets ( $p < 0.01$ ), with a geometric mean speedup of 10.8 $\times$ (95% CI: [5.94 $\times$ , 16.62 $\times$ ]).

Significance and Claims
The paper claims to be the first open-source, Python-compatible, system-level study of financial AI inference efficiency at scale. Its primary significance lies in demonstrating that co-design across algorithm, memory layout, hardware, and language can yield operationally meaningful acceleration without sacrificing exactness or Python compatibility.

The authors argue that Mojo bridges the gap between the flexibility of Python research and the performance of C++ production systems. By enabling a single codebase that achieves near-C++ throughput (approx. 60% of optimized C++ speed in a specific comparison) while offering 10 $\times$ speedup over standard Python libraries, the work positions Mojo as a scalable, production-ready stack for financial AI. The results suggest that as financial data volumes grow toward tick-level scales, the sub-quadratic scaling of this approach will become increasingly critical for maintaining real-time latency constraints. The principles are also claimed to be applicable to other data-intensive fields requiring exact nearest-neighbor retrieval, such as genomics and cybersecurity.