The Inference-Compute Frontier and a Latency-Efficient Architecture for Limit Order Book Prediction

Technical Summary: The Inference-Compute Frontier and a Latency-Efficient Architecture for Limit Order Book Prediction

Problem Statement
Limit order book (LOB) prediction requires models that are not only accurate but also fast enough to act on short-horizon signals before market states change. While scaling laws in other domains suggest a predictable relationship between model scale (compute) and performance, it is unclear if such a frontier exists in LOB prediction given the heterogeneity of predictors, the structured nature of market inputs, and the critical importance of inference latency over parameter counts. The paper investigates whether a scaling-law-style "inference-compute frontier" exists for LOB prediction and whether structural compute is a sufficient proxy for the practical constraint of latency.

Methodology
The study utilizes the FI-2010 benchmark across two experimental "lanes":

1. Scaling Lane: Uses raw LOB40 features and test cross-entropy to analyze the relationship between structural forward work and predictive loss.
2. Deployment Lane: Uses the full 144-feature set, macro-F1 scores, and measured batch-one latency to evaluate practical deployment efficiency.

Key Components:

• Structural Forward Work: A reproducible, architecture-level count of operations (e.g., split comparisons for trees, dot products for convolutions, matrix multiplications for MLPs) derived from hyperparameters. This serves as a capacity coordinate distinct from hardware-specific latency.
• Model Pools:
  • Fit Pool: A heterogeneous set of 56 non-MLPLOB configurations (decision trees, histogram gradient boosting, CatBoost, random-convolution logistic models) used to establish the empirical frontier.
  • Target Pool: MLPLOB architectures (W64, W128, W256) excluded from the initial fit to test extrapolation.
• Frontier Fitting: The authors fit a power law ($L(C) = L_\infty + AC^{-\alpha}$) to the lower envelope of test cross-entropy versus structural work.
• Latency Analysis: Measured single-observation latency on CPU is compared against structural work to determine if latency is merely noisy compute or a distinct design coordinate.
• FastBiNLOB Architecture: A new dense, axis-separable LOB mixer designed to minimize latency. It replaces attention mechanisms (Q/K/V projections, softmax) with dense temporal and feature MLPs, utilizing BiN-style normalization and residual axis-separable blocks.

Key Results

1. Existence of an Inference-Compute Frontier: A power-law relationship well summarizes the realized empirical frontier of predictive loss versus structural forward work.
   • When MLPLOB architectures are held out, the power-law fit to the low- and mid-compute non-MLPLOB frontier extrapolates across multiple orders of magnitude to the high-compute MLPLOB targets, achieving an $R^2 = 0.941$ on the excluded target frontier.
   • The W64 and W128 MLPLOB variants lie on or near this frontier ($R^2 = 0.948$ for the subset), while the W256 variant is dominated (using more work without improving loss), validating the frontier's ability to distinguish efficient from inefficient scaling.
2. Latency is Distinct from Compute: Replacing structural work with measured latency yields a significantly weaker fit ($R^2 = 0.468$) and reorders model families. For instance, CatBoost models have high structural work but low latency, while MLPLOB models have massive structural work but competitive latency due to hardware efficiency. This confirms that latency is not merely "noisy compute" but a separate design target dependent on implementation and hardware.
3. FastBiNLOB Performance:
   • Latency: The H96 variant achieves the published $y_{10}$ macro-F1 target at ~23% lower latency than the MLPLOB anchor. The H120 taper variant achieves the $y_{100}$ target at ~60% lower latency than the TLOB anchor.
   • Accuracy: Under the published comparison settings, the H120 taper variant posts state-of-the-art (SOTA) macro-F1 scores on the $y_{10}$ and $y_{100}$ horizons, though it does not outperform baselines on $y_{20}$ and $y_{50}$.

Significance and Claims
The paper makes two primary contributions:

1. Empirical Frontier Validation: It provides evidence that structural forward work is a useful, stable coordinate for defining an inference-compute frontier across heterogeneous LOB predictors on FI-2010. The successful extrapolation to excluded MLPLOB families suggests a regularity in how compute translates to loss in this domain.
2. Latency-Efficient Design: It demonstrates that maximizing compute does not guarantee low latency. By treating latency as a separate objective, the authors introduce FastBiNLOB, an architecture that preserves useful computation (dense mixing) while removing latency-heavy operations (attention).

The authors conclude that model capacity and serving latency should be optimized as separate objects. Scaling laws can identify if useful computation exists, but they do not predict how cheaply that computation can be served. The practical implication is that efficient LOB modeling requires designing architectures where computation resides in dense, hardware-friendly operators rather than simply increasing parameter counts or using attention-heavy mechanisms.

Limitations and Scope
The authors explicitly state that these claims are empirical and finite, not theorems regarding Bayes risk or optimality across all markets. The "structural forward work" is a convention, not a hardware instruction count, though the results remain robust under 2x perturbations of this count. Furthermore, the scaling and deployment lanes use different feature sets and should not be conflated. The FastBiNLOB results are presented as a "recipe-plus-architecture" outcome, acknowledging that latency performance is dependent on specific hardware and implementation details.