🤖 machine learning

Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

This paper introduces HyenaND, a subquadratic, input-dependent operator that applies global convolutions directly to native multi-dimensional data structures to overcome the limitations of standard attention and recurrent models, achieving competitive accuracy and significant speedups through a specialized CUDA implementation.

Original authors: David R. Wessels, Farhad Ramezanghorbani, David W. Romero, Alireza Moradzadeh, Olivia Viessmann, Maksim Zhdanov, John St. John, Ken Janik, David M Knigge, Yucheng Tang, Erik J Bekkers, Saee Gopal Pali

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David R. Wessels, Farhad Ramezanghorbani, David W. Romero, Alireza Moradzadeh, Olivia Viessmann, Maksim Zhdanov, John St. John, Ken Janik, David M Knigge, Yucheng Tang, Erik J Bekkers, Saee Gopal Paliwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot to understand the world. Right now, the best robots use a tool called "attention" to look at everything at once, like a student scanning a whole page of text to find the most important words. This works great for short stories, but if you give the robot a whole library, a 3D movie, or a complex weather map, the "attention" tool gets overwhelmed. It has to compare every single piece of data with every other piece, which takes so much time and energy that the robot crashes or has to squint at the data through a tiny keyhole (a process called "patchification") just to survive.

To fix this, scientists have been trying to build faster tools. One popular idea is to use "recurrent" models, which read data like a line of text, one word after another. But this is like trying to understand a 3D sculpture by only looking at it from one angle and then guessing the rest; it destroys the natural shape of the object. Another idea is to use "convolutions," which are like sliding a magnifying glass over an image to spot patterns. These are fast and keep the 3D shape intact, but they usually act like a rigid stamp, applying the exact same pattern to every part of the image without paying attention to what's actually there. The big question in this corner of computer science is: Can we build a tool that is fast enough to handle massive 3D data, keeps the natural shape of that data, and is smart enough to change its "magnifying glass" based on what it sees?

This paper introduces a new tool called HyenaND that says "yes." Think of HyenaND as a magical, shape-shifting magnifying glass that can slide over a 3D object (like a medical scan or a weather simulation) without ever flattening it into a 1D line. Unlike the rigid stamps of the past, this tool can look at the whole object first, decide what kind of pattern it needs to find, and then instantly adjust its lens to match. It does this using a clever math trick involving "Fourier transforms" (a way of turning images into sound waves to process them faster) that keeps the speed super high, growing only slightly slower as the data gets bigger, rather than exploding in cost.

The authors found that HyenaND is a game-changer for handling complex, multi-dimensional data. In their tests, they showed that HyenaND can handle tasks that would make standard "attention" tools crash due to memory limits, such as analyzing 3D medical images or simulating fluid dynamics. When they tested it on a "copying" game where the robot had to remember a specific color from a 3D grid, HyenaND was thousands of times more accurate than the old methods. They also built a special software engine called nSubQ to make sure this math trick runs lightning-fast on real computer chips, turning theoretical speed into actual time savings.

The paper suggests that while HyenaND is incredibly powerful on its own, it works even better when paired with the old "attention" tools in a hybrid team. In experiments with DNA sequences, computer vision (ImageNet), and medical imaging, these hybrid teams beat both pure attention models and other fast models that try to read data in a line. The authors argue that we don't need to choose between speed and intelligence anymore; we can have a tool that respects the 3D geometry of our data, scales to massive sizes, and still pays close attention to the details. They conclude that this approach removes a major bottleneck, allowing future AI to finally "see" the world in its native, high-resolution 3D glory without having to squint through a tiny keyhole.

Technical Summary: HyenaND and nSubQ

Problem Statement
Deep learning models increasingly process large, multi-dimensional (ND) data, such as high-resolution images, 3D medical volumes, and physical fields. The standard Transformer architecture, while natively multi-dimensional, suffers from quadratic computational complexity ( $O(L^2)$ ) relative to sequence length, hindering its application to high-resolution or long-context tasks. The common workaround, patchification, trades performance for tractability; however, shrinking patch sizes to recover fine-grained structure increases token counts, re-imposing the quadratic bottleneck.

Existing subquadratic alternatives face significant limitations when applied to ND data:

Recurrent Models (e.g., Mamba): These are inherently 1D and causal. To process ND data, they require "rasterizing" the input into ad-hoc 1D scan orders, which destroys the spatial structure and geometric inductive biases of the data. Furthermore, their input dependency is typically token-level, which is suboptimal for signals that are approximately Linear Time-Invariant (LTI) and available in full (non-autoregressive settings).
Standard Convolutions: While they preserve ND structure and offer fast FFT-based paths, traditional long convolutions lack input dependency, limiting their expressiveness compared to attention mechanisms.

Methodology: HyenaND
The authors introduce HyenaND, a subquadratic, global, input-dependent operator designed to act directly on the native geometry of multi-dimensional data without rasterization.

Architectural Reframing: The paper disentangles the "State-Space Model" (SSM) landscape into two distinct branches:
- Generalized Recurrent: Suitable for 1D non-LTI data (e.g., text).
- Generalized Convolutional: Suitable for approximately LTI, multi-dimensional data. HyenaND belongs to this branch.
Native ND Convolution: HyenaND performs convolutions directly on $N$ -dimensional grids ( $L_1 \times \dots \times L_N$ ). It utilizes a global, implicitly parametrized kernel $K$ evaluated via $N$ -dimensional Fast Fourier Transforms (FFTs), achieving $O(L \log L)$ complexity.
Input-Dependent Kernels via Registers and FiLM: To overcome the lack of input dependency in standard convolutions, HyenaND conditions the kernel on the entire input sample rather than individual tokens.
- Registers: $R$ learnable register tokens are prepended to the input grid. These are processed jointly to integrate global information.
- FiLM Conditioning: A control variable $z(x)$ is derived from the register activations via a softmax-weighted average. This variable modulates a coordinate-based SIREN (Sinusoidal Representation Networks) MLP $f_\theta$ using Feature-wise Linear Modulation (FiLM).
- Kernel Synthesis: The final kernel is $K(x) = w(c) \odot f_\theta(c; z(x))$ , where $w$ is a learnable Gaussian window. This allows the kernel to adapt to the specific input instance while maintaining the single global FFT path.
Kernel Parameterization: The kernel is defined on a continuous coordinate grid using a SIREN, allowing for resolution independence. The authors introduce a "block-diagonal multi-frequency initialization" for the SIREN's frequency scaling factor $\omega_0$ to ensure the kernel can resolve both low and high-frequency content across different channels.
Handling Boundaries: The implementation supports both circular convolutions (for periodic physical systems) and non-circular convolutions (for natural images/volumes) by zero-padding inputs to double the grid size before FFT, avoiding artificial boundary leakage.

Implementation: nSubQ
To bridge the gap between asymptotic complexity and wall-clock performance, the authors developed nSubQ, an IO-aware CUDA library.

Fused Kernels: nSubQ fuses the forward FFT, spectral modulation (applying the input-dependent kernel), inverse FFT, and output segmentation into a single CUDA kernel.
Memory Efficiency: By keeping intermediate representations in on-chip shared memory, it minimizes High Bandwidth Memory (HBM) traffic.
Performance: This fusion turns the theoretical $O(L \log L)$ advantage into practical speedups, particularly for long sequences where attention mechanisms suffer from memory bandwidth bottlenecks.

Key Contributions

Conceptual Reframing: A clear separation of subquadratic operators into recurrent (1D) and generalized convolutional (ND) families, arguing that the latter is the natural primitive for multi-dimensional, LTI data.
HyenaND Operator: A native ND, sample-level input-dependent, subquadratic operator that preserves LTI inductive biases and avoids rasterization.
nSubQ Library: A high-performance CUDA implementation that achieves wall-clock speedups for ND FFT convolutions.
Empirical Validation: Comprehensive evaluation across 1D (genomics), 2D (ImageNet), 3D (medical imaging), and PDE modeling.

Results

Controlled Spatial-Recall: HyenaND significantly outperforms attention and bidirectional Mamba on 3D "simple copy" tasks (lower MSE by 3–4 orders of magnitude) due to its translation-equivariant prior. It maintains native 3D resolution, whereas attention fails (OOM) or requires aggressive patchification.
Long-Context Genomics (1D): On OpenGenome2, hybrid HyenaND-attention models (specifically H2 with 2 MHA layers) achieved the lowest perplexity, outperforming full Transformers and pure Hyena configurations. HyenaND demonstrated stable throughput scaling up to 10M token contexts.
Computer Vision (ImageNet-1K): Pure HyenaND (ViT-5-Small backbone) achieved 81.5% top-1 accuracy, matching the attention baseline. Hybrid configurations (e.g., $(H A) \times 6$ ) outperformed both pure attention and recurrence-based Mamba hybrids, achieving 82.1% accuracy with significantly fewer FLOPs (up to 82% savings at small patch sizes).
PDE Modeling (The Well): HyenaND achieved the lowest Variance-Normalized RMSE (VRMSE) on all five datasets (fluid dynamics, MHD, etc.), outperforming both full-resolution ConvNeXt U-Nets and attention baselines. Performance gains over attention increased with sequence length.
3D Medical Segmentation (PanTS): Hybrid HyenaND-attention models achieved the highest Mean Dice scores (0.7559), surpassing pure attention and pure Hyena variants. HyenaND variants also reduced peak GPU memory allocation by ~10.8% compared to the attention baseline.

Significance and Claims
The paper claims that HyenaND and nSubQ address the fundamental constraint forcing foundation models to rely on patchification, which erodes fine-grained structural information. By providing a native ND, input-dependent, subquadratic operator, the authors enable models to ingest data at its intrinsic geometry and native resolution.

The authors position HyenaND not as a universal replacement for attention, but as a complementary specialist. Pure HyenaND stacks match strong attention baselines, while hybrid configurations (interleaving HyenaND and attention) consistently outperform both pure attention and recurrence-based hybrids. This suggests that the convolutional branch (for global, LTI structure) and the attention branch (for fine-grained selectivity) are best used in tandem for multi-modal foundation models. The work releases both the HyenaND architecture and the nSubQ library to facilitate this shift in architectural design.

Technical Summary: HyenaND and nSubQ

More like this