🤖 machine learning

Dynamic Short Convolutions Improve Transformers

This paper introduces dynamic short convolutions, an input-dependent neural primitive that enhances Transformer performance and scaling efficiency by outperforming standard and static convolutional variants across various architectures and tasks while maintaining hardware efficiency through custom Triton kernels.

Original authors: Oliver Sieberling, Bharat Runwal, Rameswar Panda, Yoon Kim

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oliver Sieberling, Bharat Runwal, Rameswar Panda, Yoon Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a long story, like a novel or a movie script. To do this, you need to remember what happened earlier and how it connects to what's happening right now.

For a few years, the best way computers (specifically AI models called "Transformers") have done this is by using a mechanism called Attention. Think of Attention like a super-powered spotlight. When the computer reads a word, the spotlight scans the entire story to find every other word that might be relevant, no matter how far away it is. This is incredibly flexible, but it can be slow and energy-intensive because the computer has to look at everything at once.

A few years ago, some researchers tried adding Convolutions to the mix. Think of a convolution as a small, local "magnifying glass" that only looks at the 3 or 4 words immediately surrounding the current word. It's fast and efficient, but it's "static." It's like a magnifying glass with a fixed lens; it zooms in the same way on every word, regardless of whether the word is a noun, a verb, or a confusing phrase.

The New Idea: The "Smart, Shape-Shifting" Magnifying Glass

This paper introduces a new tool called Dynamic Short Convolutions.

If a static convolution is a fixed magnifying glass, a dynamic convolution is a shape-shifting, smart magnifying glass.

Here is how it works in simple terms:

It looks locally: Like the old magnifying glass, it only looks at the few words right next to the current one. This keeps it fast and efficient.
It changes its mind: Unlike the old version, this new tool looks at the current word and asks, "What kind of lens do I need right now?"
- If the word is "can" (as in a metal container), the tool might decide to look back at the previous word "old" to understand "old can."
- If the word is "can" (as in "to be able to"), the tool might decide to look at the next word "swim" to understand "can swim."

The computer generates a unique filter (a lens) for every single word based on what that word is. This allows the AI to understand local context much better than before, without losing the speed benefits of looking at a small window.

What the Researchers Found

The authors tested this new "shape-shifting" tool on various AI models, ranging from small ones (150 million parameters) to large ones (2 billion parameters). Here is what they discovered:

It's Smarter: In tests designed to see if the AI could remember specific details (like a "needle in a haystack" task), the models using this new tool were significantly better at finding the right information than models using the old static tools.
It's More Efficient: They found that to get the same level of intelligence, a model with this new tool needed less computing power. They calculated that it offers a 1.33x to 1.60x advantage.
- Analogy: Imagine two cars trying to drive the same distance. The car with the new tool gets there using 30% less fuel than the standard car, or it gets there faster with the same amount of fuel.
It Works Everywhere: They didn't just test it on standard AI models. They also tried it on newer types of AI (like "Mamba" and "DeltaNet") and found it improved those models too.
It's Fast Enough: Usually, when you make a tool "smarter," it becomes slower. The authors built special software (called "Triton kernels") to make sure this new tool runs efficiently on modern computer chips. They found the speed penalty was very small (only about 8% slower for the most common version), which is a small price to pay for the big boost in intelligence.

The Bottom Line

The paper argues that Dynamic Short Convolutions are a new, essential building block for the next generation of AI. They combine the best of two worlds: the speed of looking at just a few words at a time, and the flexibility of changing how you look at those words based on the context.

The researchers conclude that this is a practical, scalable way to make AI models smarter and more efficient without needing to invent entirely new architectures from scratch. It's like upgrading a standard car engine with a smart turbo that adjusts itself to the road conditions, giving you more power without needing a bigger engine.

Technical Summary: Dynamic Short Convolutions Improve Transformers

Problem Statement

While Transformers have become the dominant architecture for large language models (LLMs) due to the scalability of attention mechanisms, they lack an inherent inductive bias toward local sequence structure. Although static short convolutions (depthwise separable convolutions with small kernel widths) have been explored to introduce locality, they apply fixed filters across all positions, limiting their ability to model context-dependent local compositions. The paper posits that a more expressive primitive is needed—one that retains the efficiency and locality bias of convolutions but allows the filtering mechanism to adapt dynamically to the input, thereby improving performance on tasks requiring dynamic local aggregation without sacrificing scalability.

Methodology

Dynamic Short Convolutions

The authors introduce dynamic short convolutions as a neural network primitive. Unlike static convolutions where weights $w$ are fixed, dynamic convolutions generate input-dependent filters $w(t)$ at each time step $t$ .

Mechanism: For a sequence of activations $x_t$ , the output is computed as $y_t := \sum_{k=0}^{W-1} w(t)_k \odot x_{t-k}$ .
Parameterization: The filter $w(t)$ $w (t)$ is generated by a learned linear transformation of the current hidden state. To maintain parameter efficiency, the authors propose two specific parameterizations:
1. Low-rank factorization: The projection is factored through a low-rank transformation of rank $R$ .
2. Head-wise tying: Dimensions are split into heads, where a single weight filter is shared across $H$ consecutive channels.
Placement: In the primary experiments, dynamic convolutions are applied to the Query (Q), Key (K), and Value (V) projections before Rotary Position Embeddings (RoPE), with a residual connection ( $X = X + \text{dynamicShortConv}(X)$ ). The authors also experiment with applying them after every linear layer in the Transformer.

Efficient Training Implementation

Dynamic convolutions are memory-bound due to low arithmetic intensity. Naïve implementations suffer from excessive High Bandwidth Memory (HBM) traffic. To address this, the authors developed custom Triton kernels:

On-chip Computation: The kernels read activations and dynamic weights, perform the full convolution on-chip, and write only the final result to HBM.
Fused Low-Rank Kernel: For the low-rank variant, the kernel fuses the second projection of the factorization directly into the convolution. It reads the low-rank inputs ( $z$ ) and the projection matrix ( $U$ ), generates weights ($zU$) on-chip, and applies the convolution immediately, avoiding the materialization of the full dynamic weight tensor in HBM.
Performance: These kernels achieve latency competitive with or faster than optimized static convolution kernels (e.g., causal_conv1d), with end-to-end training throughput slowdowns of roughly 8% for the QKV variant and 22% for the all-linear variant at the 2B parameter scale.

Key Contributions

Novel Primitive: The proposal of dynamic short convolutions as an input-dependent, locality-biased primitive that generalizes static short convolutions.
Scaling Law Analysis: Empirical evidence demonstrating that dynamic convolutions provide a compute advantage over standard Transformers.
- 1.33× advantage when applied to Q, K, and V vectors.
- 1.60× advantage when applied after every linear layer.
Hardware Efficiency: The development of custom Triton kernels that make dynamic convolutions practical for training on modern GPUs (H100) with manageable overhead.
Broad Applicability: Validation of the approach across diverse architectures, including dense Transformers, Mixture-of-Experts (MoE) models, and linear RNNs (Mamba-2 and Gated DeltaNet).

Experimental Results

Synthetic Benchmarks

Variable-Key MQAR: On a modified Multi-Query Associative Recall task where keys have variable lengths and share tokens, dynamic convolutions significantly outperformed both vanilla Transformers and those with static convolutions. This highlights the model's ability to resolve variable-length local structures dynamically.
MAD Benchmark: Dynamic convolutions showed strong performance on the Mechanistic Architecture Design benchmark, particularly on "Fuzzy Recall," where keys and values consist of variable token counts.

Language Modeling

Experiments were conducted on the Nemotron-CC corpus with model sizes ranging from 150M to 2B parameters, and a 7B-parameter MoE model.

Perplexity and Accuracy: Dynamic convolutions consistently reduced perplexity and improved zero-shot accuracy on common-sense reasoning tasks (lm-eval-harness) compared to:
- Standard Transformers.
- Transformers with static short convolutions.
- Transformers with increased parameter counts (wider FFNs) to match the parameter budget of dynamic variants.
Linear RNNs: Replacing static convolutions with dynamic ones in Mamba-2 and Gated DeltaNet significantly improved perplexity, suggesting that dynamic convolutions offer a more effective sequence mixing mechanism than redesigning the sequence mixer itself.
RULER Benchmark: Dynamic convolutions showed particular gains in multi-key, multi-query, and multi-value subtasks, aligning with their capability for input-dependent local aggregation.

Ablation Studies

Kernel Width: A width of $W=3$ or $4$ was found to be the optimal sweet spot.
Placement: Applying dynamic convolutions to all three projections (Q, K, V) yielded the best results, with the Value projection providing the most significant individual gain.
QK-Norm: The benefits of dynamic convolutions persisted even when combined with QK-norm, whereas static convolutions offered little benefit in that setting.

Significance and Claims

The paper claims that dynamic short convolutions represent a scalable, hardware-efficient, and expressive primitive for advancing Transformer-based language models. The authors argue that:

They successfully bridge the gap between the locality bias of convolutions and the expressivity of attention-like mechanisms.
The observed compute advantages (1.33× to 1.60×) suggest that dynamic convolutions can improve the performance-efficiency frontier of LLMs.
The custom Triton kernels demonstrate that these gains are achievable with a manageable end-to-end training slowdown (8–25%), making the approach practical for real-world deployment.

The authors remain modest regarding the scope of their findings, noting that while trends are consistent up to 2B parameters and a 7B MoE, further research is needed to confirm these advantages at frontier scales, with longer training horizons, and across different data mixtures. They also acknowledge that further engineering is required to fully optimize inference and support a broader range of hardware.