💻 computer science

Rethinking Attention Locality in Spiking Transformers

This paper introduces Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP), a novel method that addresses the lack of spatial locality in Spiking Transformers by combining non-overlapping local attention with a lightweight cross-boundary pathway and a hierarchical deployment strategy, achieving significant performance gains across various visual tasks with minimal overhead.

Original authors: Zeqi Zheng, Zizheng Zhu, Yuping Yan, Wenxuan Pan, Zhaofei Yu, Yaochu Jin

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeqi Zheng, Zizheng Zhu, Yuping Yan, Wenxuan Pan, Zhaofei Yu, Yaochu Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just crunch numbers like endless calculators, but think more like our own brains. This is the realm of Spiking Neural Networks (SNNs), a type of artificial intelligence that mimics the way biological neurons fire tiny electrical sparks, or "spikes," only when necessary. Instead of constantly chugging away, these networks are event-driven, meaning they stay quiet until something interesting happens, making them incredibly energy-efficient. Recently, scientists have tried to combine this efficient "spiking" style with the powerful "Transformer" architecture—the same brainy structure that helps computers understand language and recognize images. The goal? To create super-efficient AI that can see and understand the world without burning through massive amounts of electricity. However, there's a catch: when these spiking Transformers try to focus on different parts of an image, they sometimes struggle to pay attention to things that are right next to each other, like a neighbor ignoring the house right beside them.

This paper, titled "Rethinking Attention Locality in Spiking Transformers," dives into that specific problem. The authors, a team of researchers from Zhejiang University, Westlake University, and Peking University, noticed that while previous attempts to fix this "neighborly" issue made the AI smarter, they didn't actually fix the root cause. They found that some methods were just doing the math locally without actually looking at nearby pixels, and others were trying to force the same fix onto every part of the network, which didn't always work. To solve this, they invented a new system called SCLA-BCP. Think of it as giving the AI a better way to organize its neighborhood watch: it divides the image into small, tidy blocks where neighbors can chat easily, but also adds a special "boundary bridge" that lets information flow between those blocks so nothing gets left out. Their experiments show that this new approach helps the AI see better, recognize objects more accurately, and do it all while using very little extra energy.

The Problem: When "Local" Doesn't Mean "Nearby"

To understand what the authors discovered, we first need to look at how these Spiking Transformers usually work. In a standard Transformer, the AI looks at an image and tries to figure out how different parts relate to each other. It asks, "Does this pixel care about that pixel?" In a normal computer brain, this is done with a special math trick called "Softmax," which helps the AI decide which pixels are most important. But in Spiking Transformers, they remove Softmax to keep things "spiky" and efficient. The downside? The AI ends up treating all pixels almost equally, like a student who doesn't know who to sit with and ends up staring blankly at everyone in the room.

To fix this, previous researchers tried to force the AI to focus on local areas. They did this in two main ways:

The "Grouping" Method (LSSA): They tried to group pixels together based on a pattern, like putting every second pixel in a group. The problem, as the authors found, is that this is like grouping people in a room by their shoe size. Even if you are in the same group, you might be standing on opposite sides of the room. The math says you are "local," but physically, you are far apart. The authors call this a "computational-spatial locality discrepancy." The computer thinks it's looking at neighbors, but it's actually looking at strangers.
The "Uniform" Method (LRF-SSA): Other researchers tried to add a "local view" to every single layer of the AI's brain, assuming that every part of the network needed the same kind of help. The authors tested this and found that it didn't work everywhere. Just like how a toddler needs different rules than a teenager, different layers of the AI network need different levels of "local" focus. Applying the same fix to every layer sometimes made things worse or didn't change anything at all.

The Solution: SCLA-BCP

The authors realized that to truly fix the problem, they needed two things: a way to make sure the AI actually looks at physically adjacent pixels, and a way to let those pixels talk to their neighbors across the boundaries. They came up with SCLA-BCP.

SCLA (Spatially Contiguous Local Attention):
Imagine you have a giant pizza. Instead of cutting it into weird, scattered slices based on a pattern, you cut it into neat, square, non-overlapping blocks. SCLA forces the AI to only look at the pixels inside its own square block. This ensures that when the AI asks, "Who is my neighbor?", it is talking to the pixel literally sitting right next to it. This fixes the "shoe size" problem by guaranteeing that the group is a true, contiguous neighborhood.

BCP (Boundary Continuity Pathway):
But there's a new problem: if you cut the pizza into separate squares, the pepperoni on the edge of one square can't talk to the cheese on the edge of the next square. The information gets stuck. To fix this, the authors added the BCP. Think of this as a special "bridge" or a "courier service" that runs along the edges of the squares. It takes a quick look at the whole image (without cutting it up) using a simple, lightweight tool (a convolution) and passes that information back to the main system. This allows the AI to understand what's happening across the boundaries of its little blocks, ensuring that the whole picture stays connected.

How They Deployed It

The authors also realized that you can't just slap this new system onto every part of the AI. They found that for some types of AI architectures (the "ViT-like" ones), it works best if you only use it in the first half of the network. For other types (the "multi-stage" ones), it works best in the first two stages. It's like knowing that you need to teach a child to walk before you teach them to run; you don't put the running shoes on the baby. By carefully choosing where to put SCLA-BCP, they got the best results without wasting energy.

The Results: Seeing the World Better

The team tested their new method on seven different datasets, ranging from simple picture recognition (like identifying cats and dogs) to complex tasks like finding cars in a city or separating objects in a scene. The results were impressive:

Better Accuracy: On the COCO 2017 dataset (used for finding objects), their method improved the accuracy of finding objects by up to 9.50%. On the ADE20K dataset (used for understanding scenes), it improved the ability to separate different parts of an image by up to 3.42%.
Efficiency: They managed to get these big improvements while only adding a tiny amount of extra "weight" to the system. For example, on some models, they only added about 0.02 to 0.48 million parameters (the tiny building blocks of the AI) and a very small amount of extra energy (around 0.09 to 0.93 millijoules).
Proof of Concept: When they looked at how the AI was "thinking" (using a metric called Mean Attention Distance, or MAD), they saw that their method actually made the AI focus on closer neighbors, unlike the previous methods that sometimes failed to do so. Visualizations showed that their AI focused more clearly on the actual objects (like a chimpanzee) and ignored the background noise, whereas the older methods sometimes got distracted.

What This Means

The paper suggests that simply trying to make AI "local" isn't enough; you have to make sure the "local" part actually matches the physical layout of the image. The authors show that by combining a strict "neighborhood" rule (SCLA) with a "boundary bridge" (BCP), and by being smart about where to apply these rules, we can build Spiking Transformers that are not only more efficient but also much better at seeing the world. It's a reminder that in AI, sometimes the best way to see the big picture is to organize your neighborhood first, and then build a few bridges to connect them.

Technical Summary: Rethinking Attention Locality in Spiking Transformers

Problem Statement

Spiking Transformers have emerged as an energy-efficient paradigm for visual processing by utilizing spike-driven computation. However, a critical limitation persists in their core component, Spiking Self-Attention (SSA). To preserve the binary, event-driven nature of Spiking Neural Networks (SNNs), conventional Softmax normalization is removed from SSA. This removal leads to relatively uniform attention distributions with weak preferences for spatially adjacent tokens, resulting in insufficient local interactions that constrain visual representation capabilities.

Recent efforts to address this have introduced locality-enhanced SSA methods, primarily through two approaches:

Structural Modification: Methods like SGLFormer use offset-based token grouping to constrain attention scope.
Receptive Field Augmentation: Methods like LRF-SSA inject learnable dilated convolution kernels to impose locality priors.

Despite reported accuracy improvements, this paper identifies two fundamental gaps in existing research:

Computational–Spatial Locality Discrepancy: Existing methods often reduce the computational scope of attention without guaranteeing spatial locality. For instance, offset-based grouping (LSSA) creates groups based on coordinate residues modulo $g$ , which results in tokens being spatially separated rather than forming contiguous neighborhoods.
Uniform Deployment Inadequacy: Current approaches often uniformly replace all SSA layers with locality-enhanced variants. Analysis reveals that this "one-size-fits-all" strategy fails to consistently reduce the Mean Attention Distance (MAD) across all layers and architectures, indicating that locality deployment requirements are architecture-dependent.

Methodology

To address these issues, the authors propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP) and a Hierarchical Locality Deployment Strategy.

1. Spatially Contiguous Local Attention (SCLA)

SCLA directly addresses the computational–spatial discrepancy by enforcing attention within non-overlapping, spatially contiguous regions.

Mechanism: The feature map is partitioned into $w \times w$ blocks (where $w$ is the local region size). Attention is computed independently within each block using standard SSA operations (Query, Key, Value projections followed by LIF activation), but restricted strictly to tokens within the same block.
Effect: This ensures that the attention scope aligns with actual spatial neighborhoods, unlike offset-based grouping which creates scattered token sets.

2. Boundary Continuity Pathway (BCP)

A limitation of strict region partitioning is the prevention of direct information exchange between adjacent regions. BCP mitigates this by facilitating cross-boundary interaction.

Mechanism: BCP operates on the original, unpartitioned feature map using a lightweight convolutional pathway (consisting of a depth-wise convolution, batch normalization, and max pooling).
Integration: The output of BCP is integrated into the main stream via residual fusion with a learnable scalar $\alpha_l$ , initialized to 0.1. This allows the model to learn the necessary strength for boundary compensation while preserving the spike-driven computation property.

3. Hierarchical Locality Deployment Strategy

Motivated by the finding that uniform deployment is suboptimal, the authors propose an architecture-specific strategy for applying SCLA-BCP:

ViT-like Plain Architectures: SCLA-BCP replaces the first half ( $\lfloor L/2 \rfloor$ ) of the SSA layers.
Multi-stage Hierarchical Architectures: SCLA-BCP is applied to all eligible positions in Stages 1 and 2, while deeper stages retain their original structures.

Key Contributions

Systematic Analysis of Locality: The paper provides a quantitative analysis using Mean Attention Distance (MAD) to reveal that existing locality-enhanced methods do not consistently induce spatially localized token interactions. It highlights the discrepancy between computational constraints and spatial adjacency and the inadequacy of uniform layer replacement.
SCLA-BCP Architecture: The proposal of a novel module that combines spatially contiguous local attention (to enforce true spatial locality) with a boundary continuity pathway (to enable cross-region information flow).
Deployment Strategy: The development of a hierarchical deployment strategy that tailors the application of SCLA-BCP to the specific structural characteristics of ViT-like and multi-stage Spiking Transformers.
Comprehensive Evaluation: Extensive experiments across seven datasets (static and neuromorphic) covering classification, object detection, and semantic segmentation, demonstrating consistent performance gains with minimal overhead.

Experimental Results

The proposed method was evaluated on seven datasets: CIFAR-10/100, ImageNet-1K, CIFAR10-DVS, N-Caltech101, COCO 2017, and ADE20K.

Classification (ImageNet-1K): The method improved Top-1 accuracy by 0.51–0.98% over baselines across Spikingformer, QKFormer, and SDT-V3. Notably, on Spikingformer-8-768, accuracy increased from 77.64% to 78.62%, outperforming both LSSA and LRF-SSA variants.
Object Detection (COCO 2017): On SDT-V3-S/M, the approach improved mAP@50 by 9.50% and 5.60% respectively over baselines, surpassing the stronger LRF-SSA variant by 3.70% and 3.60%.
Semantic Segmentation (ADE20K): The method achieved mIoU improvements of up to 3.42% on SDT-V3-S over baselines.
Efficiency: The improvements were achieved with limited overhead. For example, on ImageNet-1K, the method introduced only 0.03–0.48M additional parameters and 0.09–0.93 mJ energy overhead compared to baselines.
Locality Validation: MAD analysis confirmed that SCLA-BCP consistently reduced token interaction distances across deployed layers compared to baselines and other locality-enhanced methods. Visualizations further showed that the method produces more concentrated responses on foreground objects compared to scattered activations in baseline models.

Significance and Claims

The paper claims that its primary significance lies in redefining how locality is implemented in Spiking Transformers. Rather than simply adding local receptive fields or constraining computation via offset grouping, the authors argue that true spatial locality requires contiguous region partitioning combined with a mechanism to handle boundary interactions.

The authors emphasize that their findings challenge the assumption that uniform application of locality enhancements is beneficial. By introducing a hierarchical deployment strategy, they demonstrate that the optimal placement of locality mechanisms is architecture-dependent. The work provides a robust, low-overhead solution that enhances the visual representation capability of Spiking Transformers across diverse tasks, validated by both quantitative metrics (accuracy, mAP, mIoU) and qualitative analysis (MAD reduction, attention visualizations).