🤖 machine learning

Hierarchical Global Attention (HGA)

Hierarchical Global Attention (HGA) is a drop-in, retraining-free method for long-context transformers that enables efficient inference on limited hardware by using a two-level routing mechanism to retrieve and attend to a sparse subset of tokens from host storage, achieving near-dense attention quality with minimal GPU memory overhead.

Original authors: Woernle Frank, Fedosov Vladimir, Grinenko Artemiy

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Woernle Frank, Fedosov Vladimir, Grinenko Artemiy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (the AI model) that has read billions of books. Usually, to answer a question, this library tries to remember every single word it has ever read in a specific conversation. If the conversation gets very long (say, 64,000 words), the library's "short-term memory" (the computer's video card or VRAM) gets completely full. It's like trying to hold a whole ocean in a teacup; the water spills, and the library can't function.

This paper introduces a new way for the library to work called Hierarchical Global Attention (HGA). Think of it as a smart librarian who doesn't try to hold every page in their hands at once. Instead, they use a clever filing system to find exactly what they need, right when they need it.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Teacup" Limit

Current AI models are like students who try to memorize the entire history of a conversation word-for-word. If the conversation is short, this is fine. But if the conversation is 64,000 words long, the student's brain (the GPU memory) explodes. They can't fit all the notes in their head, so they have to stop or crash.

2. The Solution: The "Smart Librarian" (HGA)

The authors created a "drop-in" patch. This means they didn't have to retrain the library or teach the student new ways of thinking. They just gave them a new filing system.

The system works in two levels, like a two-step search:

Step 1: The Chapter Summary (Chunk Routing)
Instead of looking at every single word from the past, the librarian first looks at summaries of chapters (chunks of text). Imagine the conversation is broken into 64-word blocks. The librarian creates a tiny "index card" for each block that says, "This block is about a car," or "This block is about a recipe."
When the student asks a question, the librarian quickly scans these index cards to find the chapters that are relevant.
Step 2: The Exact Page (Token Routing)
Once the librarian finds the right chapters, they don't just guess the answer. They go back and fetch the exact words from those specific chapters to give a precise answer.
Crucially: The final answer is calculated using the original, exact words, not the summaries. This ensures the answer is as accurate as if the student had read the whole book.

3. The Magic Trick: "Drop-In" Compatibility

Usually, to make a library smarter, you have to rebuild the whole building. Here, the authors just swapped the "memory retrieval" part.

They didn't change the library's brain (the model weights).
They didn't add any new training.
They just changed which pages the librarian is allowed to pull off the shelf at any given moment.

Because of this, they could take a massive, pre-trained AI model (Qwen3-30B) and run it on a single, powerful gaming computer (an RTX 5090) that usually couldn't handle such long conversations. The model fits because it only keeps the "hot" pages (recent words) and the "routed" pages (relevant old words) in its short-term memory, while the rest of the history sits safely on a hard drive (RAM) waiting to be fetched.

4. The Results: Fast, Cheap, and Accurate

The paper tested this with some impressive results:

The "Needle in a Haystack" Test: They hid a specific fact inside a 64,000-word document. The model found it 100% of the time, even though it was only looking at about 2% of the total text.
Speed: It was nearly 3 times faster at training and 2.4 times faster at processing long texts compared to the old method.
Accuracy: The answers were almost identical to the old method. The difference in quality was so tiny (about 0.01 to 0.02 "nats," which is a unit of information) that it's barely noticeable. The authors suggest this tiny gap isn't because the librarian is bad at searching, but because the way the AI measures "distance" in time (positional encoding) gets slightly fuzzy over very long distances.

Summary Analogy

Imagine you are writing a 64,000-word novel.

Old Way: You try to keep every sentence you've ever written in your head while writing the next one. Your brain gets tired and you make mistakes.
HGA Way: You keep the last few pages in your head. For the rest, you have a smart index. When you need to reference an idea from page 10,000, you quickly check the index, find the right chapter, flip to that page, read the exact sentence, and use it. You don't need to remember the whole book to write the next sentence, but you still get the details right.

The paper claims this method allows us to run massive AI models on standard hardware for very long tasks without losing the quality of the answers, simply by being smarter about how we retrieve information.

Technical Summary: Hierarchical Global Attention (HGA)

Problem Statement

The primary bottleneck for deploying pretrained long-context Large Language Models (LLMs) is not solely the $O(n^2)$ computational complexity of dense causal attention, but the memory requirement of the Key/Value (K/V) cache. In dense attention, the K/V cache for all historical tokens must reside in accelerator memory (VRAM). For quantized large models (e.g., FP8), the model weights themselves often consume the majority of the VRAM budget, leaving insufficient space to store the K/V cache for long contexts (e.g., 32K–64K tokens). This limitation prevents "out-of-the-box" inference of pretrained checkpoints on consumer-grade or single-GPU hardware without retraining or architectural modification.

Existing sparse attention methods often require training from scratch, introduce calibration parameters, or approximate the attention output itself, potentially degrading the quality of learned representations. There is a need for a drop-in mechanism that allows existing pretrained checkpoints to utilize sparse, memory-efficient retrieval while preserving the exactness of the final attention computation.

Methodology: Hierarchical Global Attention (HGA)

HGA is a systems-level patch designed as a drop-in replacement for dense causal attention in pretrained transformers. It preserves the original checkpoint parameters ( $W_Q, W_K, W_V, W_O$ ) and requires no retraining or calibration.

Core Mechanism

HGA shifts the long-range retrieval problem from dense token-to-token materialization to a two-level hierarchical routing process, followed by exact token-level attention:

Partitioning: The input sequence is divided into fixed-size chunks (e.g., $C=64$ tokens). Each chunk is further subdivided into groups.
Hierarchical Routing:
- Level 1 (Chunk Routing): The current chunk queries a compact table of "chunk key summaries" derived from previous chunks.
- Level 2 (Group Routing): Within selected chunks, the query routes to specific "group key summaries" to refine the selection.
- Deterministic Visibility: The routing policy combines content-based retrieval with fixed visibility windows: the current chunk, a configurable number of initial "sink" chunks, and recent local chunks are always visible.
RoPE-Aware Summaries: To ensure compatibility with Rotary Position Embeddings (RoPE), HGA constructs summaries using a mixed strategy: high-frequency RoPE pairs are rotated per token before averaging, while low-frequency pairs are averaged in raw-key space and rotated at the chunk's representative position. These summaries are computed from the existing projected keys without new linear layers.
Exact-Token Attention: Once relevant chunks or groups are selected, the router fetches the original token-level K/V vectors from host RAM (or NVMe) into VRAM. The final attention output is computed using standard softmax over this bounded, exact set of tokens. No synthetic summary values are used in the final output.

Memory Architecture

HGA introduces a tiered K/V storage abstraction:

Hot: Model weights, chunk summaries, and always-visible windows reside in VRAM.
Warm: A bounded LRU cache in VRAM holds recently routed chunks.
Cold: The full historical token K/V cache resides in host RAM (or NVMe).
This decouples GPU memory consumption from the total context length; VRAM usage depends primarily on model weights and the routed working set, not the total sequence length.

Key Contributions

Drop-In Compatibility: A mechanism to replace dense attention in pretrained GQA/MoE transformers without modifying weights, norms, or introducing calibration parameters.
Exact-Token Retrieval: Unlike methods that approximate attention outputs, HGA approximates only the retrieval stage, ensuring the final attention is computed exactly over the selected tokens.
Two-Level Routing Policy: A strategy combining deterministic sinks/local windows with content-routed retrieval from the middle context, utilizing chunk and group summaries.
Tiered Storage System: An implementation allowing full K/V history to reside in host RAM, enabling long-context inference on hardware with limited VRAM (e.g., a single RTX 5090).
Zero-Shot Performance: Demonstrated ability to run pretrained models (Qwen3-30B-A3B) at 32K–64K contexts without fine-tuning.

Experimental Results

The paper presents benchmarks across a 40M parameter SmallLM and a 30B parameter Qwen3 model.

Quality (Loss Gap):
- On a 40M SmallLM with direct weight copying (no fine-tuning), the routed model achieves a loss gap of +0.018 nats compared to dense attention at 8K tokens.
- Across context lengths from 4K to 64K, the loss gap remains within 0.01–0.02 nats of dense attention when using sparsity levels of approximately 3% to 12.5%.
- At 64K tokens, a sparsity of 6.25% maintains a gap of ~0.01 nats.
Retrieval Accuracy:
- A "Needle-in-a-Haystack" test on Qwen3-30B-A3B at 64K tokens achieved a 100% pass rate (3/3 depths) using group-level routing with ~1.9% sparsity, without any fine-tuning.
Performance:
- On the 40M SmallLM, a Triton-fused implementation achieved a 2.72× speedup in training and 2.43× speedup in prefill at 12K tokens compared to a dense baseline.
- On Qwen3-30B, the patched model runs interactively at 32K context on a single RTX 5090 (32GB VRAM), a configuration where dense K/V storage is impossible.
Fine-Tuning: Fine-tuning a 0.6B model with routed attention yielded a throughput of ~917 tokens/sec (vs. ~570 for dense) with a negligible routing cost (+0.015 nats) compared to the dense baseline.

Significance and Claims

The paper claims that HGA provides a practical, systems-level solution to the memory bottleneck of long-context LLMs. Its significance lies in:

Preserving Pretrained Knowledge: By avoiding retraining and using exact token K/V for the final output, HGA allows existing checkpoints to benefit from sparse attention without degrading their learned representations.
Hardware Efficiency: It enables the deployment of large, quantized models on consumer hardware for contexts far exceeding the VRAM capacity required for dense caching.
Error Attribution: The authors observe that the small remaining quality gap (0.01–0.02 nats) does not scale rapidly with context length. They suggest this gap is likely dominated by long-context positional encoding extrapolation rather than the routing algorithm itself, implying that HGA is a robust retrieval mechanism that can be further optimized by improving positional representations.

The authors position HGA not as a redesign of the transformer architecture, but as a compatible, hierarchical routing layer that makes long-context inference feasible on current hardware constraints.