💬 NLP

Avey-B

This paper introduces Avey-B, a reformulated, attention-free, encoder-only architecture that incorporates innovations like decoupled parameterization and neural compression to outperform standard Transformer-based encoders in token classification and information retrieval while scaling more efficiently to long contexts.

Original authors: Devang Acharya, Mohammad Hammoud

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Devang Acharya, Mohammad Hammoud

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, 100-page mystery novel.

The Old Way (Transformers/BERT):
Traditional AI models, like the famous BERT, try to read the whole book at once to understand any single sentence. They look at every word in the book and compare it to every other word to find connections.

The Problem: This is like trying to read a library while holding a magnifying glass. If the book is short, it's fine. But if the book is 100 pages long, the effort to compare every word to every other word explodes. It gets slow, expensive, and the computer runs out of memory. It's like trying to find a specific needle in a haystack by measuring the distance between every piece of hay and every other piece of hay.

The New Way (Avey-B):
The paper introduces Avey-B, a new kind of AI brain that reads differently. Instead of staring at the whole book at once, it uses a smart "librarian" system.

Here is how Avey-B works, broken down with simple analogies:

1. The Smart Librarian (The Ranker)

Imagine you are reading a chapter. Instead of trying to remember the entire book, you ask a librarian: "Hey, which 3 other chapters in this book are most relevant to what I'm reading right now?"

The librarian (called the Ranker) quickly scans the book, picks the top 3 relevant chapters, and hands them to you.
You ignore the other 97 chapters for now. This saves a massive amount of time and energy.

2. The Two-Step Thinking Process (Decoupled Parameters)

Once you have your current chapter and the 3 relevant ones, you need to understand them. The old AI (Avey) tried to mix its "hard-learned rules" with "what it sees right now" all at once.

The Flaw: It's like trying to drive a car while simultaneously arguing with your GPS. Sometimes the rules fight the reality, and the car goes in circles.
The Fix (Avey-B): Avey-B separates these tasks.
- Step A (Static): It applies its general knowledge (like grammar rules).
- Step B (Dynamic): It looks at the specific words in front of it and adjusts based on similarity.
- The Result: It's like having a co-pilot. One person drives (the rules), and the other navigates (the context). They don't fight; they take turns, making the ride smoother and more logical.

3. The Compression Machine (Neural Compression)

Here is the tricky part. If you take your current chapter and add 3 whole other chapters, you now have 4x the text to read. That's still too much!

The Innovation: Avey-B has a "summarizer" (the Compressor). Before you start reading the 4 chapters, the summarizer instantly condenses them back down into the size of just one chapter, but it keeps all the important details.
The Analogy: It's like taking a 4-hour movie, compressing it into a 1-hour highlight reel that still tells the whole story perfectly. This allows the AI to process long documents without getting overwhelmed.

4. Why is this a big deal?

The authors tested Avey-B against the current champions (like ModernBERT and NeoBERT) on two main things: Accuracy and Speed.

Accuracy: Avey-B was better at understanding text, especially at finding specific details (Token Classification) and finding relevant documents (Information Retrieval). It beat the others even though it was trained on 11 times less data than one of its rivals.
Speed (The Long Context): This is the real showstopper.
- Imagine the text length is the size of a city.
- Traditional AI gets slower and slower as the city gets bigger. By the time the city is huge (96,000 words), the traditional AI is moving at a snail's pace.
- Avey-B stays fast. Whether the text is a short note or a massive novel, it zooms through it. At the longest lengths tested, Avey-B was 3 to 11 times faster than the competition.

The Bottom Line

Think of traditional AI as a photographer trying to take a picture of a whole stadium by measuring every single fan's face against every other fan's face. It's accurate but impossible for a huge crowd.

Avey-B is a smart scout. It scans the stadium, finds the specific groups of fans that matter to the question, zooms in on them, summarizes the crowd, and gives you the answer instantly. It's faster, cheaper, and surprisingly smarter, especially when dealing with huge amounts of text.

The paper proves that you don't need the "heavy" self-attention mechanism (the old way) to build great AI. Sometimes, a smart retrieval system with a good compressor is the key to the future.

1. Problem Statement

Compact, bidirectional pretrained encoders (like BERT) are the backbone of industrial NLP due to their ability to provide high-quality contextualized representations. However, they face two critical limitations:

Scalability Bottleneck: Standard Transformer self-attention mechanisms incur quadratic time and memory complexity ( $O(N^2)$ ) with respect to sequence length $N$ . This makes extending context windows to long sequences (e.g., 10k–100k tokens) computationally prohibitive in cost-sensitive deployments.
Architectural Rigidity: While recent non-attention architectures (like Avey) have shown promise in autoregressive (decoder-only) settings for long contexts, they have not been successfully adapted to the bidirectional encoder-only paradigm, which is essential for discriminative tasks like token classification, information retrieval, and extractive QA.

Existing alternatives (linear attention, RNNs, state-space models) often struggle to match the bidirectional contextualization quality of Transformers or fail to adapt effectively to encoder-only settings.

2. Methodology: Avey-B Architecture

The authors propose Avey-B, a reformulation of the autoregressive, attention-free Avey architecture into a bidirectional encoder. Avey-B retains the core mechanism of partitioning sequences into splits and retrieving relevant context via a Ranker, but introduces three key architectural innovations to enable bidirectional processing and improve stability/efficiency.

A. Bidirectional Contextualization

Unlike the original Avey, which uses a causal mask, Avey-B removes the autoregressive constraint in the Neural Processor. This allows every token within a split to attend to both left and right contexts, enabling full bidirectional representation learning while maintaining the split-based retrieval mechanism.

B. Decoupled Static and Dynamic Parameterizations

In the original Avey, the contextualizer multiplies learned static weights with input-dependent cosine similarity scores element-wise. The authors identify this as a source of instability, where high similarity scores can be inverted by learned weights, violating the monotonicity of relevance (i.e., a more relevant token contributing less than a less relevant one).

Solution: Avey-B decouples these operations by alternating layers:
- Static Layers: Apply a learned linear transformation (cross-embedding matrix) independent of input similarity.
- Dynamic Layers: Compute context solely based on input-dependent cosine similarities (normalized).
Benefit: This preserves the monotonicity guarantee within dynamic layers (higher similarity $\to$ higher weight) while allowing static layers to shape representations and introduce inhibitory patterns without disrupting the similarity-based ordering.

C. Row-Normalized Similarity Scores

To stabilize training in dynamic layers, Avey-B normalizes cosine similarity scores by the row-wise sum (dividing each score by the sum of scores in that row).

Benefit: This creates a row-stochastic operator that bounds per-row gain, prevents the explosion of singular values with depth, and improves numerical conditioning compared to Softmax or unnormalized approaches.

D. Neural Compression

In bidirectional inference, every split must be contextualized with its top- $k$ retrieved splits. Concatenating these directly would inflate the sequence length by a factor of $k+1$ , negating efficiency gains.

Solution: Avey-B introduces a Neural Compressor within the Ranker. It maps the concatenated block of $(k+1)$ splits back to the original split size $S$ via a learned linear projection.
Benefit: The Neural Processor operates on a fixed token budget ( $S$ ) regardless of $k$ . This allows the model to retrieve a large context ( $k$ ) for relevance without incurring the computational cost of processing all those tokens, achieving linear scaling $O(N)$ in practice.

3. Key Contributions

Avey-B Architecture: The first successful bidirectional, attention-free encoder that decouples static and dynamic parameterizations to ensure stable, monotonic relevance updates.
Neural Compression Scheme: A novel mechanism that compresses retrieved context, enabling efficient long-context processing in bidirectional settings without sacrificing the benefits of large retrieval windows.
Comprehensive Evaluation: Extensive ablation studies and comparisons against state-of-the-art Transformer encoders (BERT, RoBERTa, ModernBERT, NeoBERT).
Open Source: Release of full implementation and pretrained checkpoints to foster reproducibility.

4. Experimental Results

Effectiveness (Accuracy)

Avey-B was evaluated on standard benchmarks across four categories: Sequence Classification (SC), Token Classification (TC), Question Answering (QA), and Information Retrieval (IR).

Performance: Avey-B consistently outperforms BERT and NeoBERT across all benchmarks.
TC & IR: It achieves state-of-the-art results on Token Classification and Information Retrieval, surpassing even large Transformer models (ModernBERT, RoBERTa). Notably, the Avey-B base model outperforms large Transformer encoders on TC and IR tasks despite being pretrained on ~11x fewer tokens than ModernBERT.
Robustness: Avey-B demonstrates lower variance across random seeds compared to many Transformer baselines, indicating higher optimization stability.

Efficiency (Scalability)

The authors measured throughput and latency on NVIDIA H200 GPUs across sequence lengths from 128 to 96k tokens.

Scaling Behavior: Avey-B exhibits linear scaling ( $O(N)$ ) in throughput, whereas Transformers exhibit quadratic degradation.
Power-Law Decay: Fitting a power-law model $T(N) \propto N^{-\alpha}$ $T (N) \propto N^{- α}$ :
- Avey-B: $\alpha \approx 0.44$ (optimized) / $0.33$ (unoptimized).
- ModernBERT: $\alpha \approx 0.77$ (optimized).
- NeoBERT: $\alpha \approx 0.81$ (optimized).
Latency Advantage: At 96k tokens, Avey-B is 3.38x faster than ModernBERT and 11.63x faster than NeoBERT.
Long-Context Retrieval (NIAH): In "Needle-in-a-Haystack" tests up to 96k tokens, Avey-B maintains high accuracy (dropping only ~3-4 points from 1k to 96k), while Transformer baselines fail to extrapolate beyond their training windows (4k–8k) or crash due to Out-of-Memory (OOM) errors.

5. Significance

Beyond Attention: The paper challenges the dogma that self-attention is the only viable path for high-performance bidirectional encoders. It demonstrates that retrieval-conditioned, attention-free architectures can match or exceed Transformer performance.
Industrial Viability: Avey-B offers a practical solution for industries requiring high-throughput, low-latency processing of long documents (e.g., legal, medical, or financial text analysis) where Transformer costs become prohibitive.
Architectural Insights: The findings on decoupled parameterization and row-normalization provide new design principles for building stable, deep, non-attention networks that avoid the pathological interactions between learned weights and input-dependent scores.

In summary, Avey-B represents a significant step forward in efficient NLP, offering a scalable, bidirectional alternative to Transformers that excels in both accuracy and long-context efficiency.