💬 NLP

Lizard: An Efficient Linearization Framework for Large Language Models

Lizard is a novel linearization framework that transforms pretrained Transformer-based LLMs into subquadratic architectures using adaptive, learnable modules and a hardware-aware algorithm to overcome quadratic complexity bottlenecks while achieving near-lossless performance recovery and superior length generalization compared to prior methods.

Original authors: Chien Van Nguyen, Huy Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Hu

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chien Van Nguyen, Huy Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart librarian (the Large Language Model or LLM) who has read almost every book in the world. This librarian is amazing at answering questions, writing stories, and solving puzzles. However, there's a catch: this librarian has a terrible memory system.

The Problem: The "Quadratic" Bottleneck

In the current world, if you ask the librarian to remember a story that is 10 pages long, they have to flip through all 10 pages every time they read a new sentence. If the story is 1,000 pages long, they have to flip through 1,000 pages for every single word they write.

This is called quadratic complexity. It's like trying to find a specific grain of sand in a beach by checking every single grain of sand against every other grain.

The Result: The librarian gets exhausted (computational cost), runs out of desk space (memory), and eventually gives up when the story gets too long. They can't handle "long-context" tasks like reading a whole novel or analyzing a year's worth of emails at once.

The Solution: Enter "Lizard"

The researchers created a new framework called Lizard. Think of Lizard not as a new librarian, but as a magic filing system that you attach to the existing brilliant librarian. It transforms the librarian's messy, slow memory into a super-efficient, compact one that can handle infinite-length stories without getting tired.

Here is how Lizard works, broken down into three simple parts:

1. The "Smart Gate" (Adaptive Memory)

Imagine the librarian has a bucket where they throw away old memories. Usually, they just throw things away randomly or based on a fixed schedule.

Lizard's Innovation: Lizard adds a smart gate to the bucket. This gate is like a bouncer that decides, "Do I need to remember this specific detail?"
If the detail is important (like a character's name), the gate keeps it. If it's just noise (like "the sky was blue"), the gate lets it fade away.
Why it matters: This allows the librarian to remember the essence of a 100,000-page book without needing a warehouse the size of a city to store it. It learns what to forget and what to keep, rather than just following a rigid rule.

2. The "Anchor Window" (Local Precision)

While the "Smart Gate" is great for remembering the big picture, it sometimes misses tiny, sharp details (like a specific number or a quote).

Lizard's Innovation: Lizard adds a small, high-definition magnifying glass (called the Anchor Window) that looks closely at the last few sentences.
The Analogy: Think of the Smart Gate as a wide-angle lens seeing the whole forest, and the Anchor Window as a zoom lens looking at a specific tree. Lizard combines both: it keeps the big picture in its compact memory while using the zoom lens to get the fine details right.

3. The "Hardware Fix" (Speeding Up the Engine)

The researchers noticed that when they tried to build this "Smart Gate," the math got messy and unstable on computer chips, forcing the computer to slow down and use more power.

Lizard's Innovation: They rewrote the math (like changing the recipe for a cake) so it fits perfectly into modern computer chips (Tensor Cores).
The Result: This made the training process 32% faster. It's like tuning a car engine so it runs smoother and uses less fuel without changing the car's design.

The Results: Why Should You Care?

The paper tested Lizard by taking a top-tier AI model (like Llama-3 or Mistral) and swapping its slow memory for Lizard's efficient system.

Performance: The new "Lizard" model performed almost exactly as well as the original genius librarian. It didn't lose its smarts. In fact, on some tests, it was 9 to 24 points better than other attempts to make AI faster.
Memory: The original model crashed when trying to read a 32,000-word story. Lizard read it with constant memory, meaning it used the same amount of computer power whether the story was 1 page or 1,000 pages.
Recall: If you hid a secret code in a 64,000-word document, Lizard could find it perfectly. Other methods failed as soon as the document got longer than what they were trained on.

The Bottom Line

Lizard is a bridge. It takes the powerful, expensive AI models we have today and gives them a "superpower" to read infinite amounts of text without slowing down or running out of memory.

It's like taking a Ferrari (the current AI) and giving it a hybrid engine (Lizard) that lets it drive forever without stopping for gas, while still being just as fast and smart as before. This makes it possible to run advanced AI on smaller devices or use it for tasks that require reading entire books, legal contracts, or years of data in seconds.

1. Problem Statement

Large Language Models (LLMs) based on Transformer architectures rely on softmax attention, which suffers from quadratic time and memory complexity ( $O(L^2)$ ) relative to sequence length ( $L$ ). This creates severe bottlenecks:

Training/Inference Cost: Computation scales quadratically, making long-context processing prohibitively expensive.
Memory Bound: The Key-Value (KV) cache grows linearly with context length, causing Out-of-Memory (OOM) errors during generation for long sequences.
Limitations of Existing Solutions:
- Training from Scratch: Models trained with linear attention (e.g., Mamba, RWKV) from scratch often underperform compared to Transformers on tasks requiring in-context learning and retrieval.
- Linearization Methods: Existing methods that attempt to linearize pretrained Transformers (e.g., LoLCATs, Liger) fail to match the teacher model's performance (often dropping 10–20+ points on MMLU) and struggle with length extrapolation due to fixed positional encodings (RoPE) and non-adaptive memory structures.

2. Methodology: The Lizard Framework

Lizard is a framework designed to transform pretrained Transformer-based LLMs into sub-quadratic architectures with near-lossless performance recovery. It employs a two-stage training process and a hybrid attention mechanism.

A. Hybrid Attention Architecture

Lizard replaces standard softmax attention with a combination of two components:

Gated Linear Attention (GLA) with Learnable Gating:
- Adaptive Memory Control: Instead of fixed positional encodings (RoPE), Lizard introduces a learnable gating module ( $\Gamma_i = \sigma(W_\gamma x_i)$ ). This creates a data-adaptive recurrent structure where the model dynamically decides how much past information to retain or forget.
- Length Generalization: The gating mechanism implicitly learns relative positional information through adaptive decay patterns, enabling robust extrapolation to sequence lengths unseen during training.
- State Update: It maintains a constant-memory hidden state ( $S_i = \Gamma_i S_{i-1} + \phi(k_i)v_i^\top$ ), eliminating the need for a growing KV cache.
Anchor Window Attention:
- To recover the "spiky" precision of softmax attention for local context, Lizard augments a local sliding window with learnable meta-memory tokens ( $t$ ).
- These tokens act as "attention sinks" in the denominator of the softmax computation, stabilizing the distribution and managing massive activations without contributing directly to the output value.
- This preserves fine-grained local interactions while the GLA handles global context compression.

Final Output: The final output is a weighted sum: $\hat{Y}_{lizard} = \hat{Y}_{gate} + \alpha \cdot \hat{Y}_{anchor}$ .

B. Training Strategy

Stage 1: Attention Approximation: The model is trained to mimic the teacher model's attention outputs (including RoPE patterns) using a Mean Squared Error (MSE) loss. This stage distills the teacher's attention patterns and positional awareness into the linearized architecture.
Stage 2: Language Modeling Alignment: The model is fine-tuned using standard autoregressive language modeling objectives (via LoRA) to align with downstream tasks.

C. Hardware-Aware Algorithm

Standard Gated Linear Attention is numerically unstable in low-precision formats (e.g., bfloat16) due to cumulative gate products causing underflow.

Solution: Lizard introduces a reparameterization that absorbs the cumulative gate term into the query and key projections in log-space.
Benefit: This transforms the operation into a standard General Matrix Multiplication (GEMM), enabling efficient execution on Tensor Cores without falling back to slow full-precision (float32) operations. This improves training throughput by 32–36%.

3. Key Contributions

Lizard Framework: A novel linearization method that converts pretrained Transformers into sub-quadratic architectures with near-lossless performance recovery.
Adaptive Memory & Extrapolation: Introduction of learnable gating mechanisms that replace fixed positional encodings, enabling robust length extrapolation (up to 64K tokens) and dynamic memory control.
Hardware Efficiency: A reparameterization algorithm that solves numerical instability in gated attention, enabling low-precision training and significantly accelerating throughput.
Hybrid Design: A combination of global gated recurrence and local anchor window attention that balances global context compression with local precision.

4. Experimental Results

The authors evaluated Lizard on Mistral-7B and LLaMA-3-8B teacher models.

Language Modeling Benchmarks (MMLU):
- Lizard significantly outperforms prior linearization methods. On the 5-shot MMLU benchmark, it improves upon previous methods by 9.4 to 24.5 points.
- Comparison:
  - LLaMA-3-8B-Lizard: 74.6 (Avg) / 61.2 (MMLU).
  - LoLCATs (LLaMA-3-8B): 70.7 (Avg) / 52.8 (MMLU).
  - Liger (LLaMA-3-8B): 67.6 (Avg) / 43.4 (MMLU).
- In a hybrid setup (retaining 50% softmax layers), Lizard nearly matches the teacher model (65.1 vs. 66.6 on MMLU).
Long-Context Retrieval (Associative Recall):
- Needle-in-a-Haystack: Lizard achieves perfect retrieval accuracy across various context lengths (up to 64K), whereas LoLCATs fails immediately when the sequence exceeds the training window.
- RULER Benchmark: Lizard maintains high accuracy (81.3% at 32K) closely matching the full-attention teacher model (80.5%).
Efficiency:
- Memory: Lizard maintains constant memory usage and stable throughput even at 32K sequence lengths, while the teacher model (with FlashAttention-2) runs out of memory.
- Speed: The hardware-aware algorithm provides a 32–36% speedup in forward pass latency compared to standard GLA implementations.

5. Significance

Bridging the Gap: Lizard successfully bridges the gap between the high reasoning capabilities of massive pretrained Transformers and the computational efficiency of sub-quadratic architectures.
Scalability: It provides a practical path to deploy state-of-the-art LLMs in resource-constrained environments (e.g., edge devices) with infinite context capabilities without sacrificing reasoning quality.
Overcoming Linearization Limits: By introducing learnable gating and hybrid attention, Lizard overcomes the historical trade-off where linear models either lost performance or failed to generalize to long contexts.

6. Limitations

Dependency on Teacher: The method relies heavily on a high-quality pretrained backbone; performance may degrade if the teacher model is weak.
Recall-Memory Tradeoff: While efficient, the fixed-size state representation still faces inherent tradeoffs compared to full attention models in extremely recall-intensive tasks, though Lizard pushes the Pareto frontier significantly.