💬 NLP

Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction

This paper introduces Hyper-Parallel Decoding (HPD), a novel algorithm that accelerates large language model inference for tasks like Attribute Value Extraction by leveraging the conditional independence of output sequences to enable out-of-order, parallel token generation, thereby reducing inference time and costs by up to 13.8X without compromising quality.

Original authors: Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Yilun Zhu, Shervin Malmasi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a highly skilled librarian (the AI) tasked with filling out a massive stack of product cards. Each card has a list of blank fields you need to fill in, like "Screen Size," "Display Type," and "Resolution."

The Old Way (Autoregressive Decoding):
Traditionally, the librarian fills these out one by one, strictly from top to bottom. They must finish writing "Screen Size" before they can even start thinking about "Resolution." Even though "Screen Size" has nothing to do with "Resolution," the librarian is forced to wait for the first task to complete before starting the next. It's like a single-lane road where every car must wait for the one in front to finish before moving. This is slow and expensive because the librarian is working sequentially.

The New Way (Hyper-Parallel Decoding or HPD):
The paper introduces a clever trick called Hyper-Parallel Decoding (HPD). The authors realized that for tasks like this, the answers are actually independent. Knowing the screen size doesn't change the resolution. So, why wait?

HPD allows the librarian to work on all the blank fields at the same time.

Here is how they pull off this magic trick without breaking the librarian's brain (the AI model):

The "Fake" Gaps: The librarian is trained to look at the position of words to understand order. HPD tricks the librarian by creating "fake gaps" in the sentence. Imagine the librarian sees a list where the first answer is at position 1, but the second answer is magically skipped to position 10, and the third to position 20.
Filling the Gaps: Because the librarian thinks these answers are far apart in the sentence, it doesn't matter that it's writing them all simultaneously. It can generate the first letter of "Screen Size," the first letter of "Resolution," and the first letter of "Display Type" all in the same instant.
The Assembly Line: In the next moment, it fills in the second letter of all three answers at once. It keeps doing this until all the fields are full.

The "Stacking" Bonus:
The paper also mentions a second trick called Document Stacking. Imagine instead of filling out one card, the librarian is given a stack of 10 cards and told to fill out the same fields for all of them at once. HPD combines this with the "fake gaps" trick. Now, the librarian isn't just filling 3 fields on one card; they are filling 3 fields on 10 different cards simultaneously. It's like a factory assembly line that suddenly doubled its speed by processing multiple products in parallel.

The Results:
The paper tested this on real-world e-commerce data (like TV specifications). They found that:

Speed: They could process up to 13.8 times faster than the old method.
Cost: Because it's so much faster, it costs up to 13.8 times less to run.
Quality: The answers were just as accurate as the slow, old method. In some cases, they were even slightly better.

In a Nutshell:
The paper doesn't invent a new librarian; it just invents a new way of organizing the desk. By rearranging the "position IDs" (the seat numbers) and using a special mask to tell the librarian which words to look at, they broke the rule that says "you must do one thing at a time." This turns a slow, single-lane road into a high-speed, multi-lane highway, saving huge amounts of time and money for businesses that need to extract data from millions of product descriptions.

Important Note: The authors specifically state this works for tasks where the answers are independent (like product attributes). They do not claim this works for tasks where the answer to one question depends heavily on the answer to the previous one (like writing a complex story or solving a math problem step-by-step).

1. Problem Statement

Attribute Value Extraction (AVE) is a critical sub-task in Information Extraction (IE), particularly in e-commerce, where models must extract multiple independent values (e.g., Screen Size, Resolution, HDR Support) from a single unstructured product document.

The Bottleneck: Standard Large Language Models (LLMs) use autoregressive decoding, generating tokens sequentially ( $t=1, 2, \dots$ ). This creates a dependency where the generation of the $n$ -th attribute value depends on the completion of the $(n-1)$ -th value.
The Inefficiency: This sequential nature prevents parallelization, leading to high inference latency and prohibitive costs when processing massive datasets offline.
The Opportunity: In AVE tasks, the extraction of different attribute values is conditionally independent given the input document context. The value of "Screen Size" does not theoretically require the value of "Resolution" to be known first. However, standard autoregressive generation enforces an artificial dependency chain that limits throughput.

2. Methodology: Hyper-Parallel Decoding (HPD)

The authors propose Hyper-Parallel Decoding (HPD), a novel inference algorithm that breaks the autoregressive chain to generate multiple independent attribute values simultaneously within a single prompt.

Core Mechanisms

Breaking the Dependency Chain:
- HPD assumes conditional independence: $P(V_1, \dots, V_N | p) = \prod P(V_n | p)$ .
- Instead of generating tokens $1 \to T$ sequentially, HPD generates the first token of all $N$ values in parallel, then the second token of all $N$ values, and so on.
- This reduces the number of inference steps from $T$ (total tokens) to $K_{max}$ (maximum length of a single value).
Position ID Manipulation:
- To enable parallel generation within a standard decoder-only LLM, HPD manipulates Position IDs.
- Gap Insertion: The input prompt is constructed with a "skeleton" containing attribute keys. Position IDs are incremented by a gap size ( $K_{max}$ ) after each attribute key, creating "slots" for the values.
- Out-of-Order Memory: New tokens are appended to the end of the sequence in memory (to maintain Key-Value cache continuity) but are assigned Position IDs that place them logically in the gaps.
- Custom Attention Masks: A modified causal attention mask is applied based on Position IDs rather than memory order. This ensures that when generating the $k$ -th token of a value, the model cannot "see" the $k$ -th tokens of other values (maintaining causality) but can see the prompt and previous tokens of its own value.
Document Stacking (Hyper-Parallelism):
- HPD introduces a second dimension of parallelism by stacking multiple documents ( $J$ ) within a single prompt.
- Since values across different documents are also independent, the system can decode $J \times N$ tokens in a single inference step.
Training and Fine-Tuning:
- Zero-Shot Compatibility: HPD works directly on pre-trained models without architectural changes.
- Fine-Tuning Strategy: To address the train/test mismatch (where training data is sequential but inference is parallel), the authors fine-tune models using a modified attention mask that mimics the HPD inference pattern. This allows the model to learn the specific attention distribution required for parallel decoding.

3. Key Contributions

Novel Algorithm: Introduction of HPD, which enables the generation of up to 96 tokens per prompt (via document stacking and multiple attributes) by leveraging shared memory and computation across independent sequences.
Plug-and-Play Efficiency: HPD is compatible with existing cost-reduction techniques (quantization, knowledge distillation) and standard batched inference, requiring no changes to the underlying LLM architecture.
Theoretical Generalization: While designed for AVE, the method relies on the assumption of conditional independence, making it theoretically applicable to any task with independent output structures.
Comprehensive Evaluation: Extensive testing on three e-commerce benchmarks (OA-Mine, AE110k, Amazon Reviews 2023) across various model sizes (1.7B to 32B parameters).

4. Experimental Results

The authors evaluated HPD against standard Autoregressive (AR) decoding and Speculative Decoding.

Throughput & Speed:
- HPD achieved up to 13.8X speedup in inference time compared to AR decoding.
- On the Amazon Reviews dataset, HPD achieved a 10.78X throughput increase over AR, significantly outperforming Speculative Decoding (which only achieved ~2X speedup).
Cost Reduction:
- Inference costs were reduced by up to 13.8X (e.g., from \ $7.74 to \$ 0.89 per 1k products for Qwen3-32B on Amazon Reviews).
- Extrapolating to a corpus of 500M documents, HPD could save approximately \ $597,000** compared to AR and **\$ 486,000 compared to GPT-4.1.
Output Quality:
- No Quality Degradation: HPD maintained state-of-the-art F1 scores. In some cases (e.g., AE110k), fine-tuned HPD models actually outperformed AR baselines by ~4.5%.
- Human Evaluation: Blind human tests on Amazon Reviews confirmed that HPD outputs were preferred or equal to AR outputs and GPT-4.1.

5. Significance and Impact

Solving the Offline Bottleneck: HPD specifically targets the offline batched inference scenario (common in data processing pipelines), where throughput and cost are more critical than low-latency single-request response.
Hardware Utilization: By maximizing the number of tokens generated per inference step, HPD better utilizes GPU VRAM and compute resources, overcoming the sequential limitations of standard Transformers.
Economic Viability: The drastic reduction in inference costs makes high-quality generative IE feasible for large-scale industrial applications (e.g., cataloging millions of products) where previous costs were prohibitive.
Future Directions: The paper suggests that HPD could be extended to other domains with independent output structures, provided the conditional independence assumption holds.

Conclusion: Hyper-Parallel Decoding represents a paradigm shift in LLM inference for structured extraction tasks. By decoupling the generation of independent sequences through position ID manipulation and custom attention masking, it achieves massive efficiency gains without sacrificing output quality.