💬 NLP

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

EndPrompt is an efficient method for extending the context window of large language models to 64K using only short training sequences by appending a terminal prompt with target-length positional indices, thereby achieving superior performance on benchmarks like RULER and LongBench while avoiding the high computational costs of full-length fine-tuning.

Original authors: Han Tian, Luxuan Chen, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Jinman Zhao, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Dawei Yin

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Han Tian, Luxuan Chen, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Jinman Zhao, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Dawei Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student (the AI model) who is great at reading short stories but gets confused when asked to read a whole encyclopedia. Usually, to teach this student to handle the encyclopedia, you'd have to make them read the entire thing over and over again. This is expensive, slow, and requires a massive library of long books that are hard to find.

The paper "EndPrompt" proposes a clever shortcut. Instead of forcing the student to read the whole book, you give them a short story and then attach a tiny, specific note at the very end that says, "This is the end of a very long book."

Here is how it works, broken down into simple concepts:

1. The Problem: The "Quadratic" Cost

Think of reading a long document like trying to connect every single word to every other word in the text. If you double the length of the text, the work required to connect the words doesn't just double; it quadruples. This makes training AI on long texts incredibly expensive and slow.

2. The Solution: The "Bookend" Trick

The researchers realized that the AI doesn't actually need to read the middle of a 64,000-word book to understand how to handle that length. It just needs to understand the distance between the beginning and the end.

They created a method called EndPrompt:

The First Segment: They take a normal, short piece of text (like a short story) and keep it intact. This is the "beginning" of the book.
The Second Segment: They attach a very short "terminal prompt" (a few words) to the end.
The Magic Trick: Even though the physical text is short, they tell the AI's internal clock that the short story is at position 0, and the tiny note at the end is at position 64,000.

3. The Analogy: The "Stretchy Rubber Band"

Imagine the AI's attention mechanism is like a rubber band.

Old Way: To teach the rubber band to stretch to 64,000 inches, you had to physically stretch it that far during training, which was hard and required a lot of force (computing power).
EndPrompt Way: You keep the rubber band short (physically), but you paint a mark at the very end and say, "This mark is 64,000 inches away." Because the AI uses a specific mathematical tool (called RoPE) that understands distance as a pattern (like a wave), it learns that the "wave" of distance between the start and this end mark is huge.

By keeping the story whole (not chopping it up into pieces), the AI learns the meaning of the story while simultaneously learning the math of long-distance relationships.

4. Why It Works (The "Smoothness" Theory)

The paper explains that the AI's math is "smooth." If you teach it how to handle a distance of 1 inch and a distance of 64,000 inches, the math naturally fills in the gaps for the distances in between (like 10,000 or 30,000 inches) without needing to be explicitly taught every single step. It's like teaching someone to jump a small puddle and a giant canyon; they intuitively figure out how to jump a medium-sized river in between.

5. The Results: Faster, Cheaper, Better

The researchers tested this on popular AI models (LLaMA family), trying to make them handle 64,000 words instead of just 8,000.

Efficiency: They only used short training sequences, saving massive amounts of time and money.
Performance: The AI performed better than models that were forced to train on full-length, expensive sequences.
Versatility: It worked well for various tasks like answering questions, summarizing text, and writing code, proving the AI didn't just memorize the trick but actually learned to handle long contexts.

Summary

EndPrompt is a way to teach an AI to handle massive amounts of text without making it read massive amounts of text. By keeping the training data short but "stretching" the position labels at the very end, the AI learns the concept of "long distance" efficiently. It's like teaching a runner to run a marathon by having them sprint a short distance while wearing shoes that feel like they are running a marathon.

Technical Summary: EndPrompt – Efficient Long-Context Extension via Terminal Anchoring

1. Problem Statement

Extending the context window of Large Language Models (LLMs) to handle inputs significantly longer than their pretraining limits (e.g., from 8K to 64K tokens) is a critical challenge for applications like long-document QA, code understanding, and evidence retrieval. The prevailing approach involves fine-tuning models on sequences at the target length. However, this method incurs prohibitive quadratic memory and computational costs due to the attention mechanism's scaling. Furthermore, collecting high-quality, long-form corpora is difficult.

Existing alternatives, such as position interpolation, sparse attention, or chunk-based simulation, often introduce limitations: they may require substantial long-sequence training, alter attention structures, or split contiguous text in ways that damage semantic continuity. A fundamental open question remains: Is dense supervision on full-length sequences necessary for reliable context extension, or can sparse, structured positional signals suffice?

2. Methodology: EndPrompt

The authors propose EndPrompt, a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length physical inputs.

Core Mechanism

EndPrompt constructs a training sequence composed of two segments:

Intact Short Context: The original short training text is preserved as the first segment.
Terminal Prompt: A brief, generic prompt is appended as a second segment.

Positional Index Manipulation:
While the physical sequence length remains short ( $a + b$ ), the assigned positional indices are manipulated to span the target context window ( $L$ ):

The short context retains local indices $[0, a-1]$ .
The terminal prompt is assigned indices near the target boundary $[L-b, L-1]$ .
Position Interpolation (PI): To adapt Rotary Position Embedding (RoPE) for this range, indices are rescaled by a factor $s > 1$ , effectively reducing the angular frequency of the positional encoding.

Theoretical Foundation:
The method leverages the properties of RoPE and PI:

Smoothness Constraint: PI reduces the effective frequency of positional variations, inducing a smoothness bias over the attention function. This suppresses unstable high-frequency oscillations in unobserved intermediate distances.
Shared Parameters: Transformers use shared query and key projections for all distances. By providing sparse supervision at both local (short context) and long-range (terminal prompt) extremes, the method constrains the shared parameter space, preventing the model from learning independent, unstable behaviors for unobserved intermediate distances.
Semantic Integrity: Unlike chunk-based methods that split text, EndPrompt preserves the semantic continuity of the original context, ensuring the model learns long-range interactions without disrupting local token prediction.

Training Objective

The model undergoes standard autoregressive fine-tuning. The loss function includes predictions for both the original context and the terminal prompt, with the terminal tokens assigned a non-zero weight. This forces the model to attend from the terminal anchor back to the original context across the artificially large assigned distance, acting as an implicit regularizer.

3. Key Contributions

EndPrompt (ET) Framework: An efficient context-extension method that utilizes short training sequences, positional index manipulation, and an appended end prompt to simulate long-range positional supervision.
Semantic Preservation: A design that retains the original context as an undivided segment, maintaining semantic continuity while using the end prompt as a stable terminal anchor to create long-distance relationships.
Theoretical Analysis: An explanation grounded in RoPE and Position Interpolation, demonstrating how smooth positional variation and shared Transformer parameters enable generalization over unobserved intermediate distances.
Empirical Validation: Comprehensive evaluation showing that structured sparse positional supervision can induce reliable long-context generalization without full-length training.

4. Experimental Results

The method was evaluated on LLaMA-family models (LLaMA-2 7B and LLaMA-3 8B), extending the context window from 8K to 64K.

Benchmarks:

RULER (Synthetic Benchmark): EndPrompt achieved an average score of 76.03, outperforming LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23). It demonstrated particular robustness in complex information retrieval tasks (e.g., Multi-needle, Variable Tracking).
LongBench (Downstream Tasks): EndPrompt secured the highest average score (38.30) across tasks including Single/Multi-Doc QA, Summarization, Few-Shot Learning, and Code Completion, surpassing full-length fine-tuning (35.63).
Short-Text Capabilities: After supervised fine-tuning, the method maintained strong performance on standard short-text benchmarks (MMLU, GSM8K, HumanEval, HellaSwag), achieving an average of 53.56, outperforming other baselines.

Ablation Studies:

Base Models: The method proved effective across different architectures (Mistral-7B, LLaMA-2-7B).
Extension Length: Performance remained stable when extending to 32K, 96K, and even 128K, with only minor degradation at extreme lengths.
Token Quantity: Scaling training data from 0.5B to 2.0B tokens yielded consistent performance improvements.
Prompt Robustness: The method was insensitive to specific variations in the end prompt wording, relying on the structural position rather than specific token memorization.
Compatibility: Combining EndPrompt with Positional Skip-Embedding (ET(PoSE)) yielded the highest scores on both benchmarks.

5. Significance and Claims

The paper claims that EndPrompt challenges the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. By demonstrating that models can acquire long-context capabilities from sparse positional supervision on short sequences, the work offers a practical, cost-effective alternative to full-length fine-tuning.

The authors emphasize that the success of the method stems from the synergy between:

Preserving semantic integrity by avoiding text chunking.
Utilizing the smoothness bias of Position Interpolation.
Constraining shared parameters through dual-scale supervision (local and terminal).

The paper concludes that reliable long-context adaptation can be induced through the design of informative positional supervision, keeping physical training sequences short while achieving state-of-the-art performance on long-context benchmarks.