Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs

Technical Summary: Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs

1. Problem Statement

Personalizing Large Language Models (LLMs) requires providing extensive user interaction histories and profiles to tailor responses. However, this presents two major challenges:

• Resource Constraints: Long contexts lead to high inference latency and increased API costs, making real-time deployment impractical.
• Information Noise: Not all historical data is relevant to a specific task. Naively including all signals can introduce noise and degrade performance.

Existing solutions typically use heuristic methods (e.g., selecting only recent interactions) or prompt-based summarization. The authors argue these methods treat context as a "monolithic whole" and fail to leverage the LLM's internal ability to prioritize specific information.

2. Methodology

The authors propose Attn-GS, an attention-guided context compression framework. The core insight is that an LLM's internal attention patterns naturally reveal which personalization signals are most relevant to a given task. The framework operates in two distinct stages:

Stage 1: Critical Personalization Sentence Marking

• Marking Model ($\Phi_{Mark}$): A "white-box" LLM (which allows access to attention weights) is used to process the user history ($H$) and the task description ($T$).
• Attention Extraction: The model extracts token-level attention scores from specific layers (typically middle layers). These are aggregated into sentence-level scores.
• Threshold-based Filtering: Sentences whose average attention scores exceed a certain threshold ($\alpha \times \text{max\_score}$) are identified as "important."
• Explicit Marking: These important sentences are wrapped in special tags (e.g., <start_important> and <end_important>) to create a modified context ($H^*$).

Stage 2: Summarization Based on Marked Context

• Summary Model ($\Phi_{Sum}$): A second LLM receives the marked context ($H^*$) and a specific instruction to prioritize the tagged sentences.
• Constraint Satisfaction: The model generates a condensed, high-quality user profile ($G$) that fits within a predefined token limit ($m$), ensuring the most critical signals are preserved in the compressed version.

3. Key Contributions

• Empirical Discovery: The paper provides preliminary studies proving that (a) LLM attention patterns naturally distinguish between important and unimportant personalization signals, and (b) fine-tuning the marking model significantly enhances this discriminative ability.
• Novel Framework: The introduction of the two-stage "Mark-then-Summarize" pipeline, which uses internal model signals to guide external summarization.
• Efficiency-Performance Balance: A method that achieves high-fidelity compression, maintaining performance close to the "full context" baseline while drastically reducing token counts.

4. Experimental Results

The framework was evaluated on two datasets: MovieLens-1M (movie recommendation) and LaMP-5 (personalized title generation).

• Inference-Only Setting: Attn-GS consistently outperformed all baselines (Truncation, Direct Summary, CoT, Self-Reflection, and Prompt-based identification). It achieved performance within 1.8% of the full context (using 10,000 tokens) while using only 200 tokens—a 50× reduction in token usage.
• Training and Inference Setting: When used to train a generator model ($\Phi_G$), Attn-GS provided much higher quality compressed contexts than baselines, leading to superior downstream task accuracy.
• Ablation Studies:
  • Threshold ($\alpha$): A moderate threshold (0.2–0.4) is optimal.
  • Layers: Middle layers of the LLM are most effective for marking, as early layers focus on syntax and late layers on generation, whereas middle layers capture semantic importance.
• Efficiency: Compared to other baseline methods, Attn-GS required 7× fewer tokens to reach the same accuracy level.

5. Significance

Attn-GS represents a significant step toward efficient, long-term personalization. By moving away from "black-box" summarization and toward "white-box" attention-guided compression, the authors demonstrate that we can use the model's own internal reasoning to solve the context window bottleneck. This approach is highly scalable and directly applicable to real-world production environments where minimizing latency and API costs is critical for user experience.