Technical Summary: Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction

1. Problem Statement

As Large Language Model (LLM) systems handle increasingly complex, long-context tasks, they frequently encounter context window limits. The canonical solution is context compaction, where a "compactor" summarizes the conversation history to replace the raw tokens, allowing the session to continue. While current compaction strategies are designed to preserve task continuity (e.g., the current state, next steps, and dominant objectives), they systematically fail to preserve a specific class of user instructions known as Session Constraints (SCs).

SCs are user-issued directives that constrain how a task is performed rather than what the task is (e.g., "confirm with me before any action" or "do not send emails without approval"). Unlike system prompts or durable memory, SCs are scoped to the current session and are often phrased as soft preferences embedded within task-focused turns. The paper identifies a "silent integrity failure": when a compactor drops an SC, the agent continues the task but violates the user's behavioral constraints, potentially leading to unauthorized actions or safety violations.

2. Methodology: The COMPINT Evaluation Suite

To quantify this loss, the authors introduce COMPINT, an evaluation suite designed to measure the integrity of SCs under context compaction across three distinct long-context scenarios:

1. Multi-turn Chat: Using the WildChat dataset (anonymous user-ChatGPT conversations).
2. Agentic Trajectory: Using the Hermes Agent dataset (tool-calling, multi-step reasoning, and coding).
3. Long-Horizon Research: Using the OpenResearcher dataset (autonomous deep research cycles).

Experimental Design

The evaluation constructs synthetic long-context instances (approx. 100K tokens, ~80% of a 128K window) by stitching or selecting trajectories. The methodology involves four orthogonal axes:

• SC Content: A five-category taxonomy of constraints: Action (what the agent does), Information (what data is included/excluded), Process (how the agent reasons), Preference (selection among alternatives), and Output (response formatting).
• Framing: Varying the constraint's Strength (Strict vs. Preferential) and Explicitness (Direct vs. Contextualized).
• Injection Condition: Varying the location (Top, Middle, Bottom of the history) and multiplicity (single vs. repeated) of the SC.
• Environment: The three datasets listed above.

This design yields 750 evaluation instances per compaction condition. The evaluation measures two metrics:

1. Retention: Whether the SC is semantically present in the compacted summary (evaluated via an LLM-as-a-judge).
2. Compliance: Whether the agent behaves according to the SC when queried via a multiple-choice probe on the compacted context.

3. Key Contributions

The paper makes four primary contributions:

1. Formalization of Session Constraints (SCs): A definition and taxonomy of user-issued, session-scoped constraints that are distinct from the main task objective and fragile under compaction.
2. COMPINT Benchmark: A comprehensive evaluation suite covering three long-context scenarios with controlled variables for SC content, framing, and injection.
3. Systematic Evaluation: A large-scale analysis revealing that current compaction methods retain only a small fraction of SCs, with performance varying significantly based on compactor type, context length, and SC framing.
4. SC-Aware Extractor: A proposed mitigation strategy—a plug-and-play module using a Small Language Model (SLM) to extract and maintain a running list of SCs, which is appended to the compacted context.

4. Key Results

Low Retention Rates

Current compaction methods exhibit severe SC loss:

• Non-LLM compactors (e.g., Recent-5 truncation, LLMLingua-2) retain 0% of injected SCs across all datasets.
• LLM-based compactors (using various prompts like Anthropic's or OpenClaw's pi-mono) perform better but still fail significantly, retaining an average of only 17% of SCs.
• Even high-performing commercial models (e.g., GPT-5.4-mini) show retention rates ranging from 6.7% to 98% depending on the dataset and prompt, with no consistent reliability.
• Effect Retention (normalized behavioral preservation) is low for all tested compactors, indicating that most degrade SC compliance relative to running the task without compaction.

Factors Influencing Retention

• Context Length: Retention rates decline as context length increases. Compactors perform better on shorter contexts (e.g., ~90% retention at 10K tokens vs. much lower at 100K).
• Injection Location: Retention is generally higher when SCs are injected closer to the compaction prompt (Bottom of the context), though this varies by compactor type.
• Framing: Explicit and strict phrasing ("This is an important constraint") improves retention slightly for LLM-based compactors, but the gains are marginal (e.g., +1.3% when combining both).
• SC Type: Process constraints are consistently the least retained, while Preference constraints are often best retained by LLM-based compactors, though no type achieves reliable retention (>36% max).

The SC-Aware Extractor Solution

The authors propose an SC-aware extractor that runs alongside the compactor. It uses a small language model (Qwen3.5-9B) to scan user turns, identify SCs, and maintain a persistent registry. This registry is appended to the compacted summary.

• Performance: This approach achieves >90% retention across all three datasets (90.3% on WildChat, 95.6% on Hermes Agent, 95.1% on OpenResearcher).
• Efficiency: It is highly efficient, processing 100K contexts in as little as 0.03 seconds (for single-turn research) and up to 12.93 seconds (for multi-turn chat), without modifying the compactor or the main LLM.

5. Significance and Claims

The paper argues that the loss of Session Constraints is a systematic failure of current compaction architectures, not an artifact of a specific prompt or dataset. The authors claim that:

• Architectural Separation is Required: Simply improving compactor prompts (e.g., adding explicit instructions to "preserve constraints") yields marginal improvements (retention remains <40%) and does not close the gap. The solution requires a separate, strategy-layer intervention (the extractor).
• Safety Implications: The failure to retain SCs poses direct safety risks, as agents may execute actions (e.g., sending emails, making tool calls) that users explicitly forbade for the session.
• Generalizability: The findings hold across open-source and proprietary models, different prompt styles, and diverse interaction environments (chat, agentic, research).

The work concludes that while context compaction is necessary for long-horizon tasks, current implementations compromise user-defined behavioral boundaries. The proposed SC-aware extractor offers a practical, training-free mechanism to restore these constraints without altering the core compaction pipeline.