💬 NLP

Context Parametrization with Compositional Adapters

The paper introduces CompAs, a meta-learning framework that translates multiple context chunks into algebraically composable adapter parameters, offering a more efficient, stable, and reversible alternative to in-context learning and supervised fine-tuning for scaling large language models.

Original authors: Josip Jukić, Martin Tutek, Jan Šnajder

Published 2026-01-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Josip Jukić, Martin Tutek, Jan Šnajder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overstuffed Backpack"

Imagine a Large Language Model (LLM) is a brilliant student trying to answer a question. To do their best, they need to read some background information (like instructions, examples, or facts).

Currently, there are two main ways to give this student that information:

In-Context Learning (ICL): You paste all the background info directly into the student's reading material.
- The Problem: If you give them 100 pages of examples, the student gets overwhelmed. They start forgetting the beginning of the text, get confused, and it takes a long time to read through it all. It's like trying to carry a backpack that gets heavier and heavier with every new book you add.
Fine-Tuning (SFT): You train the student specifically for that one task.
- The Problem: This is like hiring a new tutor for every single subject. It's expensive, slow, and you can't easily swap tutors if the task changes.

The Solution: COMPAS (The "Magic Notebooks")

The authors introduce COMPAS, a new way to teach the model. Instead of pasting pages of text into the prompt, COMPAS turns that text into a tiny, invisible "instruction card" (called an adapter) that is attached directly to the model's brain.

Think of it like this:

Old Way (ICL): You hand the student a stack of 50 reference books every time they take a test.
COMPAS Way: You read those 50 books once, distill the most important lessons into a single, tiny magic notebook, and tape that notebook to the student's forehead. Now, the student doesn't need to read the books again; they just look at the notebook.

The Secret Sauce: "Compositional" (The Lego Blocks)

The real magic of COMPAS isn't just making one notebook; it's about making many small notebooks that can be snapped together.

Imagine you have a set of Lego bricks.

Context 1 (e.g., a math example) becomes Brick A.
Context 2 (e.g., a science example) becomes Brick B.
Context 3 (e.g., a history example) becomes Brick C.

In the old world, if you wanted to use all three, you had to glue them into one giant, messy block of text. With COMPAS, you can just snap Brick A, Brick B, and Brick C together mathematically.

The paper proves that if you add these "adapter bricks" together, the model behaves exactly as if it had read all the original texts combined. This means:

Speed: The model doesn't have to re-read the text every time.
Stability: It doesn't get confused by long lists of information.
Flexibility: You can swap out "Brick A" for a different math example without retraining the whole system.

How It Works (The Teacher-Student Game)

To teach the model how to make these magic notebooks, the authors use a "Teacher-Student" setup:

The Teacher: A powerful model that reads the entire long text and gives the correct answer.
The Student: A model that only sees the question and the "magic notebook" (the adapter).
The Goal: The Student tries to guess the answer using only the notebook. The system adjusts the notebook until the Student matches the Teacher's performance.

They also add a special rule: The "Addition Rule." The system forces the generator to ensure that if you have two separate texts, the notebook for "Text A + Text B" is exactly the same as "Notebook A + Notebook B." This ensures the Lego blocks snap together perfectly.

Why This Matters (The Results)

The paper tested this on various tasks (like answering multiple-choice questions and finding facts in long documents). They found that:

It's Smarter: When you give the model lots of examples (like 16 or more), COMPAS performs better than just pasting them all in the prompt.
It's More Stable: As the amount of information grows, the model doesn't get "lost in the middle" of the text.
It's Reversible: The "magic notebook" is so faithful to the original text that, if you wanted to, you could actually decode the notebook back into the original text. This is a safety feature, ensuring the model isn't hiding secret data.

Summary

COMPAS is a method that turns long, messy instructions and examples into compact, mathematical "instruction cards." These cards can be added together like Lego blocks. This allows AI models to handle huge amounts of information quickly and accurately without getting overwhelmed by reading long texts every single time.

Technical Summary: Context Parametrization with Compositional Adapters (COMPAS)

1. Problem Statement

Large Language Models (LLMs) typically adapt to new tasks via In-Context Learning (ICL) or Supervised Fine-Tuning (SFT). However, both approaches face significant limitations when scaling to multiple demonstrations or retrieved passages:

ICL Inefficiency: As the number of context tokens (instructions, demonstrations, evidence) increases, ICL becomes computationally expensive due to linearly scaling inference costs. Furthermore, long contexts destabilize attention mechanisms, leading to performance degradation (the "lost in the middle" phenomenon).
SFT Rigidity: SFT requires training a separate model or adapter for each specific task or context, incurring high training overhead and lacking the flexibility to dynamically integrate new information without retraining.
Context Window Limits: When input exceeds the model's context window, neither ICL nor standard SFT can effectively utilize the full information.

While prior work has explored generating adapters from a single context, it has largely overlooked the necessity of integrating multiple chunks of information. Existing methods lack a principled mechanism to compose adapters algebraically to represent concatenated contexts, failing to capture how LLMs combine multiple sources of information.

2. Methodology: COMPAS

The authors introduce COMPAS (Compositional Adapters), a meta-learning framework that translates context into adapter parameters with a compositional structure, allowing them to be merged algebraically.

2.1. Theoretical Foundation

The framework is built on a teacher-student setup where the goal is to ensure that adapters generated from individual contexts can be composed in parameter space to replicate the effect of context concatenation in input space.

Teacher Model ( $f_T$ ): Processes the concatenated input $[c_1; \dots; c_k; q]$ .
Student Model ( $f_S$ ): Processes only the query $q$ using base parameters $\theta$ modified by the sum of adapters: $\theta \oplus \sum \phi_i$ .
Generator ( $G$ ): A network that maps a context $c$ to adapter parameters $\phi = G(c)$ .

The core requirement is that the generator acts as a monoid homomorphism from the free monoid of contexts (concatenation) to the additive monoid of adapter parameters (addition). Ideally, $G([c_1; c_2]) = G(c_1) + G(c_2)$ . The paper establishes a Compositionality Bound (Theorem 2.1), proving that the error between the student's output (sum of adapters) and the teacher's output (concatenated context) is bounded by the generator's additivity error and the student-teacher misfit, scaled by the Lipschitz constant of the student.

2.2. Architecture

Generator: Augments the base LM with trainable components: a LoRA adapter inserted into the LM and a linear bottleneck followed by projection trunks. It processes a context $c$ to produce a pooled hidden representation, which is projected into a compact latent space $z(c)$ . This latent vector is then mapped via linear projections to generate LoRA matrices ( $A$ and $B$ ) for all target modules in the student model.
Student: A standard LLM with LoRA adapters. During inference, it receives only the query and the sum of adapters generated for the relevant contexts.

2.3. Training Objectives

The training loss ( $L_{COMPAS}$ ) is a weighted sum of three components designed to enforce compositionality and faithfulness:

Student-Teacher Alignment ( $L_{ST}$ ): Minimizes the KL divergence between the student's distribution (using summed adapters) and the teacher's distribution (using concatenated contexts). This is applied for both single contexts ( $k=1$ ) and concatenated contexts ( $k=2$ ) to enforce compositional consistency.
Additivity Regularization ( $L_{ADD}$ ): Penalizes the discrepancy between the adapter generated for a concatenated context $G([c_1; c_2])$ and the sum of individual adapters $G(c_1) + G(c_2)$ . This directly minimizes the generator additivity error $\eta$ .
Reconstruction ( $L_{RECON}$ ): An auxiliary objective where the student must autoregressively reconstruct the original context tokens from the adapter state using a special [RECON] token. This ensures the adapters faithfully encode the input information and prevents trivial solutions.

3. Key Contributions

COMPAS Framework: A teacher-student meta-learning framework that encodes context into compositional adapters, enabling algebraic combination in parameter space.
Theoretical Guarantees: The establishment of theoretical conditions (Theorem 2.1) under which parameter-space addition provably approximates the behavior of context concatenation, decomposing the error into interpretable sources.
Empirical Validation: Demonstration that COMPAS outperforms standard ICL, fine-tuning, and existing generator-based methods (like GenAda and WILDA), particularly when scaling to large numbers of demonstrations or retrieved passages.

4. Experimental Results

The authors evaluated COMPAS on multiple-choice (MMLU, ARC-Challenge) and extractive QA (HotpotQA) tasks using LLaMA-3.1 (8B), LLaMA-3.2 (3B), and Qwen-2.5 (7B).

Performance vs. ICL: COMPAS consistently outperformed standard ICL across all models and shot counts. The gains were most pronounced with higher shot counts (e.g., 12–16 demonstrations), where ICL performance often degrades due to context length.
Comparison with SFT and Generators: COMPAS matched or exceeded the performance of WILDA (which fine-tunes a specific adapter per context) while maintaining the efficiency of a single generator. It also significantly outperformed GenAda, which lacks the explicit linear bottleneck and additivity constraints.
Scaling: In experiments with 16 demonstrations, COMPAS achieved higher accuracy than ICL and GenAda. For example, on MMLU with LLaMA 8B, COMPAS (4×4 composition) reached 72.2% accuracy compared to ICL's 64.2%.
Robustness: COMPAS demonstrated superior robustness to noisy, contradictory, or off-topic contexts compared to ICL, suggesting that compressing context into parameters mitigates the impact of spurious signals.
Efficiency: By decoupling inference cost from context length, COMPAS achieved significant speedups (up to 4.1× at 16-shot) and reduced peak memory usage compared to ICL, which scales linearly with context size.
Reconstruction: The framework maintained high reconstruction fidelity (token-level F1 > 85% for single units), confirming that the adapter parameters retain the semantic content of the original context.

5. Significance and Claims

The paper claims that COMPAS establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment. Its significance lies in:

Efficiency: It reduces inference costs and memory footprint by replacing long prompt tokens with compact, reusable adapter parameters.
Stability: It mitigates the instability and performance degradation associated with long-context processing in transformers.
Scalability: It offers a principled solution for handling inputs that exceed the model's context window by generating and composing adapters independently.
Safety and Interpretability: The reversible encoding of information into adapters facilitates the recovery of original input contexts, which is crucial for safety and auditing.

The authors position this work not as a replacement for all adaptation strategies, but as a scalable mechanism for integrating external evidence and multiple demonstrations without the computational penalties of traditional prompting or the rigidity of per-task fine-tuning.