DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learning

1. Problem Statement

Large Language Models (LLMs) require effective methods to be steered toward specific tasks without the heavy computational cost of fine-tuning. Existing Task Vector approaches generally fall into two categories:

• Weight-space vectors: Require fine-tuning a model for a specific task and subtracting weights from a base model (e.g., Arithmetic Task Vectors). This is computationally expensive and not scalable for dynamic tasks.
• Activation-space vectors: Involve invasive manipulation of internal hidden states during inference, often requiring auxiliary models or complex optimization.

The Gap: Current methods lack a training-free, non-invasive, and scalable solution that can extract task-specific directions directly from a frozen model without altering weights or internal states. Furthermore, existing decoding strategies often rely on auxiliary models or complex contrastive setups.

2. Methodology: DeCoVec

The authors propose DeCoVec (Decoding Space based Task Vector), a framework that constructs task vectors directly in the decoding space (output logit space) by leveraging In-Context Learning (ICL).

Core Concept

Instead of modifying model weights ($\Delta\theta$) or hidden states ($\Delta h$), DeCoVec defines the task vector as the difference in output logit distributions ($\Delta z$) between two contexts:

1. Zero-Shot Context ($C_{zs}$): The model receives only the task instruction and the query (no examples).
2. Few-Shot ICL Context ($C_{icl}$): The model receives the instruction, the query, and a set of $k$ demonstration examples.

The Pipeline

1. Context Construction:
   • A task-specific instruction template is used.
   • A demonstration sampler (Random, KATE, or BM25) retrieves $k$ examples to form the few-shot context.
2. Vector Construction:
   • The model generates logits for the same prefix $y_{1:t}$ under both contexts.
   • Zero-shot logits: $z_t^{zs} = f_\theta(C_{zs}, y_{1:t})$
   • Few-shot logits: $z_t^{icl} = f_\theta(C_{icl}, y_{1:t})$
   • Task Vector ($v_t^T$): Calculated as the difference:
     $$v_t^T = z_t^{icl} - z_t^{zs}$$
   • This vector captures the "distilled" task essence (semantic shift) induced by the examples.
3. Steering Mechanism:
   • During the actual generation of a test sample, the model uses a standard few-shot context ($C_{de}$) to generate base logits ($z_t^{de}$).
   • The task vector is injected into the decoding process:
     $$\tilde{z}_t = z_t^{de} + \lambda \cdot v_t^T$$
   • Here, $\lambda$ is a scaling factor controlling the intensity of the steering.
   • The final token is selected via $\text{argmax}(\tilde{z}_t)$.

3. Key Contributions

• Novel Operational Space: First to define and utilize task vectors strictly within the decoding space (logits), avoiding the need for weight updates or internal state manipulation.
• Training-Free & Non-Invasive: The method requires no gradient updates, fine-tuning, or auxiliary models. It works with any frozen LLM.
• Plug-and-Play Design: The task vector is constructed dynamically during inference based on the specific few-shot context, making it highly flexible.
• Robustness to Noise: The method acts as a "denoising filter," effectively extracting task semantics even when the few-shot examples are suboptimal or noisy.

4. Experimental Results

The authors evaluated DeCoVec on 7 LLMs (ranging from 0.5B to 9B parameters, including Qwen2, Yi, Llama-2/3, and Gemma-2) across three benchmarks:

• TruthfulQA: Knowledge-intensive truthfulness.
• Math-500: Mathematical reasoning.
• AQUA-RAT: Algebraic word problems.

Key Findings:

• Performance Gains: DeCoVec consistently outperformed standard few-shot baselines across all models and tasks.
  • Peak Improvement: Up to +5.50 average accuracy on Qwen2-1.5B.
  • Consistency: Small models (0.5B) benefited as significantly as larger models (9B), proving the method is scale-agnostic.
• Robustness to Ordering: Unlike standard ICL which suffers from "recency bias" (order of examples matters), DeCoVec showed strong robustness to sequential, reverse, and random ordering of demonstrations.
• Error Reduction: Analysis on Math-500 revealed that DeCoVec significantly reduced Logical/Reasoning flaws (LR) and Degeneration (DG) errors, though it slightly increased calculation errors (suggesting it steers high-level logic rather than low-level arithmetic precision).
• Efficiency: The method reduced the average number of output tokens by 10–40 tokens per response compared to baselines, mitigating the "generation degeneration" issue where models ramble.

5. Significance and Implications

• Efficient Steering: DeCoVec offers a lightweight alternative to Parameter-Efficient Fine-Tuning (PEFT) and complex activation interventions, making task adaptation accessible without computational overhead.
• Interpretability: The success of the method suggests that the difference between zero-shot and few-shot logit distributions encodes high-level task semantics, providing a new lens for understanding how LLMs internalize task instructions via ICL.
• Practical Deployment: Since it does not require weight updates or auxiliary models, DeCoVec is immediately deployable in production environments where model weights must remain frozen or where latency/cost constraints prevent heavy fine-tuning.

Limitations

• Hyperparameter Sensitivity: The scaling factor $\lambda$ requires tuning per dataset; it is not yet adaptive to individual input instances.
• Reasoning Models: The impact on models with long "Chain-of-Thought" processes needs further verification to ensure stability across multiple reasoning steps.
• Compositionality: The ability to arithmetically combine different decoding-space vectors (e.g., merging safety and reasoning vectors) is less explored compared to weight-space vectors.