💬 NLP

Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer

This paper introduces "Task Prompt Vectors," a method that leverages arithmetic operations on soft-prompts derived from task-specific differences to enable effective multi-task initialization and transfer in prompt tuning without requiring full retraining.

Original authors: Robert Belanec, Simon Ostermann, Ivan Srba, Maria Bielikova

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Robert Belanec, Simon Ostermann, Ivan Srba, Maria Bielikova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart library (a Large Language Model) that knows how to write, answer questions, and solve problems. However, teaching this library a new specific skill usually requires a massive, expensive renovation of the entire building.

Prompt Tuning is like a smarter, cheaper way to teach the library. Instead of rebuilding the whole library, you just add a small, custom "instruction card" (a soft prompt) to the front of every book the library reads. You only tweak this small card to teach the library a new task, like "write a poem" or "solve math problems."

The problem with current methods is that if you want the library to do two things at once (like write poems and solve math), you usually have to train a brand-new instruction card from scratch for that specific combination. It's like having to write a new manual every time you want to combine two skills.

This paper introduces a new tool called Task Prompt Vectors. Here is how it works, using simple analogies:

1. The "Direction" Analogy

Think of the library's instruction card as a compass needle pointing in a specific direction.

Random Start: When you first get a blank instruction card, the needle is pointing in a random direction (like a compass spinning wildly).
Training: When you train the card to solve "Math Problems," you gently push that needle until it points steadily toward "Math."
The Vector: The paper defines a Task Prompt Vector as simply the difference between where the needle started (random) and where it ended up (Math). It's not the final position; it's the movement or the push required to get there.

2. The Magic of "Arithmetic"

The most exciting discovery in the paper is that you can do math with these "pushes."

Imagine you have a "Math Push" and a "Poem Push."
Instead of training a new card from scratch, you can simply add the "Math Push" and the "Poem Push" together.
The result is a new instruction card that points in a direction that handles both math and poetry.

It's like having two different recipes for a cake. Instead of baking a whole new cake to combine chocolate and vanilla flavors, you just mix the "chocolate flavoring" and "vanilla flavoring" together and add them to a fresh batter.

3. Why This is a Big Deal

The authors tested this idea on 19 different tasks (like understanding sentences, classifying topics, and solving math). They found three key things:

It Doesn't Matter Where You Start: Usually, if you start a compass in a different random spot, it might end up pointing slightly differently. But the paper shows that these "Task Prompt Vectors" are so robust that it doesn't matter which random starting point you use. The "push" is the same. This means you can take a "Math Push" created by one person and apply it to someone else's blank card, and it will still work.
Similar Tasks Mix Well: If you add a "Math Push" to a "Science Push," it works great because they are similar. If you try to add a "Math Push" to a "Cooking Push," it might get messy. The paper shows that the vectors naturally reflect how similar the tasks are.
It Saves Time and Money: Because you can just "add" these pre-made vectors together, you don't need to retrain the model from scratch. This is especially helpful when you have very little data (like having only 5 examples of a task instead of thousands). In these "low-resource" situations, using these pre-mixed vectors often works better than other current methods.

Summary

The paper proposes a way to treat AI instructions like building blocks. Instead of building a new house every time you want a different room, you create "blueprints" (vectors) for specific rooms. You can then snap these blueprints together to create a house with multiple rooms instantly, without needing to hire a new construction crew (retraining the model).

The authors call this Task Prompt Vectors, and they show it makes AI more flexible, efficient, and easier to combine for multiple tasks at once.

Technical Summary: Task Prompt Vectors

Problem Statement

Prompt tuning is a parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs) where only a small set of continuous soft prompt weights are trained while the pre-trained model remains frozen. However, existing soft prompt-based methods often lack sufficient multi-task modularity. Typically, training a soft prompt for a new task requires a full or partial retraining process, and combining prompts for multiple tasks often degrades performance or necessitates further fine-tuning. While "task vectors" (arithmetic operations on full model weights) have shown promise in computer vision and older NLP models for achieving multi-task performance, this concept has not been effectively extended to the domain of soft prompt-based tuning. Furthermore, existing PEFT weight interpolation methods often depend heavily on specific random initializations, limiting their reusability.

Methodology

The authors introduce Task Prompt Vectors (TPVs), a novel concept that adapts the principles of task vector arithmetic to the soft prompt space.

Definition

A task prompt vector ( $\tau_{P}^{t}$ ) is defined as the element-wise difference between the weights of a soft prompt after fine-tuning on a specific task ( $\theta_{P_{ft}}^{t}$ ) and its random initialization ( $\theta_{P_{pre}}$ ):
$\tau_{P}^{t} = \theta_{P_{ft}}^{t} - \theta_{P_{pre}}$
This vector represents a direction in the embedding space that moves the prompt from a random state to a task-specific subspace.

Arithmetic Operations

The core mechanism involves prompt arithmetics, specifically the addition of task prompt vectors to combine knowledge from different tasks:
$\tau_{P_{new}} = \tau_{P_{a}} + \tau_{P_{b}}$
This combined vector can be applied to a new soft prompt initialization ( $\theta_{P}$ ) to initialize it for a target task without further training:
$\theta_{P_{new}} = \theta_{P} + \lambda \tau_{P_{new}}$
where $\lambda$ is a rescaling factor.

Experimental Setup

The authors evaluated TPVs on 19 datasets covering 8 task types (including NLI, topic classification, sentiment analysis, and math problem solving) using T5-base, LLaMa-3.1-8B-Instruct, and DeepSeek-LLM-7B-Chat models. Key experimental goals included:

Initialization Independence: Testing if TPVs work regardless of the random initialization of the target soft prompt.
Vector Similarity: Analyzing cosine similarities between vectors from related and unrelated tasks.
Multi-Task Transfer: Evaluating the performance of combined vectors on source tasks and as initializations for zero-shot and few-shot learning on target tasks.

Key Contributions and Findings

1. Random Initialization Independence

The study demonstrates that task prompt vectors are largely independent of the random initialization of the soft prompt. When a TPV trained on one random initialization is applied to a different random initialization, the downstream performance remains comparable to the original setup for the majority of tasks. This property is crucial as it allows TPVs to be shared and reused across different training runs without requiring the same seed.

2. Task Modularity and Vector Arithmetic

The authors show that adding task prompt vectors from similar tasks (e.g., MNLI + RTE) maintains or slightly improves single-task performance on both tasks. This suggests that TPVs can effectively encode task-specific knowledge in a modular way. The cosine similarity of TPVs was found to be a better predictor of transferability than the similarity of the raw task prompts themselves.

3. Low-Resource Initialization

In zero-shot and few-shot settings (0–100 shots), initializing prompt tuning with a combination of pre-trained TPVs often outperforms or matches state-of-the-art baselines like SPoT (Soft Prompt Transfer) and ATTEMPT. Specifically:

TPV combinations performed better than single-task source prompts on SciTail and IMDB.
They matched SPoT performance on AG News and Yahoo Answers.
They significantly outperformed ATTEMPT in several scenarios, particularly when the number of pre-trained source prompts was limited.

4. Comparison with Existing Methods

Unlike SPoT, which requires re-training the source prompt when the set of source tasks changes, and ATTEMPT, which depends heavily on the quality and quantity of source prompts, TPVs offer high modularity and source prompt independence. This allows for the creation of a "vector hub" where pre-trained vectors can be reused for new tasks without additional training.

Significance and Claims

The paper claims that Task Prompt Vectors provide a computationally effective alternative to state-of-the-art multi-task solutions. By leveraging simple arithmetic operations on pre-trained vectors, the method achieves high multi-task modularity without the computational cost of retraining.

The authors posit that the success of this method suggests the optimal subspace for soft prompts may have a convex shape, allowing linear approximations (vector addition) to work effectively even in non-linear spaces. Furthermore, the independence from random initialization enables the potential for a community-driven repository of task vectors.

Limitations noted by the authors:

Experiments were conducted primarily on monolingual (English) models and datasets.
The study focused on T5-base as a representative model, though results were validated on larger LLaMa and DeepSeek models.
Theoretical analysis of the geometric properties of the soft prompt space is left for future work.
The method currently focuses on NLU and NLG tasks but has not been extensively tested on multilingual settings.

In conclusion, the work establishes that task prompt vectors can be used to initialize prompt tuning effectively in low-resource settings, offering a robust, modular, and training-free approach to multi-task adaptation.