💬 NLP

Efficient Decentralized Multi-task Dataset Valuation via Model Merging

This paper proposes DMVM, a novel framework that enables efficient, privacy-preserving, and decentralized valuation of multi-task datasets by leveraging task arithmetic and secure aggregation to infer contributions directly from model parameters without requiring data sharing or retraining.

Original authors: Mohammadsajad Alipour, Mohammad Mohammadi Amiri

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohammadsajad Alipour, Mohammad Mohammadi Amiri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a bustling digital marketplace where people want to buy data to train smart computer programs (AI). But here's the catch: the sellers are protective. They don't want to show their raw data (like private medical records or personal photos) before getting paid, and they certainly don't want to wait for a long time while the buyer tries out every possible combination of data to see what works.

This paper introduces a new way to figure out how much each seller's data is worth, called DMVM. It solves three big problems: it's fast, it respects privacy, and it works when the buyer needs to solve many different problems at once (like teaching an AI to recognize cats, translate languages, and summarize news all at the same time).

Here is how it works, using some everyday analogies:

1. The Problem: The "Taste Test" Nightmare

Imagine you are a chef (the buyer) trying to make a perfect stew. You have 10 different suppliers (sellers), each offering a unique spice blend. To know which spice is the best, the old way of doing things was to:

Buy a tiny bit of every spice.
Cook a separate pot of stew for every possible combination of spices (Spice A alone, A+B, A+C, A+B+C, etc.).
Taste them all to see which combination tastes best.

The Catch: This takes forever (it's computationally expensive) and requires you to have all the spices in your kitchen at once (which violates privacy if the spices are secret recipes).

2. The Solution: "Blending the Blenders" (Model Merging)

The authors realized you don't need to cook every single pot of stew to know what the spices do. Instead, you can look at the "flavor profiles" of the individual spice blends the suppliers already made.

In the AI world, instead of retraining the model with every data combination, the sellers first train their own small models using their private data. Think of these models as "flavor profiles."

The Magic Trick: The buyer takes these pre-made "flavor profiles" and mathematically blends them together (a process called Model Merging).
The Analogy: It's like taking the "recipe adjustments" a chef made for a tomato sauce and a garlic sauce and simply adding those adjustments together to guess what a "tomato-garlic sauce" would taste like, without actually cooking the new sauce.

This allows the buyer to instantly see how well a group of sellers would work together, without ever seeing their raw data or waiting for a new model to be trained.

3. The Privacy Shield: The "Blindfolded Mixer"

Even if you blend the "flavor profiles," a sneaky buyer might try to reverse-engineer the recipe to steal a specific seller's secret spice blend.

To stop this, the paper introduces a Secure Protocol:

The Analogy: Imagine the sellers put their spice adjustments into a locked box. They add some random "noise" (like a bunch of random sprinkles) to the box so the buyer can't see the original spice.
The buyer mixes all the boxes together. The random sprinkles cancel each other out, leaving only the true combined flavor.
The Result: The buyer gets the final blended result but cannot peek inside to see any single seller's original recipe. Even if the buyer tries to subtract one blended box from another, the "random sprinkles" make the result look like gibberish, protecting the secrets.

4. Why This Matters for "Multi-Task" Learning

Most old methods only looked at one task at a time (e.g., "How good is this data for recognizing cats?"). But modern AI needs to do many things at once.

The Analogy: Imagine a student who needs to study for Math, History, and Art exams. A data seller might be great at helping with Math but terrible for Art.
DMVM's Role: It calculates the value of a seller's data based on how it helps the student do well in all subjects simultaneously. It figures out the "marginal utility"—how much extra value that specific data adds to the whole study plan.

The Bottom Line

The paper claims that DMVM is:

Fast: It skips the slow process of retraining models from scratch.
Private: Sellers never have to share their raw data or let the buyer see their individual models.
Accurate: In their tests (using image recognition and language tasks), the values they calculated were very close to the "perfect" values you would get if you could have retrained everything (which is usually impossible in real life).

In short, they found a way to fairly pay data sellers in a crowded, private marketplace by mathematically "mixing" their contributions without ever needing to see the ingredients.

Technical Summary: Efficient Decentralized Multi-task Dataset Valuation via Model Merging

Problem Statement

The paper addresses the challenge of dataset valuation in decentralized data marketplaces, specifically within multi-task learning contexts. Existing valuation methods face three critical limitations:

Computational Intractability: Traditional approaches, such as Shapley-based methods, require retraining models on exponentially growing numbers of dataset subsets ( $2^n$ ), which is infeasible for large-scale markets or large models.
Privacy Constraints: Most methods assume centralized access to raw data for valuation. In realistic marketplaces involving sensitive data (e.g., medical or personal data), sellers are unwilling to share raw datasets with a central coordinator or buyers prior to purchase.
Single-Task Focus: Current literature predominantly evaluates data utility for a single predefined task. However, modern buyers often seek to optimize performance across multiple downstream tasks simultaneously. Valuing data in a multi-task setting is more complex because a dataset's utility varies across tasks and cannot be captured by a single metric.

The core problem is how to accurately estimate the marginal contribution of a seller's dataset to a buyer's multi-task model without accessing raw data, without a trusted central coordinator, and without the prohibitive cost of repeated retraining.

Methodology: DMVM Framework

The authors propose DMVM (Decentralized Multi-task Valuation via Model Merging), a framework that leverages model merging and task arithmetic to approximate dataset contributions.

1. Model Merging as a Utility Proxy

Instead of training a model on combined datasets (which requires data access), DMVM approximates the utility of a coalition of sellers by merging their independently fine-tuned models.

Process: All sellers start from a shared pre-trained model $\theta_{pre}$ . Each seller $i$ fine-tunes this model on their private dataset $D_i$ to obtain a task vector $v_i = \theta_i - \theta_{pre}$ .
Merging: To estimate the utility of a subset of sellers $S$ , the buyer (or a protocol) computes a merged model $\theta_{merge}^S = \theta_{pre} + \alpha \sum_{i \in S} v_i$ , where $\alpha$ is a scaling factor.
Utility Estimation: The utility $\hat{u}(S)$ is calculated by evaluating the merged model on the buyer's target validation tasks. This avoids retraining; the utility is derived directly from the arithmetic combination of model parameters.

2. Privacy-Aware Secure Aggregation

To prevent the leakage of individual model parameters (which could allow reconstruction of private data or models), DMVM introduces a secure protocol:

Secure Aggregation: Sellers use random mask matrices to hide their individual contributions. They send masked versions of their task vectors to a buyer, who reconstructs only the sum (the merged model) without seeing individual $\theta_i$ .
Unmergeable Transformation: To prevent a malicious buyer from recovering a specific seller's model by subtracting merged models of overlapping subsets (e.g., $\theta_{\{1,2,3\}} - \theta_{\{1,2\}}$ ), the protocol applies a transformation (e.g., random permutation matrices for MLPs or scalar scaling for linear layers). This preserves the inference behavior of the merged model but destroys the linear algebraic structure required to isolate individual weights.

3. Theoretical Guarantees

The paper derives error bounds characterizing the approximation quality of DMVM:

Approximation Error: Theorem 4.1 establishes that the difference between the true Shapley value (ground truth) and the DMVM estimate is bounded by terms related to the learning rate, the number of optimization steps, and the Lipschitz constant of the loss function. The error grows quadratically with the number of sellers $n$ but can be controlled by setting $\alpha = 1/n$ .
Sampling Error: Theorem 4.2 shows that the deterministic $2^n$ summation can be replaced by Monte Carlo sampling ( $m$ samples), introducing a statistical error that decays as $O(1/\sqrt{m})$ .
Privacy Error: Theorem 4.3 bounds the deviation caused by excluding subsets of size 1 (to ensure no single model is shared), showing the error remains bounded.

Key Contributions

Efficient and Decentralized Framework: DMVM enables multi-task dataset valuation without retraining or centralized data access, operating fully in a decentralized manner without a trusted broker.
Privacy-Aware Protocol: The integration of secure aggregation and unmergeable transformations ensures that neither raw data nor individual model parameters are revealed during the valuation process.
Theoretical Guarantees: The authors provide formal error bounds linking the model merging approximation to ground-truth multi-task marginal contributions.
Empirical Validation: The framework is validated on Computer Vision (CV) and Natural Language Processing (NLP) benchmarks, demonstrating stable performance across varying numbers of sellers and target tasks.

Experimental Results

The authors evaluated DMVM against baselines including Dataset Shapley (Ground Truth), MC-Dataset-Shapley, and DU-Shapley.

Accuracy: In NLP tasks, DMVM consistently outperformed the DU-Shapley baseline in both ranking correlation (Kendall's $\tau$ and Spearman's $\rho$ ) and Mean Squared Error (MSE) relative to the ground truth. In Vision tasks, DMVM achieved strong agreement (Kendall's $\tau \approx 0.78-0.81$ ) while operating under stricter privacy constraints than the baselines.
Efficiency: As shown in Figure 2, DMVM significantly reduces computational time compared to retraining-based methods. It replaces expensive training loops with model merging and inference. The Monte Carlo variant (MC-DMVM) further trades a small amount of accuracy for substantial speed gains.
Robustness: The method remained stable across different numbers of target tasks ( $|T|$ ) and seller counts ( $n$ ).

Significance and Claims

The paper claims to bridge the gap between dataset valuation and model merging, establishing a scalable, privacy-aware, and theoretically grounded foundation for multi-task marketplaces.

Paradigm Shift: It reframes valuation from a "repeated retraining" problem to a "model composition" problem.
Practicality: By eliminating the need for raw data access, DMVM makes dataset valuation feasible in real-world scenarios where data privacy is paramount (e.g., healthcare, finance).
Multi-Task Alignment: Unlike previous methods focused on single tasks, DMVM explicitly aligns with multi-task generalization behavior, providing a more accurate valuation for buyers with complex, multi-objective goals.

The authors conclude that model merging serves not only as a tool for multi-task adaptation but also as an efficient mechanism for decentralized market valuation, offering a practical alternative to computationally prohibitive and privacy-invasive traditional methods.