🤖 machine learning

Capacity and Redundancy Trade-offs in Multi-Task Learning

This paper proposes a Capacity-Redundancy identity to reframe negative transfer in multi-task learning as a consequence of limited shared capacity and weak task redundancy, deriving theoretical conditions for clustered sharing and a gradient-based redundancy proxy, while empirically demonstrating that clustered LoRA significantly reduces interference and outperforms random partitions.

Original authors: Asif Khan

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Asif Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the conductor of a massive orchestra, but instead of violins and drums, your musicians are different prediction tasks: one wants to guess if a movie review is happy or sad, another wants to identify the speaker in a recording, and a third wants to translate a sentence into French. In the world of machine learning, this is called Multi-Task Learning (MTL). The goal is to teach one giant brain to do all these jobs at once, hoping that learning one skill helps it learn the others.

However, there's a catch. Sometimes, trying to do too many things at once makes the brain get confused. This is called negative transfer, where learning a new task actually makes the old tasks worse. To understand why, we need two simple ideas: Capacity and Redundancy. Think of Capacity as the size of the brain's "short-term memory" or the number of notes it can hold in its head at once. Think of Redundancy as how much the tasks have in common. If two tasks are like "detecting a cat" and "detecting a dog," they share a lot of features (ears, fur, tails), so they have high redundancy. If one task is "detecting a cat" and the other is "calculating the price of a stock," they have almost no redundancy; they are strangers.

For years, scientists have wondered: When should we force these tasks to share the same brain space, and when should we give them their own private rooms? If we force unrelated tasks to share a tiny memory, they will fight over space and crash. But if we give them too much space, we waste energy. This paper dives into that exact tug-of-war, trying to find the mathematical rules for when sharing helps and when it hurts.

The Great Brain Heist: Capacity vs. Clutter

The authors of this paper, Asif Khan from Harvard Medical School, propose a new way to look at this problem using a concept they call the Capacity–Redundancy (CR) identity. Imagine the shared part of the AI model as a narrow tunnel (the "bottleneck") through which all information must pass to get to the different tasks.

The paper's main finding is a mathematical rule that acts like a budget constraint: The total amount of useful information the tunnel can carry is limited by its size (Capacity) plus how much the tasks overlap (Redundancy).

Here is the playful part: If your tasks are highly redundant (like the cat and dog detectors), they can "piggyback" on the same bits of information. It's like two friends sharing a single umbrella; they both stay dry without needing two umbrellas. But if the tasks are unrelated (like the cat detector and the stock calculator), they have no redundancy. In this case, the tunnel has to carry two completely different streams of water. If the tunnel is too narrow, the streams crash into each other, causing a flood of confusion. This is negative transfer. The paper proves that if you don't have enough "redundancy" to share the load, you must increase the "capacity" (make the tunnel wider) as you add more tasks, or the performance will inevitably drop.

The "Clustered" Solution: Grouping Friends, Separating Strangers

So, what's the fix? The paper suggests a strategy called Clustered Sharing. Instead of forcing every single task to share the same narrow tunnel, you group the tasks that are "best friends" (high redundancy) together and give them their own shared tunnel. The "strangers" (low redundancy) get their own private tunnels or separate lanes.

The authors derived a precise mathematical condition (Theorem 5.3) that tells you exactly when this grouping works. It's a trade-off: You gain by reducing the "interference" (the fighting between unrelated tasks), but you lose a little bit of "redundancy" (because you aren't sharing as broadly). The paper shows that grouping is only worth it if the reduction in fighting is bigger than the loss in sharing.

To prove this, they looked at a popular technique called LoRA (Low-Rank Adaptation), which is like adding small, lightweight "training wheels" to a massive, frozen AI model. They treated the size of these training wheels (the "rank") as the capacity budget.

The Experiment: They tested this on real-world data, like the GoEmotions dataset (guessing emotions from text) and the GLUE8 benchmark (a mix of language tasks).
The Result: When they used a small budget (low rank), forcing all tasks to share one set of wheels made the model perform poorly. But when they grouped similar tasks and gave each group its own set of wheels, the model got much smarter.
The Proof: They even measured a "residual coupling" number (denoted as $\hat{\Delta}$ ), which is like a "confusion meter." They found that the clustered approach significantly lowered this confusion meter compared to the "share everything" approach.

What the Paper Rules Out (and What It Doesn't)

It's important to know what this paper doesn't say.

It rules out the idea that "more sharing is always better." The paper explicitly argues that blindly sharing parameters across unrelated tasks is a recipe for disaster when capacity is limited.
It rules out the idea that gradient similarity is just a lucky guess. The authors prove mathematically that looking at how the tasks' "gradients" (the directions the model wants to move to learn) align is a valid way to predict if tasks are redundant. If the gradients point in similar directions, the tasks are likely good friends to share with.
It does not claim to have solved the problem forever. The results are based on specific models (Gaussian models for theory, LoRA on BERT for experiments). The authors suggest that while their math holds up, there might be other ways to measure "synergy" (where tasks help each other in complex, non-redundant ways) that this specific "redundancy" metric doesn't capture yet.

The Takeaway for the Curious Teen

Think of this paper as the ultimate guide to organizing a group project. If you have a group of friends who all love the same topic, you can put them in one small room and they'll work great together (high redundancy, shared capacity). But if you throw a group of friends who hate each other, or who are interested in totally different things, into that same tiny room, they'll just argue and get nothing done (negative transfer).

The paper gives us the math to know exactly when to split the group up. It tells us that if you have a limited budget (like a small classroom or a small computer chip), you shouldn't just throw everyone in together. Instead, you should look at who gets along (redundancy), group them, and give each group just enough space to thrive. By doing this, you can squeeze the most performance out of your limited resources, turning a chaotic mess of tasks into a well-orchestrated symphony.

Technical Summary: Capacity and Redundancy Trade-offs in Multi-Task Learning

1. Problem Statement

Multi-Task Learning (MTL) aims to improve efficiency and generalization by training a single model on multiple prediction tasks simultaneously. However, a persistent challenge is negative transfer, where training on additional tasks degrades performance rather than improving it. While often viewed as an optimization artifact, this paper posits that negative transfer is fundamentally a consequence of limited shared capacity and weak task redundancy.

In the context of Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs), specifically using methods like Low-Rank Adaptation (LoRA), a critical design choice arises: should multiple tasks share a single lightweight module to encourage transfer, or should each task receive a private module to avoid interference? Existing hybrid approaches (e.g., task clustering, mixture-of-adapters) lack a unifying theoretical framework to determine when sharing is beneficial versus when specialization is required.

2. Methodology and Theoretical Framework

The paper introduces a Capacity–Redundancy (CR) identity that decomposes the sum of per-task predictive informations into joint predictive information, label redundancy, and a residual coupling term.

2.1 The Capacity–Redundancy (CR) Inequality

The authors model the shared encoder in MTL as a finite-capacity channel with a budget $I(X; Z_s) \leq C_s$ , where $Z_s$ is the shared latent representation. They derive the following inequality:
$\sum_{t=1}^T I(Z_s; Y^{(t)}) \leq C_s + TC(Y^{(1:T)})$
Where:

$I(Z_s; Y^{(t)})$ is the predictive information for task $t$ .
$C_s$ is the shared capacity.
$TC(Y^{(1:T)})$ is the Total Correlation among labels, serving as a measure of label redundancy.

The inequality is derived from an exact identity:
$\sum_{t=1}^T I(Z_s; Y^{(t)}) = I(Z_s; Y^{(1:T)}) + \left( TC(Y^{(1:T)}) - TC(Y^{(1:T)} | Z_s) \right)$
The term $\Delta = TC(Y^{(1:T)} | Z_s)$ represents the residual coupling (interference) remaining after conditioning on the shared representation. If $\Delta > 0$ , the shared representation has not fully factorized the joint task distribution.

2.2 Key Theoretical Results

Impossibility of Maintaining Signal without Scaling: For weakly dependent tasks (low $TC$), adding tasks to a fixed-capacity bottleneck forces the average per-task information to decay as $O(1/T)$ . To maintain a constant per-task signal level, capacity must scale linearly with the number of tasks unless redundancy is high.
Gradient–TC Bridge: In a Gaussian multi-task model, the paper proves that gradient cosine similarity preserves the sign and ordering of label correlations (under matched noise). This formally justifies using gradient similarity as a proxy for redundancy ordering to guide task clustering.
Clustering Gap Decomposition: The authors derive a necessary and sufficient condition for when clustered sharing (sharing within groups, not globally) outperforms global sharing. The gain is decomposed into:
$\text{Gain} = (\text{Interference Reduction}) - (\text{Redundancy Loss})$
Clustered sharing is strictly beneficial if and only if the reduction in residual coupling (interference) exceeds the loss of shared redundancy across cluster boundaries.
Shared–Private Extension: The framework is extended to include task-private features ( $Z_t$ ). The total information bound becomes $C_s + \sum C_t + TC(Y^{(1:T)})$ . Private features are shown to reduce the residual coupling $\Delta$ by absorbing heterogeneity that the shared bottleneck cannot encode.

2.3 Application to LoRA

The paper models LoRA as a capacity-constrained channel where the rank $r$ limits the effective capacity $C_s$ . It predicts that for heterogeneous tasks, a single globally shared LoRA will saturate quickly as rank increases, whereas clustered or private adapters are necessary to prevent interference.

3. Empirical Results

The authors validate their theory through synthetic data and real-world NLP benchmarks using a frozen bert-base-uncased encoder with LoRA adapters.

3.1 Synthetic Validation

CR Inequality Tightness: In a linear-Gaussian model, the bound is shown to be tight when the encoder spans the task subspace.
Residual Coupling Estimation ( $\hat{\Delta}$ ): The authors propose estimating the residual coupling $\Delta$ using validation residual correlations. They demonstrate that this estimator is sensitive to the regularization parameter $\epsilon$ in the log-determinant calculation, identifying an optimal range ( $\epsilon = 10^{-5}$ ) that minimizes bias.

3.2 Real-World Benchmarks

Experiments were conducted on the GoEmotions dataset (12 emotion labels) and the GLUE8 benchmark (8 diverse NLP tasks).

GoEmotions: Clustered routing (grouping tasks by gradient similarity) consistently outperformed global sharing and private-only baselines under matched rank budgets (e.g., at rank 8, Clustered AUROC: 0.821 vs. Shared: 0.708). Crucially, clustered routing resulted in a systematically lower estimated residual coupling $\hat{\Delta}$ , confirming reduced interference.
GLUE8: Similar trends were observed, though improvements were more modest due to the heterogeneous nature of the tasks (syntax vs. sentiment).
Conclusion: Similarity-guided routing effectively reduces residual interference and improves performance when adapter capacity is limited.

4. Key Contributions

Isolation of the Slack Term: The paper identifies $\Delta = TC(Y^{(1:T)} | Z_s)$ as the specific measure of interference left unresolved by a shared representation.
Clustering Gap Decomposition: Provides a necessary and sufficient condition (Theorem 5.3) for when clustered sharing is superior to global sharing, framing the trade-off as interference reduction versus redundancy loss.
Gradient–TC Bridge: Formally justifies gradient cosine similarity as a proxy for redundancy ordering in Gaussian models, linking optimization dynamics to information-theoretic redundancy.
Operational Limits for PEFT: Quantifies the trade-off between adapter rank (capacity) and task subspace dimension, offering a theoretical basis for when to share or specialize parameters in LoRA-based fine-tuning.

5. Significance and Claims

The paper claims to provide a distribution-level converse on achievable predictive information through a shared latent, independent of specific training algorithms or estimators. Its primary significance lies in:

Reframing Negative Transfer: Moving the explanation from optimization artifacts to fundamental limits of capacity and redundancy.
Prescriptive Design Rules: Offering an "if-and-only-if" rule for sharing parameters based on task similarity and redundancy.
Practical Guidance for PEFT: Demonstrating that under strict capacity budgets (low rank), similarity-based clustering is a principled way to mitigate interference, whereas naive global sharing can amplify negative transfer.

The authors note that the CR bound focuses on redundancy and does not capture complementary information (synergy) across tasks, suggesting future work could extend the framework using partial information decomposition.