Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

1. Problem Statement

Multi-Task Learning (MTL) aims to train a single model to solve multiple tasks simultaneously, improving generalization and efficiency. However, current Parameter-Efficient Fine-Tuning (PEFT) methods (e.g., LoRA, Adapters) are largely designed for single-task adaptation.

• The Conflict: Existing PEFT-MTL approaches often rely on task-specific modules (e.g., separate adapters or decoders for each task) to avoid negative transfer. While this prevents interference, it fails to achieve true parameter sharing, leading to redundant computation and memory usage as the number of tasks increases.
• The Challenge: How can a model retain the efficiency of parameter sharing while allowing shared weights to behave in a task-specific manner without duplicating parameters?

2. Methodology: Free Sinewich

The authors propose Free Sinewich, a framework inspired by the biological mechanism of oscillatory multiplexing in the brain (where the same neurons perform different functions at different frequencies). The core idea is to reuse a single set of low-rank parameters for all tasks by modulating them via frequency switching.

Key Components:

1. Sine-AWB (Sinewich) Layer:

   • Base Construction: Instead of standard LoRA ($M = AB^T$), the method integrates a convolutional prior into the low-rank decomposition. It fuses a linear projection ($A$), a spatial convolution ($W$), and another linear projection ($B^T$) into a single equivalent kernel $M_{AWB} = A W B^T$. This preserves spatial priors crucial for dense prediction tasks.
   • Frequency Switching: A task-specific matrix $M_t$ is generated by applying a sinusoidal transformation to the shared base matrix:
     $$M_t = \sin(\omega_t \cdot M_{AWB})$$
     Here, $\omega_t$ is a task-dependent frequency coefficient. This non-linear transformation effectively increases the rank of the low-rank adapter without adding new trainable parameters, allowing the shared base to "vibrate" into different task-specific forms.

2. Lightweight Clock Net (LCN):

   • A small, shared MLP that takes task tokens as input and outputs the scalar frequency $\omega_t$ for each task.
   • The output is bounded (using tanh and learnable scale/offset) to stabilize training. It acts as a "translator" mapping task identity to a specific oscillation frequency.

3. Low-Pass Filtering:

   • The sine transformation can introduce high-frequency noise. To ensure stability and preserve structural details, a Gaussian low-pass filter is applied to the transformed matrix $M_t$ before it is used as a convolutional kernel.

4. Shared Decoder Architecture:

   • Unlike previous methods that use independent decoders for each task, Free Sinewich employs a shared decoder group. The main convolutional layers in the decoder are replaced by the frequency-switchable Sine-AWB operators. This ensures that the decoder weights are also shared and modulated by frequency, drastically reducing the parameter count.

3. Key Contributions

• Frequency-Switching Mechanism: A novel approach to PEFT-MTL that enables true parameter reuse across tasks by modulating a shared low-rank base with task-specific sine frequencies, rather than duplicating parameters.
• Sine-AWB Layer: A hybrid module combining low-rank adaptation with convolutional priors, enhanced by a non-linear sine transformation to boost effective rank and expressivity.
• Lightweight Clock Net: A minimal overhead component that dynamically assigns frequencies to tasks, stabilizing the modulation process.
• Theoretical Insight: The paper demonstrates theoretically that sine modulation enhances the effective rank of low-rank adapters, while frequency separation decorrelates weights between tasks, mitigating negative transfer.

4. Experimental Results

The method was evaluated on two standard dense prediction benchmarks: PASCAL-Context and NYUDv2, using a Swin Transformer Tiny backbone.

• Performance-Efficiency Trade-off:
  • On PASCAL-Context, Free Sinewich achieved a +5.39% average relative improvement ($\Delta_m$) over single-task fine-tuning baselines with only 6.53M trainable parameters.
  • It outperformed the previous state-of-the-art (TADFormer) which required more parameters (7.38M) for a lower improvement (+4.24%).
  • Notably, Free Sinewich with a lower rank ($r=32$) outperformed TADFormer with a higher rank ($r=64$), highlighting the efficiency of the frequency-switching mechanism.
• NYUDv2 Results: The method achieved near-zero $\Delta_m$ (-0.52) compared to full fine-tuning, matching full fine-tuning performance with a fraction of the parameters.
• Ablation Studies:
  • Removing the sine transformation caused the largest performance drop, confirming it as the primary driver of gains.
  • Removing the Clock Net or Low-pass filter also degraded performance, proving their role in stability and noise suppression.
  • Parameter Sharing: Variants that removed cross-task sharing (Independent Base or Independent Decoder) resulted in lower performance and higher parameter counts, validating the necessity of the shared-frequency approach.

5. Significance

• Paradigm Shift: Free Sinewich moves beyond "pseudo-sharing" (where tasks have separate paths) to "true sharing" (where the same weights perform different functions based on frequency). This mimics biological efficiency more closely than current deep learning approaches.
• Scalability: By decoupling task specialization from parameter count, the framework scales efficiently to many tasks without linearly increasing memory or compute requirements.
• Dense Prediction: The integration of convolutional priors (AWB) makes the method particularly effective for dense prediction tasks (segmentation, depth, normals) where spatial context is critical, addressing a gap in standard LoRA-based MTL methods.

In conclusion, Free Sinewich offers a compact, scalable, and highly effective paradigm for multi-task learning, demonstrating that frequency-based modulation can unlock the full potential of parameter-efficient fine-tuning.