💻 computer science

Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing

The paper introduces ParaX, a parameter-efficient fine-tuning method for vision models that employs a dynamic parameter routing mechanism to generate input-dependent low-rank weight matrices from shared expert centers, thereby enhancing feature diversity and performance across complex dense prediction tasks.

Original authors: Meng Lou, Stanley Yu, Yizhou Yu

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meng Lou, Stanley Yu, Yizhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, highly skilled library of books (a pre-trained AI model) that knows a lot about the world. You want to teach this library a new, specific skill, like recognizing different types of birds or finding objects in a messy room.

Traditionally, there are two ways to do this:

Full Fine-Tuning: You rewrite the entire library. This works great, but it's expensive, slow, and you end up with a massive new version of the library for every single new skill you want to learn.
Parameter-Efficient Fine-Tuning (PEFT): You try to teach the library using just a few sticky notes or small bookmarks. This is cheap and fast, but often the library doesn't learn the new skill very well because the "bookmarks" are too simple.

The paper introduces a new method called ParaX that tries to get the best of both worlds: the low cost of sticky notes with the high performance of rewriting the whole library.

The Problem with Current "Sticky Notes"

The authors argue that current methods (like LoRA or AdaptFormer) have two main flaws:

They are "One-Size-Fits-All": Imagine a teacher giving the exact same lecture to a beginner, an expert, and a child. Current methods use the same "rules" for every single image they see, regardless of what's in the picture. They don't adapt to the specific details of the input.
They are "Isolated": If you have a team of workers (layers in the AI), current methods make each worker learn in a silo. They don't talk to each other. This leads to everyone doing the same thing (redundancy) instead of collaborating to solve a complex problem.

The ParaX Solution: The "Shared Expert Center"

ParaX changes the game by treating the AI's learning tools like a Mixture of Experts (MoE).

The Analogy: The Master Tool Shed
Imagine the AI has a massive Shared Tool Shed (the Expert Center) filled with thousands of specialized tools (trainable parameter matrices).

Old Method: Every worker in the factory gets their own tiny, fixed toolbox. They can't change their tools based on the job.
ParaX Method: Every worker has a Smart Router. When a new task arrives (an image comes in), the Smart Router looks at the image and says, "Oh, this is a tricky scene with lots of small details. I need to grab Tool #4 and Tool #12 from the Master Shed and combine them to make a custom wrench just for this moment."

How It Works (The Magic Steps)

Dynamic Routing: Instead of using a fixed set of rules, ParaX looks at the specific image and dynamically decides which "tools" (parameter matrices) from the Shared Tool Shed to use. It's like a chef tasting a soup and instantly deciding which specific spices to add, rather than following a rigid recipe.
Shared Knowledge: Because all the workers (layers in the network) pull from the same Master Tool Shed, they naturally start to learn from each other. If one layer figures out a great tool combination for a specific type of edge, that knowledge is available to the whole team. This reduces redundancy and makes the AI smarter at seeing complex scenes.
Multi-Scale Mixing: ParaX also adds a feature that lets it look at things at different "zoom levels" (like looking at a tree from far away to see the shape, and up close to see the leaves) simultaneously, which is crucial for tasks like finding objects in a picture.

The Results: Why It Matters

The authors tested ParaX on difficult tasks like:

Semantic Segmentation: Coloring every pixel in an image to say what it is (e.g., "sky," "car," "person").
Object Detection: Finding and boxing objects in a picture.
Panoptic Segmentation: A super-hard mix of the above two.

The Claim:
ParaX achieved better results than the previous best "sticky note" methods, and in some cases, it even beat the expensive "rewrite the whole library" method, all while using less than 4% of the trainable parameters.

In simple terms: ParaX taught the AI to be a master chef who can cook a gourmet meal using a tiny, shared set of ingredients and a smart plan, whereas other methods were stuck using pre-packaged meals that tasted okay but weren't great.

Summary

ParaX is a new way to teach AI models new skills. Instead of giving the AI a static, one-size-fits-all set of instructions, it gives the AI a dynamic, shared toolbox that it can customize on the fly for every single image it sees. This makes the AI much better at understanding complex scenes without needing to be retrained from scratch.

Technical Summary: ParaX - Adapting Vision Models with Dynamic Parameter Routing

1. Problem Statement

Parameter-efficient fine-tuning (PEFT) aims to adapt pre-trained vision models to downstream tasks using a minimal number of trainable parameters while maintaining performance comparable to full fine-tuning. While successful in classification, existing PEFT methods struggle with complex dense prediction tasks (e.g., object detection, semantic segmentation). The paper identifies two primary limitations in current adapter-based approaches (e.g., AdaptFormer, Mona):

Representation Deficiency: Existing adapters often employ input-agnostic low-rank modeling. They learn static transformations that must work universally for all inputs, failing to support input-dependent adaptations necessary for capturing complex spatial dependencies in dense predictions. This results in a smaller Effective Receptive Field (ERF) compared to full fine-tuning.
Feature Redundancy: Adapters embedded in different network layers are typically isolated, lacking cross-layer interaction. This leads to redundant representations where different layers capture similar information, as evidenced by high Centered Kernel Alignment (CKA) similarity between layers in existing methods.

2. Methodology: ParaX

The authors propose ParaX, a novel adapter-style PEFT method built upon a simple Mixture-of-Experts (MoE) architecture. Instead of learning isolated adapter weights for each layer, ParaX introduces a Shared Expert Center and a Dynamic Parameter Routing mechanism.

2.1 Shared Expert Center

ParaX constructs a large, shared repository of trainable parameter matrices (experts) for each stage of the hierarchical vision network.

Channel Transformations: The center contains pairs of matrices ( $E_A, E_B$ ) for down-projection and up-projection.
Spatial Transformations: To enhance spatial modeling, the center includes sets of matrices ( $S_A, S_B, S_C$ ) designed to generate multi-kernel depthwise convolution kernels.
Capacity: The size of the expert center ( $M$ ) and the latent dimension ( $\hat{C}$ ) are hyperparameters controlling the total parameter count.

2.2 Dynamic Parameter Routing

During the forward pass, each ParaX module dynamically generates weight matrices tailored to the current input and module position:

Routing Mechanism: A lightweight router network processes the input feature (via Global Average Pooling and linear layers) to produce dynamic gating vectors ( $G$ ).
Dynamic Weight Generation: These gating vectors selectively aggregate the parameter matrices from the Shared Expert Center.
- For channel projection: $W_1 = G_1 \times E_A$ and $W_2 = G_2 \times E_B$ .
- For spatial mixing: The router generates dynamic gating vectors to combine expert matrices into dynamic depthwise convolution kernels ($D2Convs$).
Input-Dependence: Unlike static adapters, the resulting weight matrices are input-dependent, allowing the model to customize feature representations based on specific input variations.

2.3 Dynamic Multi-scale Spatial Mixing

ParaX integrates multi-scale spatial modeling by sequentially stacking dynamic depthwise convolutions with increasing kernel sizes (e.g., 3, 5, 7). These outputs are aggregated via a Spatially-varying Aggregation (SA) module, which uses spatial attention to dynamically recalibrate features across scales. This design is explicitly input-dependent, contrasting with the fixed, input-independent weights used in methods like Mona.

3. Key Contributions

Dynamic Parameter Routing: The paper introduces a mechanism where trainable parameter matrices act as "experts" that are dynamically aggregated via gating vectors. This facilitates low-rank adaptation in an input-dependent manner, generating more powerful and customized feature representations.
Shared Expert Center: By sharing the same expert center across multiple network layers, ParaX promotes implicit cross-layer feature interaction. This diversifies information flow and reduces feature redundancy without requiring explicit cross-layer connections.
Input-Dependent Spatial Modeling: The method extends dynamic routing to spatial transformations, generating input-specific multi-scale convolution kernels, thereby addressing the representation deficiency of static adapters in dense prediction tasks.

4. Experimental Results

The authors evaluated ParaX on diverse visual recognition tasks using Swin and ConvNeXt backbones, comparing against full fine-tuning and state-of-the-art PEFT methods (VPT, LoRA, AdaptFormer, Mona, etc.).

Semantic Segmentation (ADE20K): Using Swin-L, ParaX achieved 52.0% mIoU, surpassing full fine-tuning (51.2%) by 0.8% while using less than 4% of the trainable parameters. It also outperformed the strong baseline Mona by 0.4%.
Object Detection & Instance Segmentation (COCO2017): With ConvNeXt-L, ParaX improved APb/APm by 1.4%/1.6% over full fine-tuning and by 0.6%/0.4% over Mona.
Panoptic Segmentation (COCO2017): ParaX improved PQ by 1.7% and 2.0% over AdaptFormer when integrated with Swin-B and ConvNeXt-B, respectively.
Image Classification: On ImageNet-R (a dataset with significant domain shift), ParaX outperformed Mona by 1.6% in top-1 accuracy, demonstrating robustness to domain shifts.
Generalization: The method showed superior performance on ViT architectures, Mamba-based models (VMamba), human pose estimation, and remote sensing segmentation.

5. Significance and Claims

The paper claims that ParaX addresses the fundamental limitations of input-agnostic modeling and feature redundancy in current PEFT methods. By leveraging a shared expert center and dynamic routing, ParaX achieves:

Superior Representation Capacity: The input-dependent nature of the weights allows the model to capture long-range dependencies and complex spatial contexts more effectively, evidenced by larger ERFs comparable to full fine-tuning.
Enhanced Feature Diversity: The shared expert center facilitates implicit cross-layer interactions, reducing redundancy and allowing layers to extract more representative features.
Efficiency: Despite the dynamic routing and multi-scale convolutions, ParaX maintains competitive computational efficiency and significantly reduces storage costs (requiring <4% of trainable parameters) compared to maintaining full fine-tuned models for multiple tasks.

The authors conclude that ParaX sets a new state-of-the-art for parameter-efficient adaptation in dense prediction tasks, demonstrating that dynamic, input-dependent parameter generation is crucial for bridging the performance gap between PEFT and full fine-tuning.