🤖 machine learning

GD-FPS: Growth-Driven Feedforward Parameter Selection for Efficient Fine-Tuning

The paper proposes GD-FPS, a strictly gradient-free, forward-pass-only parameter selection method that efficiently identifies optimal fine-tuning subsets by scaling intrinsic weight magnitudes with relative activation growth, achieving competitive performance while significantly reducing memory usage and latency compared to existing gradient-based approaches.

Original authors: Kenneth Yang, Wen-Li Wei, Jen-Chun Lin

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenneth Yang, Wen-Li Wei, Jen-Chun Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a pre-trained AI model) that has read almost every book in the world. Now, you want to teach this library a very specific new skill, like recognizing rare birds or diagnosing a specific type of tumor.

The old way of doing this was Full Fine-Tuning: You forced the entire library to re-read every single book and rewrite every single page to fit the new topic. This is like trying to renovate a skyscraper while it's still occupied. It takes forever, costs a fortune in energy (computing power), and requires a massive amount of space (memory) to hold all the blueprints and notes.

Then, people tried Addition-Based Methods (like Adapters or LoRA). Instead of rewriting the whole library, they built small, new "add-on" rooms or shelves. While this saved some space, it made the building more complicated to navigate (engineering complexity) and slowed down how fast you could find a book (inference latency).

Next came Selection-Based Methods (like GPS). The idea was smart: "Let's just pick the most important pages in the existing library to rewrite and leave the rest frozen." However, to figure out which pages were important, the old method required a massive, exhausting "backward pass." It was like asking the librarian to read every book, write a detailed critique of every sentence, and then check their notes to see which sentences mattered most. This was still incredibly slow, used up all the memory, and the results were a bit shaky because they depended on which random books the librarian happened to pick first (stochastic noise).

The New Solution: GD-FPS (Growth-Driven Feedforward Parameter Selection)

The authors of this paper propose a new, much smarter way to pick the pages to rewrite. They call it GD-FPS.

Here is how it works, using a simple analogy:

1. The "Before" Snapshot (The Anchor)
Imagine you take a snapshot of the library's current state using a standard set of books (the pre-training data). You note down how "loud" or active each bookshelf is when people are browsing these standard books. This is your Anchor. It represents how the library normally behaves.

2. The "After" Snapshot (The Growth)
Now, you bring in the new, specific books (the downstream task, like bird identification). You watch the library again and see how the bookshelves react to these new books.

3. The "Growth" Calculation
Instead of doing a massive, backward-looking critique, GD-FPS simply asks: "Which bookshelves got significantly louder or more active when we switched to the new books compared to the old ones?"

The Magic Formula: It looks at two things:
- Intrinsic Strength: How important is this bookshelf usually? (Weight magnitude).
- Relative Growth: How much more active did it get with the new books compared to the old ones? (Activation growth).

If a bookshelf was already loud with the old books and stayed loud with the new ones, it's just doing its normal job. But if a bookshelf was quiet before and suddenly got very loud with the new books, that's a sign of growth. That bookshelf is critical for the new task.

Why is this a big deal?

The paper claims three major advantages over the previous "Selection" methods:

It's a Sprint, not a Marathon (Speed): The old method had to run a complex backward calculation (like a full audit) to find the important pages. GD-FPS just runs a quick forward pass (like a quick glance). The paper says this makes the selection process 2.7 times faster.
It Fits in a Backpack (Memory): Because it doesn't need to store massive amounts of "audit notes" (gradients) for the whole library, it uses 18 times less memory. This means you can run this on much cheaper, smaller computers.
It's Reliable (Deterministic): The old method was like a coin flip; if you picked a different random batch of books to start with, you might end up rewriting different pages. GD-FPS is deterministic. No matter who runs it or how they shuffle the books, it will always pick the exact same important pages because it looks at the entire dataset's average behavior, not a random sample.

The Results

The authors tested this on 26 different visual tasks, ranging from identifying specific types of flowers and dogs to spotting tumors in medical scans (semantic segmentation).

Performance: GD-FPS performed just as well as, or even better than, the best existing methods (including the complex "add-on" methods and the previous "selection" method).
Efficiency: It achieved these results while being drastically faster and lighter on memory during the setup phase.

In short, GD-FPS is like a smart librarian who can instantly identify exactly which few pages of a massive encyclopedia need to be updated for a new topic, without needing to re-read the whole thing or build new rooms, saving time, money, and space.

Technical Summary: GD-FPS

Problem Statement

Parameter-Efficient Fine-Tuning (PEFT) is essential for adapting large-scale pre-trained models to downstream tasks without the prohibitive memory and computational costs of full fine-tuning. However, existing PEFT approaches face significant limitations:

Addition-based methods (e.g., Adapters, LoRA) introduce new trainable parameters, increasing model storage, inference latency, and engineering complexity regarding optimal module placement.
Selection-based methods (e.g., Gradient-based Parameter Selection, GPS) avoid adding parameters but rely on computing full-model gradients during a selection phase. This approach incurs massive memory usage (identical to full fine-tuning), substantial computational latency due to backward passes, and suffers from stochastic noise caused by batch sampling, making the selection process non-deterministic and sensitive to environment configurations.

The core challenge is to identify a sparse subset of parameters for fine-tuning that is memory-efficient, fast, deterministic, and robust, without relying on computationally expensive gradient calculations.

Methodology: Growth-Driven Feedforward Parameter Selection (GD-FPS)

The authors propose GD-FPS, a strictly gradient-free method that identifies optimal parameters entirely through forward passes. The method operates on the premise that parameters critical for domain adaptation are those that exhibit significant activation growth relative to the model's pre-training behavior, scaled by their intrinsic weight magnitude.

The framework consists of two distinct phases:

1. Prior-Anchoring Phase

Goal: Establish a behavioral baseline (anchor) representing the model's pre-training state.
Process: A forward pass is performed on a randomly sampled subset of the original pre-training data ( $D_{pre}$ ).
Output: The expected norm of input activations for each weight is computed: $\mathbb{E}_{x \sim D_{pre}}[|a_k^{(i-1)}(x)|]$ .
Efficiency: This is a one-time procedure shared across all downstream tasks, requiring negligible amortized overhead (e.g., 27.38 seconds for 12,800 images on a single GPU).

2. Growth-Quantifying Phase

Goal: Measure task-specific adaptation by comparing downstream activations against the pre-training anchor.
Process: A forward pass is performed on the downstream dataset ( $D_{down}$ ) to compute the task-specific activation expectation: $\mathbb{E}_{x \sim D_{down}}[|a_k^{(i-1)}(x)|]$ .
Scoring: The final importance score ( $I_{GD-FPS}$ ) for a weight $w_{k,j}^{(i)}$ is calculated by scaling its intrinsic magnitude by the relative growth of its activation:
$I_{GD-FPS}(w_{k,j}^{(i)}) = |w_{k,j}^{(i)}| \cdot \left( \frac{\mathbb{E}_{x \sim D_{down}}[|a_k^{(i-1)}(x)|]}{\mathbb{E}_{x \sim D_{pre}}[|a_k^{(i-1)}(x)|]} - 1 \right)$
Selection: The top- $k$ scoring parameters per neuron are selected for fine-tuning.

This formulation ensures that parameters with high static activation (task-agnostic) but low relative growth are deprioritized, while those showing a significant "surge" in activation specific to the new domain are selected.

Key Contributions

Gradient-Free Selection: GD-FPS eliminates the need for backward passes during the parameter selection stage, directly addressing the memory and latency bottlenecks of gradient-based selection methods like GPS.
Deterministic Robustness: By relying on global statistical properties (expected activation magnitudes) rather than stochastic batch gradients, GD-FPS guarantees a deterministic selection process that is invariant to random seeds, batch sizes, and data sampling variations.
Growth-Driven Mechanism: The introduction of a relative-growth metric against a pre-training anchor allows the method to distinguish between parameters that are inherently active and those that are truly adapting to the new task.
Comprehensive Evaluation: The method is validated across 26 visual tasks, including image classification (FGVC and VTAB-1k) and semantic segmentation, demonstrating its generalizability across different architectures (ViT, Swin) and tasks.

Experimental Results

The paper evaluates GD-FPS against state-of-the-art baselines, including Full Fine-Tuning, Addition-based methods (Adapter, LoRA, VPT), and the primary selection-based competitor, GPS.

Image Classification (FGVC & VTAB-1k):
- On the FGVC benchmark, GD-FPS achieved a mean accuracy of 92.07%, outperforming GPS (91.78%) and SSF (90.72%).
- On the diverse VTAB-1k benchmark, GD-FPS achieved 75.04% mean accuracy, performing on par with GPS (75.18%) and significantly outperforming other general-purpose PEFT methods.
Semantic Segmentation (VOC2012 & Cityscapes):
- Using a Swin-L backbone, GD-FPS achieved mean mIoU of 83.62% and mAcc of 89.78%.
- It successfully surpassed the Full Fine-Tuning baseline in both metrics and outperformed other general PEFT methods, trailing only the specialized, heavily engineered ProPETL method.
Efficiency Gains (vs. GPS):
- Memory: Reduced peak GPU memory usage by nearly 18× during the selection phase.
- Latency: Accelerated parameter selection execution by over 2.7×.
- Compute: Decreased computational cost from 100.9 GFLOPs to 16.8 GFLOPs per image.
Robustness:
- GD-FPS maintained an Intersection Ratio (IR) of >97% across varying random seeds, batch sizes, and data ratios.
- In contrast, GPS showed a 20–30% drop in IR under similar variations, highlighting its sensitivity to stochastic factors.

Significance and Claims

The paper claims that GD-FPS offers a memory-efficient, fast, and robust solution for fine-tuning large-scale models. By strictly utilizing forward passes, it resolves the fundamental trade-off between selection-based efficiency and the computational costs of gradient computation.

The authors assert that GD-FPS demonstrates that high performance, efficiency, and structural robustness can coexist within selection-based PEFT. It provides a scalable alternative for resource-constrained environments, eliminating the engineering complexity of addition-based methods and the prohibitive memory/latency costs of gradient-based selection, all while guaranteeing a deterministic outcome.