(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork

This paper proposes PASS, a novel recurrent hyper-network framework that leverages visual prompts and weight statistics to effectively identify channel importance and derive high-quality structural sparsity, achieving superior accuracy and speedup compared to existing pruning methods across various architectures and datasets.

The Gist

Imagine you have a massive, super-smart robot chef (a large neural network) that can cook any dish in the world. It's incredibly talented, but it's also huge, heavy, and requires a kitchen the size of a warehouse to operate. You want to make it smaller and faster so it can fit in a regular home kitchen, but you can't just start chopping off random limbs or organs, or it might forget how to cook. You need to know exactly which parts are essential and which are just "extra" weight.

This is the problem the paper PASS solves.

Here is the story of how they did it, using some everyday analogies:

1. The Problem: The "Blind" Sculptor

Usually, when engineers try to shrink these giant AI models (a process called structural pruning), they act like blind sculptors. They look at the model's internal weights (the "muscles" of the AI) and try to guess which channels (the "pipes" that carry information) are useless. They often use simple math rules or guesswork.

The problem? These methods ignore two things:

• The Flow: If you cut a pipe in the first layer, it might break the flow for the second layer. They are connected.
• The Context: They treat the model like a static machine, ignoring the actual food (the images) it's trying to cook.

2. The Solution: The "Visual Prompt" as a Flashlight

The authors of this paper had a brilliant idea. Instead of just looking at the robot's internal wiring, they decided to shine a flashlight on the task.

In the world of AI, a "Visual Prompt" is like a special sticker or a colored patch you stick onto an image before showing it to the AI. Think of it as a "hint" or a "mood setter."

• The Analogy: Imagine you are trying to find the best route through a dark maze. A standard method is to feel the walls blindly. The PASS method is like turning on a flashlight (the visual prompt) that illuminates the path, helping you see which walls (channels) are actually important to keep and which are dead ends.

3. The Engine: The "Recurrent HyperNetwork" (The Smart Assistant)

To put this all together, they built a new tool called PASS. Think of PASS as a super-smart assistant with a very specific job:

• It looks at the model: It checks the "muscle strength" (weights) of the AI.
• It looks at the prompt: It reads the "flashlight" (visual prompt) to understand the context.
• It remembers the past: This is the "Recurrent" part. Imagine the assistant is walking through the AI layer by layer. When it decides to cut a pipe in Layer 1, it remembers that decision when it gets to Layer 2. It knows, "Oh, I cut that pipe earlier, so I need to be careful about what I cut next to keep the water flowing."

This assistant doesn't just guess; it learns a pattern. It creates a map (mask) that says, "Keep these channels, cut those ones," specifically tailored for that type of image.

4. The Results: A Leaner, Faster Chef

When they tested this new assistant (PASS) on six different datasets (like recognizing cars, food, or textures) and four different AI architectures, the results were amazing:

• Better Accuracy: The trimmed-down AI models actually performed better than other trimmed models. It's like taking 30% of the weight off a race car, but it still runs faster because you removed the drag.
• Speed: They got the same performance with much less computing power (FLOPs).
• Transferability: The best part? The "map" (the strategy) the assistant learned for one task (like recognizing cats) worked surprisingly well for other tasks (like recognizing dogs) without needing to be retrained from scratch. It's like learning to ride a bike; once you have the balance, you can ride a motorcycle too.

Summary in a Nutshell

PASS is a new way to shrink giant AI models. Instead of just randomly cutting parts off, it uses a special visual hint (prompt) and a smart, memory-keeping assistant to figure out exactly which parts of the AI are essential.

It's like upgrading from a blunt axe to a laser-guided scalpel. The result is a smaller, faster, and smarter AI that doesn't lose its brain in the process.

Technical Summary

1. Problem Statement

Large-scale neural networks achieve state-of-the-art performance but suffer from massive computational and memory costs, hindering deployment. Structural model pruning (removing entire channels/filters rather than individual weights) is a preferred compression technique due to its hardware-friendly acceleration. However, a critical bottleneck remains: how to accurately estimate channel significance.

Existing methods often rely on:

• Heuristics or static metrics (e.g., weight norms, Taylor expansion) that ignore the sequential dependency between layers.
• Model-centric approaches that focus solely on weight statistics, neglecting the potential of input data (visual prompts) to reveal structural importance.
• Lack of inter-layer dependency modeling, which can disrupt gradient flow and structural pathways when pruning.

The authors posit that input editing via visual prompts can provide crucial information to dissect the relevance of structural components, but this potential has not been fully leveraged for generating the pruning mask itself.

2. Methodology: The PASS Framework

The authors propose PASS (Visual Prompt Locates Good Structure Sparsity), a novel, end-to-end, data-centric framework that uses a Recurrent HyperNetwork to generate high-quality sparse channel masks.

Core Concept

PASS treats channel pruning as a sequence generation problem. It argues that the decision to prune a channel in layer $i$ ($M^{(i)}$) should depend on:

1. Previous Layer's Mask ($M^{(i-1)}$): To preserve structural pathways and gradient flow.
2. Current Layer's Weights ($W^{(i)}$): To utilize intrinsic weight statistics.
3. Visual Prompts ($V$): To leverage data-centric insights and input space characteristics.

Architecture Components

1. Recurrent HyperNetwork (LSTM Backbone):

   • Uses a Long Short-Term Memory (LSTM) network to capture sequential dependencies between layers.
   • Input: The LSTM takes the current layer's weights (processed) and a visual prompt embedding as input, along with the hidden state from the previous layer's mask.
   • Mechanism: It operates in an "auto-regressive" manner: $M^{(i)} = \text{LSTM}_\theta(\tilde{W}^{(i)}, g_\omega(V))$, where $\tilde{W}^{(i)}$ is the weight tensor masked by the previous layer's output.

2. Visual Prompt Encoder:

   • A learnable 3-layer CNN ($g_\omega$) extracts representations from the raw visual prompt $V$.
   • These representations serve as the initial hidden state for the LSTM, effectively "priming" the pruning process with data-specific context.

3. Mask Generation & Optimization:

   • Embedding to Mask: Since LSTM outputs fixed-length embeddings but channel counts vary, a linear layer maps embeddings to channel-wise importance scores.
   • Differentiability: A "straight-through estimator" is used during backpropagation to allow binary mask selection (keeping top $s\%$ channels) to be differentiable.
   • Global Pruning: Instead of uniform pruning per layer, PASS employs Global Pruning, eliminating the lowest-scoring channels across all layers simultaneously to optimize the overall sparsity ratio.

4. Training Strategy:

   • Phase 1 (Learning PASS): Jointly optimizes the visual prompt $V$, encoder weights $\omega$, and LSTM weights $\theta$ to minimize the loss on the pruned network.
   • Phase 2 (Fine-tuning): The generated sparse subnetwork is fine-tuned on the target dataset with the mask fixed.

3. Key Contributions

• Data-Centric Pruning: Introduces the concept of using visual prompts not just for fine-tuning, but as a primary signal to discover which channels are important, bridging the gap between prompting techniques and structural pruning.
• Recurrent Mechanism for Dependency: Proposes an LSTM-based hypernetwork to explicitly model the inter-layer dependency of channel masks, ensuring that pruning decisions in one layer consider the state of the previous layer to maintain structural integrity.
• PASS Framework: A unified, end-to-end algorithm that integrates visual prompts, weight statistics, and recurrent logic to generate sparse masks.
• Transferability: Demonstrates that both the learned sparse masks and the hypernetwork itself possess high transferability to unseen tasks and architectures.

4. Experimental Results

The authors evaluated PASS across 6 datasets (CIFAR-10/100, Tiny-ImageNet, Food101, DTD, StanfordCars) and 4+ architectures (ResNet-18/34/50, VGG-16, ResNeXt-50, ViT-B/16, Swin-T).

• Performance Superiority:

  • At the same FLOPs level, PASS outperforms baselines (Group-L1, GrowReg, Slim, DepGraph, ABC Pruner) by 1%–3% in accuracy (e.g., on Food101).
  • To achieve comparable accuracy (e.g., 80%), PASS provides a 0.35× higher speedup than the best baselines.
  • In some cases (e.g., CIFAR-100, DTD), PASS-pruned models even surpassed the fully fine-tuned dense models.

• Transferability:

  • Masks learned on Tiny-ImageNet successfully transferred to CIFAR-10/100 and StanfordCars with minimal performance drop.
  • The Hypernetwork itself (trained on one task) proved more transferable than static masks, suggesting it captures generalizable topological rules.

• Ablation Studies:

  • Inputs: Removing either the visual prompt or weight statistics caused significant accuracy drops, confirming the necessity of both.
  • Recurrence: Replacing the LSTM with CNN or MLP (destroying the recurrent nature) degraded performance, validating the importance of modeling layer dependencies.
  • Pruning Strategy: Global Pruning consistently outperformed Uniform Pruning.
  • Prompt Size: An "Additive" visual prompt strategy performed better than "Expansive," with an optimal prompt size of 16 pixels.

5. Significance

This paper represents a paradigm shift in structural pruning by moving from a purely model-centric view (analyzing weights) to a data-model co-design view.

• Novelty: It is the first work to leverage visual prompts specifically for locating structural sparsity rather than just adapting a pruned model.
• Efficiency: It offers a robust method to create highly efficient subnetworks without sacrificing accuracy, making large models more deployable on edge devices.
• Generalizability: The demonstration that a single hypernetwork can learn to prune diverse architectures and transfer across tasks suggests a new direction for "universal" pruning strategies in deep learning.

In conclusion, PASS proves that input editing (visual prompts) is a powerful, underutilized tool for understanding model structure, and that recurrent mechanisms are essential for handling the complex dependencies inherent in deep neural network pruning.