💻 computer science

Slimmable ConvNeXt: Width-Adaptive Inference for Efficient Multi-Device Deployment

The paper introduces Slimmable ConvNeXt, a width-adaptive inference framework that leverages ConvNeXt's modern design to enable efficient multi-device deployment with a single shared weight set, achieving superior accuracy across varying computational constraints compared to existing CNN and Vision Transformer approaches without requiring complex normalization mechanisms.

Original authors: Janek Haberer, Jon Eike Wilhelm, Olaf Landsiedel

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Janek Haberer, Jon Eike Wilhelm, Olaf Landsiedel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of chefs (a computer vision model) who need to cook meals for a crowd. Sometimes, the kitchen is huge with a full staff and fancy equipment (a powerful cloud server). Other times, you're cooking in a tiny camper van with just one burner and a limited supply of ingredients (a mobile phone with a low battery).

Traditionally, if you wanted your chefs to work in both settings, you'd have to hire two completely different teams: one big team for the big kitchen and a tiny, specialized team for the camper van. You'd have to train them separately, store their recipes separately, and switch between them depending on where you are. This is expensive and messy.

The Problem with Old "Flexible" Chefs
Some researchers tried to create a "super-chef" who could shrink or grow on the fly. They built a single team that could operate with 100 chefs, 50 chefs, or 10 chefs. However, these old methods had a major flaw: they required a complex "switchboard" (called switchable batch normalization) to keep track of statistics for every possible team size. It was like having a different set of measuring cups for every single ingredient size, making the kitchen chaotic and hard to train.

The New Solution: Slimmable ConvNeXt
This paper introduces a new type of chef team called Slimmable ConvNeXt. The authors discovered that the modern design of the ConvNeXt architecture is naturally perfect for shrinking and growing without the messy switchboard.

Here is how it works, using simple analogies:

1. The "LayerNorm" Magic (No Switchboard Needed)

Old flexible models needed special rules to handle different team sizes. Slimmable ConvNeXt uses a technique called LayerNorm.

The Analogy: Imagine a traditional team where the manager needs to know exactly how many people are in the room to give instructions. If the room size changes, the manager panics and needs a new rulebook.
The New Way: LayerNorm is like a manager who gives instructions based on what each person is doing right now, regardless of how many people are in the room. Whether you have 100 chefs or 10, the manager adapts instantly. This means you don't need the complex "switchboard" anymore. The training process becomes much simpler and cleaner.

2. The "Inverted Bottleneck" (Easy to Slice)

ConvNeXt uses a structure called an "inverted bottleneck."

The Analogy: Think of a standard sandwich: Bread (small), Meat (big), Bread (small). If you want to make a smaller sandwich, you have to cut the bread and the meat, which is tricky.
The New Way: An inverted bottleneck is like a sandwich where the filling is huge, but the bread is thin. The "meat" (the heavy calculation) is concentrated in the middle. When you want to shrink the model, you just slice off the outer edges of the meat. It's very easy to cut a slice off the middle without messing up the whole structure. This makes it easy to create smaller versions of the model just by "slicing" the channels (the width of the data).

3. How It Works in Practice

The researchers trained one single model that contains multiple "nested" versions of itself inside it.

The Training: Imagine the chefs practicing every day. Some days they practice with the full team of 100. Other days, they practice with only 50, and other days with just 25. They all share the same knowledge base (weights). Because they practice in all these different sizes, they learn to be good at any size.
The Result: When you deploy the model, you don't need to reload a new file. You just tell the model, "Hey, we only have battery for 25 chefs today," and it instantly switches to its 25-chef mode. If tomorrow you have a full power source, it switches back to 100 chefs.

The Results: Better Performance, Less Math

The paper tested this on a massive dataset called ImageNet-1k (a library of 1.28 million images). They compared their new "Slimmable ConvNeXt" against the best existing flexible models (which were mostly based on Vision Transformers, a different type of AI architecture).

The Winner: The Slimmable ConvNeXt was faster and more accurate.
- At a medium computing level, it scored 80.8% accuracy. The next best competitor scored 78.4%.
- At a very low computing level (like a weak phone), it scored 77.4%, while the competitor scored 73.0%.
The Scale: They tested small, medium, and large versions of their model. The larger versions were even better at shrinking down without losing too much accuracy. For example, the largest version could run at full power with 82.8% accuracy, and even when shrunk to half-size, it stayed incredibly strong.

Why This Matters (According to the Paper)

The paper claims this is the first time this specific "slicing" technique has been applied to ConvNeXt.

Simplicity: It removes the need for complex normalization switches required by older methods.
Efficiency: Because it doesn't use "self-attention" (a heavy process used by Transformer models), it requires fewer mathematical operations (GMACs) to get the same result.
Versatility: It allows a single model to run on a massive server, a laptop, or a mobile phone without needing to store multiple different files.

In short, the authors built a "Swiss Army Knife" of AI models. Instead of carrying a knife, a screwdriver, and a corkscrew separately, you have one tool that can transform into any of them instantly, and it does it better and more efficiently than the previous generation of multi-tools.

Technical Summary: Slimmable ConvNeXt

Problem Statement

Deploying vision models across devices with heterogeneous resource constraints—or on a single device where compute availability fluctuates due to battery state, thermal throttling, or latency deadlines—typically necessitates training and maintaining separate models for each target platform. This approach is costly and impractical for dynamic environments. While width-adaptive inference offers a solution by training a single set of shared weights containing multiple nested subnetworks, prior approaches face specific architectural limitations:

Classical CNNs: Methods like Slimmable Networks and US-Nets require switchable batch normalization to maintain separate running statistics for different channel widths, adding complexity and parameters.
Vision Transformers (ViTs): Recent scalable methods (e.g., HydraViT, MatFormer) rely on attention mechanisms that impose structural constraints on how subnetworks can be extracted (e.g., dropping attention heads in multiples of head dimension or retaining full embedding dimensions), limiting the granularity and range of scalability.

Methodology

The paper proposes Slimmable ConvNeXt, the first application of channel-width slimming to the ConvNeXt architecture family. The method leverages specific design choices inherent to ConvNeXt to simplify the training pipeline and improve efficiency.

Architectural Design

Each ConvNeXt block is assigned a slimming ratio $p \in (0, 1]$ , which determines the fraction of active channels.

Channel Slicing: All weight tensors within a block (depthwise convolution, pointwise convolutions, and LayerNorm parameters) are sliced to retain only the first $\lfloor p \cdot C \rfloor$ channels.
Inverted Bottlenecks: The intermediate expansion layer (4 $\times$ expansion) is also sliced to $4 \cdot \lfloor p \cdot C \rfloor$ , preserving the expansion ratio.
Residual Handling: When a slimmed block produces fewer channels than its input, the output is zero-padded to match the original dimension before adding the residual connection. This avoids introducing additional projection parameters.
Normalization: Unlike classical slimmable CNNs, ConvNeXt uses LayerNorm instead of Batch Normalization. LayerNorm computes statistics per sample and channel slice, allowing it to adapt naturally to any channel count without requiring switchable normalization layers or separate running statistics.

Training Procedure

The model is trained using a joint training procedure with alternating widths:

Stochastic Selection: A set of $p$ -lists (defining the slimming ratio for every block) is defined. During training, one $p$ -list is randomly selected per batch.
Shared Weights: All subnetworks share the same weights. A subnetwork with ratio $p < 1$ uses a strict subset of the parameters used at $p=1$ .
Optimization: The model undergoes standard forward-backward passes through the selected subnetwork. This stochastic alternation is analogous to tail-drop training in HydraViT but applied to channel dimensions.
AutoSlim (Optional): To optimize for non-uniform per-block widths, the paper investigates AutoSlim. This greedy search starts with uniform widths and iteratively reduces the ratio of the least important block (based on validation accuracy drop) until a target average ratio is reached. This typically preserves wider early and late stages while aggressively slimming intermediate stages.

Key Contributions

Architecture Suitability: The authors demonstrate that ConvNeXt's design—specifically the use of LayerNorm and inverted bottlenecks—makes it an ideal candidate for width-adaptive slimming. This eliminates the overhead of switchable batch normalization required by prior CNN approaches and avoids the structural constraints of attention-based ViT methods.
Simplified Pipeline: The approach yields a simpler training pipeline than both prior CNN and ViT slimmable methods, requiring no switchable normalization, knowledge distillation, or sandwich training rules.
Performance: The method achieves state-of-the-art accuracy across a wide range of compute budgets on ImageNet-1k, outperforming both ViT-based and classical CNN baselines.

Experimental Results

The authors evaluated Slimmable ConvNeXt on ImageNet-1k across three model sizes: ConvNeXt-Tiny (T), Small (S), and Base (B). All models were trained from scratch for 600 epochs.

ConvNeXt-Tiny (3 Subnetworks)

Full Capacity (4.5 GMACs): 80.8% top-1 accuracy.
Half Capacity (1.2 GMACs): 77.4% top-1 accuracy.
Comparison: Outperforms HydraViT's 6-head subnetwork (78.4% at 4.6 GMACs) by 2.4 percentage points and its 3-head configuration (73.0% at 1.3 GMACs) by 4.4 percentage points. It also surpasses MatFormer-S and SortedNet-S at comparable GMACs.

ConvNeXt-Base (4 Subnetworks)

Full Capacity (15.4 GMACs): 82.8% top-1 accuracy.
Half Capacity (4.0 GMACs): 81.6% top-1 accuracy.
Comparison: At half capacity, it achieves 81.6% with 4.0 GMACs, surpassing HydraViT's 80.2% at 4.6 GMACs (13% fewer operations). It covers a compute range of 1.1 to 15.4 GMACs from a single set of weights, whereas MatFormer is limited to roughly half-to-full compute.

Scaling and Robustness

Graceful Degradation: Larger models (ConvNeXt-B) exhibit more graceful degradation under slimming compared to smaller models (ConvNeXt-T), retaining 98.7% of their full-capacity accuracy at half capacity.
Subnetwork Count: Increasing the number of subnetworks (e.g., from 3 to 5) generally leads to diminishing returns or slight accuracy drops for smaller models, but ConvNeXt-B remains robust, with 4 subnetworks actually improving peak accuracy.
AutoSlim: AutoSlim provides modest improvements at low capacities (e.g., +1.0% at quarter capacity for ConvNeXt-T) but offers marginal or slightly negative benefits at full capacity, likely due to the zero-padding strategy in residual connections.

Significance and Claims

The paper claims that Slimmable ConvNeXt provides a single, elastic model capable of dynamically adjusting its computational cost at inference time to match varying resource budgets. This enables deployment scenarios where a single model can serve a heterogeneous fleet of devices or adapt to runtime constraints (battery, thermal, latency) without reloading weights.

The authors emphasize that the combination of LayerNorm (eliminating switchable normalization) and inverted bottlenecks (facilitating straightforward parameter slicing) allows ConvNeXt to achieve higher accuracy than state-of-the-art baselines at nearly every operating point, while maintaining the simplicity and hardware compatibility of convolutional architectures. The work suggests that for deployment scenarios requiring reduced capacity, starting from a larger base model and slimming aggressively may yield better results than training a smaller model at full width.