🤖 AI

CPUBone: Efficient Vision Backbone Design for Devices with Low Parallelization Capabilities

This paper introduces CPUBone, a new family of vision backbone models specifically optimized for CPU-based inference by employing grouped convolutions and reduced kernel sizes to balance computational cost with hardware efficiency, thereby achieving state-of-the-art speed-accuracy trade-offs on devices with low parallelization capabilities.

Original authors: Moritz Nottebaum, Matteo Dunnhofer, Christian Micheloni

Published 2026-03-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Moritz Nottebaum, Matteo Dunnhofer, Christian Micheloni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive library. You have two very different types of workers available to help you:

The Super-Team (GPUs): Imagine a stadium filled with 10,000 workers, all working in perfect sync. They can move thousands of books at once. They are incredibly fast at big, parallel tasks.
The Solo Librarian (CPUs): Now imagine a single, very smart librarian working alone. They are great at thinking, making decisions, and handling complex logic, but they can only move one or two books at a time. They don't have a stadium of helpers.

For years, computer scientists have been designing "library organization systems" (AI models) specifically for the Super-Team. They build systems that require moving huge piles of books simultaneously to be efficient.

The Problem:
When you try to run these "Super-Team" systems on the Solo Librarian's computer (like a standard laptop or phone processor), it gets stuck. The librarian tries to do too many things at once, gets overwhelmed, and ends up moving slower than if they just did the job one book at a time. The system is "heavy" with too many instructions (called MACs or Multiply-Accumulate operations), and the librarian's brain can't keep up with the parallel demands.

The Solution: CPUBone
The authors of this paper, Moritz, Matteo, and Christian, asked: "What if we design a library system specifically for the Solo Librarian?"

They created a new family of AI models called CPUBone. Here is how they did it, using simple analogies:

1. The "Grouping" Trick (The Assembly Line vs. The Team)

Standard AI models often try to process all the information in a picture at once.

The Old Way: Imagine the librarian trying to sort 100 different types of books by color, size, and author all in one giant, chaotic pile. It's confusing and slow.
The CPUBone Way: They decided to group the books. Instead of one giant pile, they split the work into two smaller, manageable piles. The librarian sorts the "Red Books" pile, then the "Blue Books" pile.
The Result: By splitting the work into two groups (instead of one giant group), the total amount of work drops significantly, but the librarian doesn't get confused trying to juggle everything. It's like a solo worker switching from "juggling 10 balls" to "juggling 2 balls, then the next 2."

2. The "Smaller Steps" Trick (The Kernel Size)

In AI, the "kernel" is like the size of the window the librarian looks through to read a book.

The Old Way: The librarian uses a giant 3x3 window. They have to look at a huge area to understand a single page. This requires a lot of mental energy (computing power).
The CPUBone Way: They shrunk the window to 2x2.
The Result: The librarian takes smaller, quicker glances. Because the window is smaller, they have to do fewer calculations to understand the page. On a powerful Super-Team, a big window is fine because they have the muscle to handle it. But for the Solo Librarian, a smaller window means they can finish the job much faster without losing accuracy.

3. The "Fused" Shortcut

They also combined steps. Imagine the librarian usually has to:

Pick up a book.
Walk to the shelf.
Put it down.
Walk back.
Pick up the next book.

CPUBone combines steps 2 and 3 into one smooth motion. This "Fused" approach saves the librarian from making unnecessary trips back and forth, keeping the workflow smooth and efficient.

The Big Win

The paper tested these new models on real devices:

Raspberry Pi (a tiny, cheap computer).
Pixel Phone (a standard smartphone).
Intel Laptops (standard office computers).

The Result: CPUBone was significantly faster than other popular AI models on these devices, often running 2x to 4x faster, while still being just as smart (accurate).

Why This Matters

Most AI research focuses on making models faster for powerful servers or high-end graphics cards. But most of the world runs on standard CPUs (laptops, phones, IoT devices).

CPUBone is like a custom-tailored suit for the Solo Librarian. It stops trying to force the librarian to act like a stadium of workers. Instead, it gives them a workflow that matches their natural strengths: doing things sequentially, efficiently, and without getting overwhelmed.

In a nutshell:

Old AI: "Let's do 1,000 things at once!" (Great for Super-Teams, terrible for Solo Librarians).
CPUBone: "Let's do 2 things at a time, but do them really efficiently." (Perfect for Solo Librarians).

This means you can now run smarter, faster AI on your everyday devices without needing expensive, power-hungry hardware.

1. Problem Statement

Recent advancements in vision backbone architectures have primarily optimized for hardware with high parallel processing capabilities, such as GPUs, mobile phones with NPUs, and AI accelerators (e.g., NVIDIA Jetson). These designs often rely on massive parallelism to achieve efficiency.

However, Central Processing Units (CPUs) remain widely used in resource-constrained and practical scenarios but operate under fundamentally different architectural constraints:

Limited Parallelism: CPUs cannot parallelize operations to the same extent as GPUs.
The MAC Bottleneck: Models optimized for high parallelism often fail on CPUs because the sheer number of Multiply-Accumulate (MAC) operations becomes a performance bottleneck under limited concurrency.
Inefficiency of Existing Solutions: Standard efficiency techniques, such as Depthwise Convolutions (DWConv), reduce MAC counts but suffer from low hardware efficiency (low MACs per second, or MACpS) on CPUs due to memory access costs and lack of parallelism.

The core problem is the lack of vision backbones specifically designed to balance low MAC counts with high hardware execution efficiency (MACpS) on CPU-only devices.

2. Methodology

The authors propose CPUBone, a new family of vision backbones designed to maximize MACpS while minimizing total MACs. The methodology involves two primary modifications to standard convolutional blocks (specifically the Mobile Inverted Bottleneck, or MBConv):

A. Grouped Convolutions (Groups = 2)

Instead of standard convolutions (groups=1) or depthwise convolutions (groups= $C_{in}$ ), the authors introduce Grouped MBConv (GrMBConv) and Grouped Fused MBConv (GrFuMBConv) with groups = 2.

MAC Reduction: Theoretically, increasing groups to 2 halves the MAC count ( $M \propto 1/groups$ ).
Hardware Efficiency: Unlike depthwise convolutions, which degrade MACpS significantly, groups=2 maintains high MACpS on CPUs. The authors found that while groups=2 slightly reduces MACpS compared to ungrouped convolutions on some ARM CPUs, the massive reduction in total MACs results in a net latency gain.

B. Kernel Size Reduction (3×3 $\to$ 2×2)

The authors reduce the convolutional kernel size from the standard $3 \times 3$ to $2 \times 2$ .

MAC Reduction: This reduces the kernel area by approximately 56% ( $1 - 4/9$ ), significantly lowering computational cost.
Hardware Efficiency: Crucially, on CPUs, this reduction maintains or even improves MACpS for fused blocks (FuMBConv). In contrast, on GPUs, reducing kernel size often hurts performance due to memory access patterns, but the paper focuses on CPU optimization where this trade-off is favorable.

C. Architecture Design (CPUBone)

Macro Architecture: Inspired by LowFormer, CPUBone uses MBConv blocks as the main component and integrates LowFormer Attention in the final two stages.
Hybrid Strategy:
- Fused vs. Unfused: The authors apply Fused variants (GrFuMBConv) for layers with channel dimensions $< 256$ (where fused blocks show higher MACpS) and Unfused variants (GrMBConv) for layers with $\ge 256$ channels.
- Kernel Sizing: The $2 \times 2$ kernel reduction is applied primarily to the last two stages where computation is concentrated, preserving $3 \times 3$ kernels in earlier layers to maintain a larger receptive field and accuracy.
- Upsampling: Transpose convolutions in the attention mechanism are replaced with nearest-neighbor upsampling to avoid latency spikes on CPUs.

3. Key Contributions

Novel MBConv Variants: Introduction of GrMBConv and GrFuMBConv, which achieve a dual benefit: significantly reduced MAC counts (23–45% less) while maintaining high hardware efficiency (MACpS) on CPUs.
Hardware-Efficiency Analysis: A comprehensive empirical study demonstrating that:
- Grouping (groups=2) and kernel reduction (2x2) preserve or improve MACpS on CPUs, unlike on GPUs where these changes often degrade performance.
- Depthwise convolutions have inherently low MACpS on both CPUs and GPUs, making them less ideal for CPU optimization than grouped convolutions.
CPUBone Architecture: A new family of models (B0–B3) that achieves State-of-the-Art (SOTA) Speed-Accuracy Trade-offs (SAT) across diverse CPU devices (Raspberry Pi 5, Google Pixel 4/7, Intel Xeon).
Downstream Transferability: Demonstration that CPUBone's efficiency transfers effectively to object detection and semantic segmentation tasks.

4. Results

The paper evaluates CPUBone against recent backbones (e.g., MobileNetV4, RepViT, FasterNet, LowFormer) on ImageNet-1K and downstream tasks.

ImageNet Classification:
- CPUBone-B0 achieves 77.6% Top-1 accuracy with 24.2ms latency on a Raspberry Pi 5. This is significantly faster than competitors like EffFormerV2-S0 (36.3ms) and FasterNet-T1 (32.8ms) with comparable or better accuracy.
- CPUBone-B3 achieves 83.1% accuracy with 199.8ms latency on Pi 5, outperforming RepViT-M2.3 (227.0ms) which has slightly lower accuracy (82.5%).
- CPUBone consistently shows lower latency across ARM, Pixel, and Intel CPUs compared to models of similar accuracy.
Downstream Tasks:
- Semantic Segmentation (ADE20K): CPUBone models achieve higher mIoU with significantly lower latency (up to 3x faster) than comparable models. For example, CPUBone-B2 achieves 42.1% mIoU at 338.2ms, while LowFormer-B2 achieves 42.8% mIoU at 808.1ms.
- Object Detection (COCO): CPUBone models outperform others in AP (Average Precision) while being up to 4x faster than comparable models.
Ablation Studies:
- Using groups=4 or 8 reduces MACs further but degrades accuracy and MACpS significantly.
- Using standard (ungrouped) MBConv blocks results in lower accuracy and higher latency, proving the necessity of the grouped design.
- Replacing nearest-neighbor upsampling with transpose convolutions increases latency by ~25% on ARM CPUs.

5. Significance

This work addresses a critical gap in computer vision deployment: CPU-only inference. While the industry focuses heavily on GPUs and NPUs, many real-world applications (edge devices, legacy systems, cost-sensitive deployments) rely on standard CPUs.

Paradigm Shift: It challenges the assumption that "low MAC count" (via depthwise convolutions) is the only path to efficiency. Instead, it proves that high MACpS (hardware-efficient execution) is the dominant factor for CPU performance.
Practical Impact: CPUBone enables high-performance vision tasks on low-cost, low-power hardware without requiring specialized accelerators, making advanced AI more accessible for edge computing.
Design Philosophy: It establishes a new design philosophy for CPU-optimized backbones: Grouped Convolutions (groups=2) + Reduced Kernel Sizes, tailored to the specific memory and parallelism constraints of CPU architectures.