🤖 AI

Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators

This paper presents "Sparse by Command," a hardware-software co-designed approach that leverages task commands to dynamically skip unnecessary computation via lightweight gating networks and a specialized tiled accelerator, achieving up to 76% FLOP reduction and 2.4x latency speedup for multi-task inference without altering model architecture.

Original authors: Afzal Ahmad, Gaoyu Mao, Shoubo Hu, Hui-Ling Zhen, Mingxuan Yuan, Xinyu Chen, Wei Zhang

Published 2026-07-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Afzal Ahmad, Gaoyu Mao, Shoubo Hu, Hui-Ling Zhen, Mingxuan Yuan, Xinyu Chen, Wei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the conductor of a massive, high-tech orchestra. In the world of artificial intelligence, this orchestra is a "neural network," a brain-like computer program designed to make decisions. Usually, when this orchestra plays a piece of music, every single musician picks up their instrument and plays, no matter what kind of song is being performed. If the song is a gentle lullaby, the heavy drums and blaring trumpets still play at full volume, even though the song doesn't need them. This is incredibly wasteful. It uses up a lot of energy and takes a long time, just like driving a giant truck to the corner store to buy a single apple.

In the specific field of "multi-task inference," this problem is even bigger. Here, one single computer brain is asked to do many different jobs at once, like steering a car, braking, or turning left. The problem is that the computer doesn't know which job it's doing until it starts working, so it prepares for all jobs at once, wasting power on things it doesn't need. Scientists have tried to fix this by making the computer "sparse," which is a fancy word for "empty." They've tried to tell the computer to skip some notes, but usually, the computer still has to check every single note to see if it should skip it, which takes time and energy. This paper asks a simple question: What if the computer knew exactly which job it was going to do before it started playing, so it could tell the musicians to put down their instruments before they even picked them up?

The authors of this paper, a team from Hong Kong and Huawei, have built a clever new system called "Sparse by Command" that solves this problem. They realized that in tasks like autonomous driving, the "command" (like "turn left" or "brake") is known way before the computer starts looking at the road. They used this free piece of information to create a smart switchboard. Instead of the computer guessing which parts of its brain to use, a tiny, super-fast helper network looks at the command and instantly flips a switch to turn off the specific parts of the main brain that aren't needed for that moment.

Think of it like a smart home. If you tell your house "I'm going to bed," the system doesn't just dim the lights in the living room; it knows you aren't in the kitchen, so it cuts the power to the kitchen lights entirely before you even walk in. The authors built a special computer chip (on an FPGA, which is like a programmable Lego board for electronics) that understands these switches. When the command says "brake," the chip skips 76% of the work. When it says "turn left," it skips about 71%. Because the chip physically stops the work from happening instead of just pretending to do it, it saves a massive amount of time and battery power.

The results are quite impressive. When they tested this on a simulated driving game called CARLA, the system became much faster. The time it took to make a decision dropped from 9.12 milliseconds down to between 3.74 and 4.44 milliseconds. That is a speedup of about 2.1 to 2.4 times! It also used less than half the energy, dropping from 263 millijoules to around 108–128 millijoules per decision. Crucially, the car didn't crash or get lost; it still completed 100% of the driving routes perfectly.

The paper also shows why this approach is better than older methods. They tried to just "prune" (cut out) parts of the brain permanently, like removing the drums from the orchestra forever. But that didn't work well because sometimes you need the drums, and sometimes you don't. If you remove them permanently, the orchestra can't play the songs that need them. The "Sparse by Command" system is smarter because it keeps all the musicians ready but only calls them in when the specific song needs them. They even tested this on a different type of computer brain (a Transformer) and found it worked there too, suggesting this idea could be used for many different kinds of smart machines, not just cars.

In short, this paper proves that if you give a computer a clear instruction about what it's supposed to do, you can tell it to skip all the busywork it doesn't need. By building a custom chip that listens to these instructions and physically skips the unnecessary work, they made a smart system that is faster, cheaper to run, and just as good at its job. It's a reminder that sometimes, the best way to be efficient isn't to work harder, but to know exactly what not to do.

Technical Summary: Sparse by Command

Problem Statement

Multi-task inference models, particularly in real-time autonomous systems, utilize a single neural network backbone to serve diverse operating modes (e.g., lane following, turning, braking). While these models share parameters, current hardware executes identical computations regardless of the active task. This uniform execution wastes energy and cycles on task-irrelevant operations, fetching weights and processing output channels that are not required for the specific command.

Existing solutions fall into two categories with distinct limitations:

Static Sparse Accelerators: Designs like SCNN and Cambricon-S exploit sparsity determined at training or compile time. These patterns are static and do not adapt to runtime task signals.
Dynamic Neural Networks: Approaches like SkipNet and BlockDrop learn input-conditional gating policies. However, on standard GPUs/CPU, "skipping" a block often still incurs overhead (fetching weights, allocating registers) because the hardware lacks alignment with the skipping granularity. Consequently, FLOP reductions do not translate to proportional speedups on real accelerators.

A critical gap exists: multi-task models possess a runtime signal—the task command—that is known before inference begins and remains stable for the duration of the task. Prior work has not exploited this signal to drive structured, hardware-aligned sparsity.

Methodology

The authors present a full-stack hardware/software co-designed system that bridges dynamic neural networks and sparse hardware accelerators. The approach consists of three integrated components:

1. Task-Conditional Gating Network

A lightweight Multi-Layer Perceptron (MLP) maps a discrete task command vector (e.g., "turn left," "brake") to a binary execution mask over the backbone's output feature tiles.

Granularity: The accelerator processes data in fixed-size "tiles" (groups of output channels). The gating network predicts a binary mask for each tile.
Training Pipeline: The system employs a three-phase training process:
- Phase 1: Train a dense backbone.
- Phase 2: Jointly train the backbone and gating network using a soft mask with a sparsity penalty ( $\lambda$ ) to encourage tile pruning.
- Phase 3: Binarize the masks using a hard threshold and the straight-through estimator, fine-tuning the backbone to operate under discrete tile execution.
Efficiency: The gating network adds less than 0.12% to the total parameter count and produces masks in nanoseconds, effectively amortizing the overhead to zero over the hundreds of frames a single command persists.

2. Instruction Set Architecture (NISA)

The authors introduce a Neural Instruction Set Architecture (NISA) that natively supports tile-level sparsity.

Encoding: Each 64-byte instruction includes a 128-bit bitmask field representing the active/inactive status of output feature tiles for that layer.
Execution: The hardware scheduler reads these masks. If a tile is masked (bit = 0), the tile manager skips the entire execution pipeline for that tile: no weight fetch, no activation load, and no computation. This skipping propagates to subsequent layers, as masked outputs are not loaded as inputs for the next layer.

3. Tiled Inference Accelerator

A custom FPGA accelerator (implemented on AMD/Xilinx Alveo U50) is designed to execute these masked instructions with zero overhead.

Architecture: Features a double-buffered memory system, INT8 datapath, and configurable parallelism (OC_PAR, IC_PAR, PP_PAR).
Tile Manager: Orchestrates DMA and compute pipelines. It checks the instruction bitmask before initiating DMA or compute for a tile. Skipped tiles cost only a single cycle for the bit check and shift register advance.
Zero Overhead: The datapath remains unchanged; sparsity is controlled entirely via instruction-level masks.

Key Contributions

Task-Conditional Gating Mechanism: A method to predict per-tile binary execution masks from a task command vector, trained jointly with the backbone to learn which tiles are dispensable for specific tasks.
Native Tile Mask ISA: An instruction set architecture that carries per-tile bitmask fields, allowing hardware to skip masked tiles entirely without software intervention or datapath modification.
Co-Designed Accelerator: A tiled inference accelerator with INT8 support and cycle-accurate performance counters that natively executes sparse tile instructions, demonstrating that tile-level sparsity requires hardware support to achieve speedup.
End-to-End Evaluation: A full-stack evaluation on an FPGA integrated with the CARLA autonomous driving simulator, demonstrating significant latency and energy reductions while maintaining driving quality.

Results

The system was evaluated on a closed-loop visuomotor driving task with six distinct commands (e.g., follow lane, turn left, brake).

Sparsity and FLOPs: Task-conditional sparsity reduces FLOPs by 66–76% across different commands. The pruning is task-dependent; for instance, "Brake" prunes more aggressively (18.7% active tiles) than "Turn Left" (31.3% active tiles).
Latency: On-device latency decreases by 51–59%, dropping from 9.12 ms (dense) to 3.74–4.44 ms (sparse), representing a 2.1–2.4× speedup.
Energy: Energy per inference drops from 263 mJ to 108–128 mJ.
Driving Quality: The sparse model maintains 100% route completion on hard evaluation routes with Cross-Track Error (CTE) comparable to or better than the dense baseline.
Comparison with Static Pruning: Static structured pruning collapses in performance at ~50% sparsity, whereas the task-conditional approach maintains fidelity at 69–81% sparsity.
Comparison with GPU: On an NVIDIA RTX 4090, applying the same masks results in a 22% slowdown compared to dense execution because the GPU still fetches weights and occupies cores. This highlights that tile-level sparsity only yields speedup when supported by dedicated hardware that skips fetch and compute.
Generality: The mechanism was successfully extended to a ViT-Base backbone, masking MLP tiles and achieving 40–62% FLOP reduction, confirming applicability beyond CNNs.

Significance and Claims

The paper claims that task-conditional sparsity is a distinct and effective paradigm that leverages the inherent structure of multi-task models. Unlike prior work that treats sparsity as a static property or relies on input-dependent gating that incurs per-input overhead, this approach uses the task command—a "free" signal available before inference—to drive structured, hardware-aligned skipping.

The authors emphasize that their contribution is not merely a new pruning algorithm, but a full-stack co-design that proves tile-level sparsity requires specific hardware support (ISA and accelerator) to translate FLOP reductions into actual latency and energy savings. They demonstrate that without this hardware alignment, sparsity mechanisms can degrade performance on general-purpose accelerators like GPUs. The work establishes a foundation for efficient, task-adaptive inference in edge autonomous systems where power and latency are critical constraints.