🤖 machine learning

SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

SelectInfer is a neuron-level optimization framework that enables efficient on-device Large Language Model inference by using an offline profiler to identify and selectively load and compute only the most important neurons, thereby significantly reducing memory and computational costs without compromising task performance.

Original authors: Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow, Lin Wang

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow, Lin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of knowledge, so vast that it contains the answer to almost any question you could ask. This is what scientists call a Large Language Model (LLM). Think of it like a super-smart robot brain that has read almost every book on the internet. But here's the catch: this brain is so huge and heavy that it usually needs a massive, expensive data center with powerful supercomputers to run. It's like trying to carry a full-sized library in your backpack; it just doesn't fit.

Recently, people have wanted to put this "library" into smaller devices, like the smart gadgets in your pocket or the computers in self-driving cars. These are called "edge devices." The problem is, these gadgets have very limited space and battery power. If you try to force the whole library onto them, the device crashes or runs so slowly it's useless. Scientists have tried to shrink the library by squishing the books (making them smaller but harder to read) or throwing away entire shelves (which makes the robot forget things). But these old tricks often break the robot's ability to think clearly. The big question is: Can we make this giant brain fit in a small backpack without losing its genius?

Enter SelectInfer, a new idea from researchers at Paderborn University that suggests a clever way to solve this puzzle. Instead of trying to shrink the whole library or throw away random books, SelectInfer acts like a very smart librarian who knows exactly which books you need right now.

Here is how it works, using a simple story. Imagine your robot brain is a massive factory with thousands of workers (called "neurons"). In a normal factory, every single worker shows up to work every day, even if they have nothing to do. This wastes space and energy. The researchers discovered that these workers actually fall into two groups. Some are "Generalists" who always show up, no matter what the job is—like the people who keep the lights on and the coffee machine running. Others are "Specialists" who only show up for specific tasks, like the team that fixes the plumbing or the team that paints the walls.

The paper shows that for any given task, you don't need all the specialists, and you definitely don't need the ones who aren't working today. SelectInfer uses a two-step magic trick to make the factory run on a tiny device:

Selective Loading (The Backpack Trick): Before the robot even starts working, the researchers use an "offline profiler" (a kind of scout) to figure out which Generalists and Specialists are the most important. Then, when the robot is ready to go, it only packs those specific workers into its backpack (memory). It leaves the rest behind. This is like only packing the tools you need for a camping trip instead of the whole hardware store. This allows the robot to fit into devices that previously were too small to hold it at all. For example, on a device with only 8 GB of memory (like an NVIDIA Jetson Orin Nano), a model that usually needs 9 GB of space can now squeeze in because it's only carrying the essential 70% of its workers.
Selective Computation (The On-Demand Trick): Even with a lighter backpack, the robot still has to ask every worker to do their job, which takes time. SelectInfer adds a second rule: once the robot starts working, it only asks the most relevant workers to actually do the math for that specific moment. If the robot is writing a poem, it doesn't need the plumbing team to calculate the rhyme scheme. It only wakes up the poets. This makes the robot think much faster.

The researchers tested this on three different robot brains (Llama3.2-3B, Llama3.2-1B, and Qwen2.5-3B) on a tiny computer chip. They found that by using this method, they could run these giant models on devices that were previously impossible to use. For the 3-billion-parameter model, SelectInfer made it run about 1.53 times faster than the current best method of squishing the data (4-bit quantization) and more than 13 times faster than trying to pull data from a slow disk.

Crucially, the paper argues that this doesn't make the robot "dumber." In fact, for many tasks like translation and summarizing stories, SelectInfer was actually more accurate than other methods that tried to save space. The researchers found that while other methods either saved memory but ran slowly, or ran fast but forgot things, SelectInfer managed to balance all three: it saved memory, sped up the thinking, and kept the answers accurate.

The paper explicitly rules out the idea that you need to throw away the whole model or retrain it from scratch to make it fit. It also shows that simply picking random workers to leave behind (random loading) causes the robot to fail completely, proving that you need a smart map to know who to keep. While the method requires a one-time "scouting" phase to figure out which workers are important, the results suggest this is a practical and powerful way to bring super-smart AI to the small gadgets in our pockets without needing a supercomputer in the cloud.

Technical Summary: SelectInfer

Problem Statement

Large Language Models (LLMs) face significant barriers to deployment on resource-constrained edge devices (e.g., NVIDIA Jetson Orin Nano) due to high memory and computational demands. Existing optimization strategies fail to simultaneously address the three critical dimensions of edge deployment: memory footprint, inference speed, and output accuracy.

Quantization reduces memory but often degrades decoding throughput and offers only coarse-grained control (e.g., 4-bit vs. 8-bit).
Pruning requires costly retraining and permanently removes weights, lacking runtime adaptability.
Offloading (e.g., to disk) is bottlenecked by I/O bandwidth and often incompatible with unified memory architectures found in edge SoCs.
Runtime Sparsity methods (e.g., CoreInfer) improve speed by skipping inactive neurons but require the full model to reside in memory, preventing the deployment of larger models on memory-constrained hardware.

Methodology

SelectInfer is a neuron-level optimization framework that enables efficient LLM inference through selective loading and selective computation, guided by an offline profiling phase. The core insight is that LLM neurons exhibit a dual structure: a stable core of model-specific (general-purpose) neurons activated universally across tasks, and task-specific neurons that drive performance for individual inputs.

1. Offline Phase: LLM Profiler

Before deployment, SelectInfer utilizes an offline profiler to analyze neuron activations across diverse datasets (e.g., QA, translation, summarization).

Scoring: For each neuron $n$ , a score $s_n$ is calculated based on the sum of absolute activations over a set of tokens $T$ : $s_n = \sum_{t \in T} |a_n(t)|$ .
Classification: Neurons are categorized into two groups:
- Base Neurons: Task-specific neurons identified by merging scores from prompts within a specific task.
- Secondary Neurons: Model-specific neurons identified by merging scores across different tasks.
Output: The profiler generates ranked neuron files identifying the most important neurons for specific tasks and general model behavior.

2. Online Phase: Optimization

During runtime on the edge device, SelectInfer applies two mechanisms:

Selective Loading (Memory Optimization):
Instead of loading the full model, SelectInfer loads only a subset of neurons into memory. It prioritizes base neurons (task-specific) and fills the remaining memory budget with secondary neurons (model-specific) up to a defined loading threshold $\gamma$ . This allows models that would otherwise exceed device memory (e.g., 3B models on 8GB devices) to run by discarding irrelevant weights entirely.
Selective Computation (Speed Optimization):
Even with a reduced memory footprint, computing all loaded neurons can be slow. SelectInfer further restricts computation to a subset of the loaded neurons ( $\epsilon < \gamma$ ).
- Static Neurons: Correspond to the base neurons; they are always computed regardless of the input.
- Dynamic Neurons: Selected at runtime during the prefill phase. The system computes activation values for all loaded neurons, sorts them, and selects the top $\alpha$ neurons per token. It then aggregates these to find the top $\beta$ most frequent neurons for the prompt.
- The total computation ratio $\phi = \delta + \epsilon$ is user-configurable, allowing a trade-off between speed and accuracy.

Key Contributions

Offline Profiler: A mechanism to identify and rank model-specific and task-specific neurons, establishing a "dual structure" of LLM activation.
Selective Loading: A deployment strategy that loads only relevant neurons, enabling the execution of models larger than the device's available memory.
Selective Computation: A runtime mechanism that dynamically computes only the most relevant neurons (static + dynamic), accelerating inference without significant accuracy loss.
Tunable Framework: A system that treats memory, speed, and accuracy as independently tunable parameters, offering fine-grained control compared to the coarse knobs of quantization or pruning.

Evaluation Results

The framework was evaluated on the NVIDIA Jetson Orin Nano (8 GB shared memory) using Llama3.2-1B, Llama3.2-3B, and Qwen2.5-3B models across QA, translation, and summarization tasks.

Feasibility: SelectInfer enabled 3B-scale models (requiring ~8.5–9.5 GB natively) to run on the 8 GB Jetson device by loading only 70% of neurons.
Memory Footprint: SelectInfer reduced memory usage by ~13.2% for Llama3.2-3B (from 6.88 GB to 5.97 GB) compared to the original model, and significantly more compared to CoreInfer (which requires full model loading). While 4-bit quantization achieved lower absolute memory usage, SelectInfer maintained higher precision for loaded weights.
Accuracy: SelectInfer maintained competitive accuracy. On translation tasks (WMT16), SelectInfer achieved ~~25–26 BLEU for Llama3.2-3B, significantly outperforming CoreInfer (~~4–5 BLEU) and CoreInfer with random loading. It generally outperformed or matched 4-bit quantization on translation and summarization tasks.
Speed: SelectInfer achieved a decoding throughput of ~9.85 tokens/s for Llama3.2-3B. This represents a 1.53× improvement over 4-bit quantization and a 13× improvement over disk-offloading approaches.
Comparison: SelectInfer outperformed CoreInfer in accuracy (due to offline profiling) and memory efficiency (due to selective loading). It offered better speed and accuracy than quantization on translation tasks while avoiding the hardware limitations of low-precision kernels.

Significance and Claims

The paper claims that SelectInfer represents a practical step toward deploying LLMs on edge devices by solving the "trilemma" of memory, speed, and accuracy. Unlike prior work that optimizes only one or two dimensions, SelectInfer provides a balanced and tunable operating point across all three.

The authors emphasize that their approach:

Requires no retraining, distinguishing it from pruning and distillation.
Is compatible with unified memory architectures, unlike offloading methods.
Offers fine-grained control, allowing users to adjust the percentage of loaded and computed neurons to match specific device constraints and application needs.
Generalizes across model families, demonstrating effectiveness on both Llama and Qwen architectures.

The work positions SelectInfer as a necessary evolution beyond coarse-grained compression, enabling the execution of larger, more capable models on hardware previously deemed insufficient for such tasks.