🤖 machine learning

Kilobyte Models: Neural Networks as a Seed and a Quantized Latent

This paper introduces "Kilobyte Models," a compression technique that stores neural networks as a compact quantized latent vector and a reproducible integer seed instead of full weights, enabling extreme storage efficiency while maintaining accuracy comparable to aggressive weight quantization.

Original authors: Sahil Rajesh Dhayalkar

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sahil Rajesh Dhayalkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a massive, intricate blueprint for a robot to a friend on the other side of the world. The blueprint is so huge that it would take days to mail, and your friend's mailbox is tiny. In the world of artificial intelligence, these "blueprints" are called neural networks, and they are the brains behind everything from photo filters to self-driving cars. Usually, to send a brain, you have to send every single number that makes it work—millions or even billions of them. This is a huge problem for devices like smartwatches or phones that have limited storage space or slow internet connections.

For a long time, scientists tried to solve this by shrinking the numbers themselves (making them less precise) or cutting out the ones that seemed less important. But there's a different way to think about it: instead of sending the whole blueprint, what if you just sent a tiny "recipe" or a secret code? If your friend has a standard set of tools (a random generator) and the recipe, they can rebuild the exact same robot in their own kitchen. This idea relies on the fact that these AI brains don't actually need billions of unique numbers to work; they often hide in a much smaller, simpler space. The big question is: Can we shrink a giant AI brain down to the size of a few text messages without losing its smarts?

This paper introduces a clever new method called "Kilobyte Models" that attempts to do exactly that. Instead of storing the actual weights (the numbers that make the AI think), the author proposes storing just two tiny things: a random "seed" (like a starting number for a game) and a very short, compressed "latent" vector (a tiny list of instructions). When the AI needs to run, the device uses the seed to regenerate the giant, random "scaffolding" it needs, and then uses the tiny list of instructions to tweak that scaffolding into the final brain.

The researchers found that this approach works surprisingly well. In their experiments, they managed to shrink a neural network down to just a few kilobytes—about the size of a short text message—while keeping it almost as smart as the original. For example, they compressed a model that recognizes handwritten numbers (MNIST) into just 2 KB and it still got 98.6% of the answers right. Even more impressively, they showed that this method is much better at handling extreme compression than traditional methods. While a standard AI model might crash and become useless if you try to shrink it down to 4 bits (a very tiny amount of data), their "Kilobyte Model" stayed strong and accurate.

The paper also explored using this as a "delta" or a tiny update for a giant, pre-trained AI. Imagine you have a super-smart AI that knows everything about animals, and you want it to learn specifically about cats. Instead of sending the whole new brain, you just send a 4 KB "patch" (the seed and the tiny latent vector) that tells the giant brain how to shift its focus. This patch was thousands of times smaller than the full model but still allowed the AI to learn the new task effectively.

However, the author is careful to note that this isn't a magic wand for everything. The method works best when the "recipe" (the latent vector) is big enough to hold the necessary instructions; if the task is too hard or the recipe too small, the AI gets a bit confused. Also, while the file you send is tiny, the computer still has to rebuild the full brain in its memory to run it, so it doesn't make the computer run faster, just easier to carry. Finally, this trick currently works for standard AI networks but hasn't been tested on the massive "Large Language Models" that power chatbots yet.

In short, the paper suggests that by trading the storage of millions of numbers for a tiny seed and a short instruction list, we can fit powerful AI brains into spaces as small as a few kilobytes. It's like realizing you don't need to mail the whole library to a friend; you just need to mail them a specific book number and a few notes, and they can print the exact book they need using their own printer.

Technical Summary: Kilobyte Models

Problem Statement

The storage and transmission costs of trained neural networks scale linearly with their parameter count, creating a critical bottleneck for bandwidth-constrained deployments such as edge devices, over-the-air updates, and massively scaled adapters. While standard compression techniques (pruning, quantization, low-rank factorization, distillation) reduce the description of weight tensors, they still require storing a significant portion of the original parameter set. The author identifies a need for an extreme compression paradigm where the deployable artifact is not the weights themselves, but a compact "recipe" for regenerating them.

Methodology

The paper introduces Kilobyte Models, a framework that represents a neural network as a deterministic integer seed and a short, quantized latent vector. This approach builds upon Mapping Networks (Sen and Mukherjee 2026), which express network weights as a nonlinear function of a compact trainable latent vector and a fixed random basis.

Core Mechanism

Reparameterization: Instead of optimizing weights $\theta$ directly, the model optimizes a latent vector $z \in \mathbb{R}^d$ (where $d \ll P$ , and $P$ is the total parameter count). The weights are generated via a fixed map $g(z)$ :
$g(z) = \sigma(W_0 z + \alpha \|z\|_2^2 \mathbf{1} + b_0)$
Here, $W_0$ is a fixed projection matrix, $b_0$ is a fixed initialization center, and $\sigma$ is a nonlinearity (tanh).
Seed-Based Regeneration: The fixed components ( $W_0$ and $b_0$ ) are not stored. Instead, they are deterministically regenerated from an integer seed $s$ using a reproducible, block-wise random basis. This allows the receiver to reconstruct the full projection matrix without storing it, provided they possess the seed.
Artifact Composition: The deployable artifact $A$ $A$ consists of:
- Seed ( $s$ ): A 64-bit integer (8 bytes) used to regenerate the basis.
- Quantized Latent ( $Q_b(z^*)$ ): The trained latent vector quantized to $b$ bits.
- Normalization Parameters ( $\theta_{norm}$ ): A small set of normalization affine parameters (scale/shift) stored verbatim, as they lie outside the range of the generation map.
Scalable Basis Generation: To handle networks where the projection matrix exceeds device memory, the author employs a block-wise generation strategy. The matrix is constructed row-by-row using a pseudo-random generator seeded by a combination of the master seed and the block index, avoiding full matrix materialization.
Fine-Tuning as a Delta: The framework extends to fine-tuning by treating the latent vector as a compressed adapter. The map generates a displacement from a frozen pretrained base ( $\theta_{ft} = \theta_{pre} + W_0 z$ ), requiring only the seed and the quantized delta to be transmitted.

Training Regimes

Single Latent Vector Training (SLVT): A single latent vector generates all parameters.
Layer-Wise Training (LWT): Separate latent vectors are assigned to each layer, allowing for finer control and allocation of latent budget proportional to layer difficulty.
Quantization Strategies: The paper evaluates both Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT), where the latent is fine-tuned with simulated rounding to survive aggressive bit-width reduction.

Key Contributions

Formalization of the Kilobyte Model: The author defines a deployable artifact sized by latent dimension $d$ and bit width $b$ rather than parameter count $P$ , formalizing the trade-off between storage and capacity.
Scalable Seeded Basis: Introduction of a block-wise, reproducible random basis that enables weight regeneration for large networks without requiring the projection matrix to reside in memory.
Fine-Tuning Adaptation: Extension of the parameterization to create kilobyte-sized adapters for pretrained models, offering a hyper-compressed alternative to standard parameter-efficient fine-tuning methods.
Empirical Validation: Demonstration that Kilobyte models can match the accuracy of aggressively quantized networks while occupying a fraction of the storage space.

Experimental Results

The author evaluated the approach on Convolutional Neural Networks (CNNs), LSTMs, and Multi-Layer Perceptrons (MLPs) across various datasets (MNIST, FashionMNIST, Beijing Air Quality, HIGGS, Covertype, Oxford-IIIT Pets).

Extreme Compression: A 4-bit mapped model achieved 98.6% accuracy on MNIST using only 2 KB of storage (a 26 $\times$ reduction over a quantized baseline).
Regression Tasks: On a time-series regression task, a 4-bit QAT model achieved an MSE of 0.00061 using just 140 bytes, roughly 370 $\times$ smaller than the full-precision model.
Fine-Tuning Efficiency: When fine-tuning a ResNet-50 on the Oxford-IIIT Pets dataset, a 4-bit QAT adapter reached 88.4% accuracy in just 4 KB. In contrast, a full ResNet-50 compressed to 4-bit PTQ collapsed to 2.9% accuracy.
Comparison to SOTA: The Kilobyte adapter outperformed VeRA (a parameter-efficient method) at strict kilobyte budgets (4 KB vs. 13.9 KB for similar accuracy). While LoRA and VeRA achieved higher accuracy with larger storage budgets (up to 0.6 MB), the Kilobyte model maintained superior performance in the ultra-low storage regime.
Quantization Impact: Quantization-Aware Training (QAT) was found to be critical for maintaining accuracy at low bit widths (4-bit), whereas Post-Training Quantization (PTQ) suffered significant performance drops.

Significance and Claims

The paper claims that Kilobyte models represent a powerful strategy for bandwidth-constrained deployments by decoupling the storage size of a model from its parameter count. By shifting the compression burden entirely to a compact latent space and leveraging reproducible random bases, the author demonstrates that models can be stored and transmitted in kilobytes while retaining high accuracy.

The author positions this work as a complement to, rather than a replacement for, standard compression techniques. While standard methods shrink weight tensors, Kilobyte models avoid representing the full weight set during optimization entirely. The framework is particularly significant for scenarios where storage size is the binding constraint (e.g., over-the-air updates for edge devices), offering a 5 to 160 $\times$ reduction in artifact size compared to full fine-tuning or standard parameter-efficient methods, albeit with a slight accuracy trade-off on complex datasets.

Limitations Acknowledged:

Inference Compute: The method does not compress inference latency or memory usage, as weights are regenerated as dense floating-point values before inference.
Capacity Bottleneck: Accuracy is limited by the latent dimension $d$ , leading to slight performance gaps on harder datasets compared to full networks.
Dependency: Fine-tuning requires the base pretrained model to be present on the deployment device.
Architecture Scope: The current implementation does not extend to Transformers or Large Language Models.