🤖 AI

NeuronFabric: A Software Reference Architecture for On-Chip Transformer Training with Local Adam

This paper introduces NeuronFabric, a software reference architecture and C# prototype for on-chip transformer training with local Adam updates, which validates numerical correctness and demonstrates that a BF16W configuration (BF16 weights with FP32 optimizer moments) reduces memory requirements to fit within the BRAM capacity of a Xilinx ZCU102 device for future FPGA and ASIC implementations.

Original authors: Evgeny Ukladchikov

Published 2026-06-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Evgeny Ukladchikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: A Self-Contained Learning Machine

Imagine you are trying to teach a robot to write Shakespeare. Currently, most AI training works like a student with a very slow teacher.

The student (the chip) reads a sentence and tries to guess the next word.
The student makes a mistake and calculates how they were wrong.
The student has to run all the way to the teacher's office (the host computer) to get the answer key.
The teacher updates the student's notes, and the student runs back to try again.

This "running back and forth" is slow and wastes a lot of energy.

NeuronFabric proposes a different idea: A student who keeps the answer key in their own pocket.
In this system, the chip doesn't just guess; it calculates its own mistakes, looks at its own notes, and updates its own memory instantly, all without ever leaving the room. The paper proves that this "self-updating" math works correctly in software, setting the stage for a future chip that can do this physically.

Key Concepts Explained

1. The "Local Adam" (The Self-Correction Mechanism)

In standard AI, the "optimizer" (the part that fixes mistakes) usually lives on a separate computer. NeuronFabric puts the optimizer inside every single neuron.

Analogy: Imagine a classroom where every student has their own private whiteboard. When they get a question wrong, they don't wait for the teacher to come over. They immediately erase their mistake, write the correction, and move on.
The Paper's Claim: The authors built a software version of this in C# (a programming language). They proved that when every neuron updates itself locally, the math still works, and the robot learns to write coherent text.

2. The "BF16W" Trick (The Backpack Strategy)

To fit this self-updating system onto a small chip (like an FPGA), memory is the biggest problem.

The Problem: Usually, to learn, a robot needs to carry three heavy backpacks for every piece of knowledge:
1. The knowledge itself (Weights).
2. A record of how fast it learned (Momentum).
3. A record of how much it varied (Variance).
- If the backpacks are too heavy, the robot can't fit them all in its small room (SRAM).
The Solution (BF16W): The authors realized they could shrink the "Knowledge" backpack. Instead of carrying a heavy, high-precision backpack (32-bit), they carry a lightweight, slightly less precise one (16-bit).
The Result: They kept the "learning records" heavy and precise, but made the "knowledge" backpack half the size.
- Analogy: It's like packing for a trip. You keep your expensive, fragile jewelry (the learning stats) in a heavy, secure box. But you swap your heavy winter coat (the weights) for a thin, lightweight jacket. You save enough space in your suitcase to fit your sleeping bag (the activation buffers) so you don't have to sleep on the floor.
The Paper's Claim: This trick saved just enough space to fit the entire learning system onto a mid-range chip (ZCU102) without needing to connect to external memory.

3. The "Vocabulary Budget" (The Room for Thought)

The authors discovered a hidden trap when building small AI models.

The Trap: If you have a tiny budget for memory, and you try to teach the robot a huge dictionary (vocabulary), the dictionary takes up almost all the space. There is no room left for the robot to actually think or learn grammar.
Analogy: Imagine you have a small notebook. If you spend 90% of the pages just writing down the alphabet and a dictionary, you only have a few pages left to write a story. The story will be nonsense.
The Fix: The authors used a "byte-level" approach (treating every character as a single token) and a small vocabulary (256 characters). This meant the "dictionary" only took up a small corner of the notebook, leaving plenty of room for the robot to learn how to write Shakespearean sentences.
The Paper's Claim: They showed that if you don't manage this "vocabulary tax," small models fail to make sense, no matter how good the math is.

4. The "No-Host" Goal (The Independent Chip)

The ultimate goal of this research is a chip that doesn't need a computer (CPU) to tell it how to learn.

Current State: The paper describes a software prototype. It runs on a standard computer to prove the math is right.
Future State: The goal is to put this exact same software logic onto a physical chip (FPGA).
The Claim: If they can get the physical chip to do this, it will be able to train itself on new data without ever sending data to a server or waiting for a computer to update its weights. It would be a truly "on-chip" learning machine.

What Did They Actually Achieve?

They built a proof-of-concept: They wrote a program in C# that trains a small AI model (334,000 parameters) entirely on its own.
They proved the math works: The AI learned to generate text that looks like Shakespeare (e.g., "HAMLET: Break him of him..."). It wasn't perfect, but it was coherent.
They proved the memory math works: They calculated exactly how much space is needed and showed that with their "lightweight jacket" (BF16W) trick, the whole system fits on a specific chip (ZCU102) with room to spare.
What they did NOT do: They did not build the physical chip yet. They did not measure the speed or power of the chip. They only measured the software running on a normal computer.

Summary

Think of this paper as the blueprint and the simulation for a new type of car engine.
The author says: "I have designed an engine where the fuel injection and spark plugs talk directly to each other without a central computer. I have built a computer simulation of this engine, and it drives perfectly. I have also calculated that this engine is small enough to fit in a compact car. Now, I need to build the actual metal engine to prove it runs."

The paper is the proof that the design is sound; the physical chip is the next step.

Technical Summary: NeuronFabric

Problem Statement

Current publicly documented neural accelerators generally separate the compute phase from the optimizer state update. Inference-only chips (e.g., Groq LPU, Apple ANE, IBM NorthPole) lack on-chip gradient computation, while training accelerators (e.g., Cerebras, Tenstorrent) typically offload weight updates to a host CPU or external GPU. This architectural separation creates significant bottlenecks:

Weight Traffic: Inference requires loading all weights from external memory (HBM) for every forward pass.
Training Latency: The chip computing gradients does not apply them; updates are orchestrated separately, often requiring data movement off-chip.
Continuous Learning Limitations: Inference chips cannot update weights from new data without reloading the entire model from an external cluster, a process taking minutes to hours.

The paper posits that a silicon chip where every neuron holds its own weights and runs the Adam optimizer locally could minimize off-chip traffic and enable true on-chip training. However, before committing to silicon, the mathematical validity and numerical correctness of such a "local Adam" architecture must be established.

Methodology

The authors developed NeuronFabric, a complete software reference architecture implemented in C# without external machine learning frameworks. The system is designed so that every software abstraction maps directly to a hardware block (e.g., NeuronCore to BRAM, Forward() to DSP48 chains).

Core Architecture:

Local Adam: Each NeuronCore stores its own weight vector $w$ and bias $b$ . During the backward pass, it computes gradients and updates weights in-place using the Adam algorithm. There is no weight bus between cores.
Model Configuration: A 334K parameter autoregressive transformer (Pre-LN) trained on the Shakespeare corpus.
- Dimensions: Embedding $d=88$ , 4 heads, Feedforward $f=264$ , 4 layers.
- Vocabulary: 256 (byte-level).
- Techniques: Weight tying (output projection reuses the embedding matrix) to save parameters.

BF16W Precision Scheme:
To address the strict SRAM constraints of FPGA training, the authors introduced BF16W (Brain Floating Point 16 Weights).

Storage: Weights are stored as BF16 (2 bytes).
Computation: Weights are cast to FP32 for forward/backward computation and Adam moment updates.
Moments: The Adam first ( $m$ ) and second ( $v$ ) moments remain in FP32 (4 bytes each) to ensure stability.
Update: After the Adam step, the updated FP32 weight is rounded back to BF16 and written to storage.

Vocabulary-Budget Constraint:
The paper identifies a critical constraint for small-parameter models: the embedding table ( $|V| \times d$ ) consumes a disproportionate share of the parameter budget ("vocabulary tax"), leaving insufficient capacity for the transformer layers to learn structural patterns. This was quantified across three domains, showing that when the remaining reasoning capacity ( $P_{reason}$ ) drops below ~20K, coherent text generation fails.

Key Contributions

Software Reference Implementation: A complete, framework-free C# implementation of a transformer with full backpropagation and local Adam updates, running training, evaluation, and checkpointing in a single binary.
BF16W Validation: Demonstration that storing weights in BF16 while keeping moments in FP32 reduces SRAM usage from 12 bytes/parameter to 10 bytes/parameter with a negligible convergence penalty (+0.020 validation loss on Shakespeare).
Vocabulary-Budget Quantification: Empirical evidence showing that for fixed-budget hardware, the embedding table dominates the model, necessitating either small vocabularies or specialized multi-chip architectures (e.g., separate embedding chips).
FPGA Feasibility Target: A concrete configuration (334K BF16W model) that fits entirely within the 4.0 MB on-chip BRAM of a Xilinx ZCU102 FPGA, leaving 660 KB of headroom for activation buffers, without requiring DDR access during training.

Results

The system was validated on a Windows 11 machine (AMD Ryzen 9 9900X CPU, NVIDIA RTX 4090 GPU) using the Shakespeare corpus (80K samples).

Correctness: Numerical gradient checks (finite differences vs. analytical gradients) passed for all components, confirming the mathematical correctness of the local Adam implementation.
Convergence:
- GPU FP32 (Oracle): Achieved a validation loss of 1.5224.
- CPU BF16W: Achieved a validation loss of 1.5426.
- The gap of +0.020 is attributed to the precision difference between BF16 weight storage and FP32 computation.
Generative Quality: The model produced coherent character-level text with recognizable Shakespearean dialogue structures and named-character patterns, confirming that the training loop converges to a functional solution.
SRAM Efficiency:
- FP32 Adam (12 bytes/param): 4.00 MB total (fills ZCU102 BRAM exactly, leaving zero room for activations).
- BF16W Adam (10 bytes/param): 3.34 MB total (leaves 660 KB free for activations and control state).

Significance and Claims

The paper explicitly states its modest scope: NeuronFabric is a research prototype, not a production accelerator. It does not contain FPGA measurements or timing closure results.

The paper's primary significance lies in:

Establishing Prior Art: It serves as a public architectural disclosure and software reference to establish the "local Adam update" architecture as prior art for future FPGA/ASIC implementations.
Proof of Concept: It demonstrates that the math works and the numbers are right for a transformer that trains entirely on-chip with local optimizer state, without a host CPU in the update loop.
Defining the Path Forward: It establishes the 334K BF16W Shakespeare model as a falsifiable target for future FPGA implementation. If the same loss curve can be achieved on the FPGA without host intervention, the viability of on-chip transformer training becomes a measured fact rather than a software projection.

The authors conclude that the next immediate step is to implement this specific training loop on the ZCU102 FPGA to verify that the software results translate to hardware, thereby validating the "no host CPU" training paradigm.