💬 NLP

MiniGPT: Rebuilding GPT from First Principles

This paper presents MiniGPT, a compact, single-notebook PyTorch implementation that rebuilds the core GPT autoregressive pipeline from first principles to demonstrate the training and generation capabilities of character-level language models on the Tiny Shakespeare dataset without introducing new architectural innovations.

Original authors: Jibin Joseph

Published 2026-05-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jibin Joseph

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to write like William Shakespeare. You could buy a pre-made, super-smart robot that already knows everything, but that doesn't help you understand how it works. Or, you could try to build a tiny robot from scratch, using only the basic tools of math and logic, to see if you can get it to mimic the Bard's style.

That is exactly what this paper, MiniGPT, is about. The author, Jibin Joseph, didn't invent a new kind of robot brain. Instead, he built a small, simple version of a famous AI architecture (GPT) from the ground up to show exactly how the pieces fit together.

Here is a breakdown of the paper using simple analogies:

1. The Goal: Building a "Toy" Brain

Think of modern AI models as massive, complex skyscrapers. They are amazing, but it's hard to see how the plumbing and wiring work inside.

The Paper's Approach: MiniGPT is like building a LEGO model of that skyscraper. It's small, it fits on a desk, and every single brick is placed by hand. The goal isn't to build a building that can house a city; it's to prove you can build a working structure from the very first brick without using a pre-fabricated kit.

2. The Training Data: The "Tiny Shakespeare"

To teach this robot, the author used a very small dataset called "Tiny Shakespeare."

The Analogy: Imagine you are trying to teach a child to speak by reading them one single, short storybook over and over again. You aren't giving them the entire library of human knowledge; just one book.
The Method: The robot doesn't read whole words like "Romeo" or "love." Instead, it reads letter by letter (character-level). It sees "R", then "o", then "m", then "e", then "o".
- Why do this? It makes the job harder (the robot has to figure out where words start and end), but it makes the code much simpler to understand because you don't need a complex dictionary to translate the text first.

3. The Learning Process: The "Next Letter" Game

The robot learns by playing a simple game: "Guess the next letter."

How it works: The robot sees the letters "H", "e", "l", "l". It has to guess the next letter. If it guesses "o", it gets a point. If it guesses "z", it loses a point.
The "Causal Mask": This is a crucial rule. The robot is only allowed to look at the letters before the current spot. It cannot peek at the future. It's like a student taking a test who can only look at the questions they have already answered, not the ones coming up next. This forces the robot to learn patterns based on what came before.

4. The Two Experiments: The "Small" vs. The "Stronger"

The author ran two different versions of this experiment to see how size and settings matter.

The Baseline (The Small Robot):
- This was a tiny brain with 4 layers of "thinking" and about 800,000 parameters (the knobs and dials the robot adjusts).
- Result: It learned the basics. After 3,000 tries, it got pretty good at guessing the next letter, reaching a "loss" (a score of mistakes) of 1.72. It proved the whole system works.
The Stronger Configuration (The Bigger Robot):
- This robot was bigger: 6 layers, 10.77 million parameters, and it looked at twice as many letters at once (context length).
- Result: It learned much faster and made fewer mistakes (loss dropped to 1.47).
- The Catch (Overfitting): The paper noticed something interesting. After a certain point (step 1750), the robot started memorizing the book instead of learning the patterns. It got perfect at guessing the next letter in the training book, but if you showed it a new sentence, it would stumble.
- The Lesson: The author had to stop training early and save the "best" version of the robot, rather than waiting until the very end. It's like a student who memorizes the practice test answers perfectly but fails the real exam because they didn't understand the concepts.

5. The Output: What Does the Robot Write?

When the author asked the "Stronger" robot to write a story starting with "ROMEO:", what happened?

The Good: The robot started writing in a way that looked like Shakespeare. It used capital letters for names, added colons, put in line breaks, and used punctuation correctly. It learned the "dance moves" of the language.
The Bad: The actual meaning was often nonsense. It might write, "So did let us continue them home," which sounds Shakespearean but doesn't make logical sense.
The Takeaway: The robot learned the style (the formatting and rhythm) but not the deep meaning. It's like a parrot that can perfectly mimic the sound of a human conversation without understanding what the words mean.

6. The "Temperature" Knob

The paper also played with a setting called "Temperature," which controls how creative or safe the robot is.

Low Temperature (0.7): The robot is very safe and repetitive. It picks the most obvious next letter every time. The text is stable but boring.
High Temperature (1.2): The robot takes risks. It picks less likely letters. The text becomes more creative and varied, but it also starts making up weird words and spelling things incorrectly.

Summary of the Paper's Claims

What it is: A clear, step-by-step guide on how to build a small AI language model from scratch in Python.
What it proved: You can build a working model that learns to predict text letter-by-letter.
What it found: Bigger models learn faster and better, but they can easily memorize the training data if you aren't careful.
What it is NOT: It is not a new invention, it is not a super-smart AI, and it is not ready to write novels or replace human writers. It is a teaching tool to show how the magic of AI actually works under the hood.

In short, MiniGPT is the "kitchen recipe" for AI. It doesn't claim to be a Michelin-star meal, but it shows you exactly how to chop the onions, mix the batter, and bake the cake so you understand the process.

Technical Summary: MiniGPT – Rebuilding GPT from First Principles

Problem Statement

While high-level explanations and diagrams of Generative Pre-trained Transformer (GPT) models are abundant, the core pipeline from raw text to autoregressive generation often remains obscured by engineering complexities in large-scale implementations. Existing compact references like nanoGPT are valuable, but the authors argue that rebuilding the core concepts independently in a single, self-contained environment (a Jupyter notebook) provides a more direct and reproducible path for understanding the full lifecycle of a GPT-style model. The goal is not to propose a novel architecture or achieve state-of-the-art performance, but to demystify the implementation details of a decoder-only Transformer trained on character-level data.

Methodology

The paper presents MiniGPT, a compact, from-scratch implementation of a GPT-style autoregressive language model written entirely in PyTorch. The implementation is designed to be a single-notebook reconstruction of the following components:

1. Architecture

MiniGPT follows the standard decoder-only Transformer design:

Tokenization: Character-level tokenization is used, where every unique character in the dataset maps to a unique integer ID. This results in a small vocabulary (65 characters for the Tiny Shakespeare dataset) and eliminates the need for external subword tokenizers.
Embeddings: The model utilizes learned token embeddings and learned positional embeddings to encode sequence order, as self-attention alone does not inherently capture position.
Transformer Blocks: The core consists of a stack of pre-LayerNorm Transformer blocks. Each block contains:
- Causal Multi-Head Self-Attention: Uses scaled dot-product attention with a lower-triangular causal mask ( $M_{ij} = -\infty$ for $i < j$ ) to prevent tokens from attending to future positions.
- Feed-Forward MLP: A two-layer network expanding the embedding dimension by a factor of four, using GELU activation and dropout.
- Residual Connections: Applied around both the attention and MLP submodules to facilitate optimization in deeper networks.
Output Head: A final LayerNorm followed by a linear layer projecting hidden states to vocabulary logits. In the stronger configuration, weight tying is applied between the input token embedding matrix and the output language-model head to reduce parameter count.

2. Training Pipeline

The training process converts raw text into next-token prediction tasks:

Data Preparation: The text is split into fixed-length blocks (context length). The input sequence ( $x$ ) is a window of tokens, and the target sequence ( $y$ ) is the same window shifted by one position.
Optimization: The model is trained using cross-entropy loss.
- Baseline: Uses AdamW with a fixed learning rate ( $3 \times 10^{-4}$ ), batch size 32, and 3000 iterations.
- Stronger Configuration: Employs AdamW with parameter grouping (applying weight decay only to 2D+ tensors), a warmup phase followed by cosine learning rate decay, gradient clipping, mixed-precision training (FP16), and checkpoint selection based on the lowest validation loss.

3. Generation

Autoregressive text generation is performed by sampling from the predicted probability distribution of the next token. The implementation supports:

Temperature Sampling: Adjusting the sharpness of the probability distribution.
Top-k Sampling: Filtering the $k$ most likely tokens before sampling (though in this specific character-level experiment with $V=65$ , $k=200$ effectively disables truncation).

Key Contributions

The paper explicitly lists the following contributions:

From-Scratch Implementation: A complete PyTorch implementation of a small decoder-only GPT-style model, written independently without copying code from existing repositories.
End-to-End Pipeline: A documented path from raw text processing to training, validation, and autoregressive generation within a single notebook.
Experimental Analysis: Baseline and stronger experiments on the Tiny Shakespeare dataset, including detailed loss tracking and validation-based checkpoint selection.
Qualitative Analysis: An evaluation of generated samples to assess the model's ability to learn local structure (speaker names, punctuation, dialogue formatting) versus long-range coherence.
Practical Discussion: A candid analysis of limitations regarding model scale, tokenization efficiency, overfitting, and sampling behavior.

Results

The paper evaluates two configurations on the Tiny Shakespeare dataset:

Baseline Model (0.83M parameters):
- Configuration: 4 layers, 4 heads, 128 embedding dimension, context length 128.
- Performance: After 3000 iterations, it achieved a validation loss of 1.7236 (perplexity ~5.60).
- Outcome: Confirmed the correctness of the training loop, validation logic, and loss computation.
Stronger Model (10.77M parameters):
- Configuration: 6 layers, 6 heads, 384 embedding dimension, context length 256, with advanced regularization and scheduling.
- Performance: Reached a best validation loss of 1.4780 (perplexity ~4.38) at step 1750.
- Overfitting Observation: The model began to overfit after step 1750; while training loss continued to drop to 0.6105 by step 5000, validation loss increased to 1.7055. This highlighted the necessity of validation-based checkpoint selection.
- Generation Quality: The stronger model generated text with recognizable Shakespeare-style features, including uppercase speaker names, colons, line breaks, and dialogue formatting. However, long-range coherence and semantic consistency remained limited.

Significance and Claims

The paper positions MiniGPT as a reproducibility and educational tool rather than a contribution to language modeling performance or architecture.

Modest Claims: The authors explicitly state that MiniGPT does not introduce a new architecture or claim state-of-the-art results. Its value lies in making the "main GPT-style training pipeline explicit, compact, and reproducible."
Educational Value: It demonstrates that a functional autoregressive language model can be built from first principles in a single notebook, clarifying design choices such as pre-LayerNorm, causal masking, and weight tying.
Limitations: The paper acknowledges that the character-level tokenization limits long-range coherence compared to subword methods, and the small dataset scope restricts the model to learning local stylistic patterns rather than broad language capabilities.
Reproducibility: All code is publicly available in a Jupyter notebook format, allowing users to inspect, modify, and rerun the entire experiment from raw text to generated samples.

In conclusion, MiniGPT serves as a transparent reference implementation that bridges the gap between theoretical descriptions of Transformers and practical, runnable code, emphasizing the importance of understanding the full pipeline from data preprocessing to sampling strategies.