Runs in this tab  ::  nothing precomputed  ::  your text, not mine

A GPT you can train on your own text

Paste anything below and a 41,000-parameter transformer learns it here, in your browser, in well under a minute. Then read what its attention heads are doing and ask it to write. I wrote every backward pass in it by hand, so the last section checks them.

Almost everything you have read about transformers was written with PyTorch doing the calculus. This one has no .backward() anywhere in it. I derived the gradient of attention, LayerNorm, GELU and cross-entropy on paper and typed each one out as explicit NumPy, then compiled the package to WebAssembly so it can run here.

Which means the thing training below is the same code you can read in the repository, and the gradients moving its weights are ones I worked out by hand. The last section checks all 41,472 of them against the definition of a derivative, on the exact model you are about to train.

01
Experiment one of four · training

Give it text. Watch it learn to write.

Character by character, from nothing. Paste your own or keep the default, then press train and read the samples as they come.

The model reads one character at a time and has one job: given everything up to here, what comes next. It starts knowing nothing, including that q is followed by u and that words have spaces between them. Everything it ends up knowing about your text, it learns from being wrong about that question a few thousand times.

Which makes the loss readable. It's how surprised it was, averaged over every character it tried to predict. A model guessing uniformly over an alphabet of V characters pays log V every time, so with the 24 characters in the passage below, 3.18 means it has learned nothing at all. Watch the curve cross that line and you're watching it work out that some characters follow others.

loading
Width48
Context32
Blocks2
Steps600
   not trained
the sample it writes will appear here
Finding · one

The sample above regenerates every 25 steps, so you can watch it cross from noise into whatever you gave it.

02
Experiment two of four · sampling

Write a prompt. Watch it answer, one character at a time.

The bars are what it thinks comes next, before it commits. Temperature decides how much notice it takes of them.

Generating is the same forward pass you just trained, run one character at a time. The model turns your prompt into a score for every character in the alphabet, those scores become probabilities, one is drawn, and it goes back in as input for the next. The bars below are that distribution, caught before the draw.

Temperature divides the scores before they become probabilities. At zero it always takes the tallest bar, which is why it loops: the same context gives the same character, forever. Raise it and the smaller bars get a real chance, so it stops repeating and starts inventing spellings. Somewhere in the middle is a model that sounds like your text without reciting it.

type here and the distribution below updates on every keystroke
Temperature0.5
0 always takes the top bar; higher rolls the dice
Length160
waiting on a trained model
train it first
Finding · two

Nothing here is retrieval. It has no copy of your text, only 41,472 numbers that were nudged toward it.

03
Experiment three of four · attention

Where each head is looking

One square per pair of characters: how much the row attends to the column. The top-right triangle is empty because nothing may read the future.

Attention is how a character decides which earlier characters are worth reading. Every position produces a query, every position produces a key, and the match between them becomes a weight. The character then takes a weighted average of what those positions hold. So each row of the grid below is one character asking the question, and the brightness along it is the answer.

The upper right is blank by construction. Position i may only read positions up to i, because a model allowed to see the next character would score perfectly and learn nothing. Each block runs several of these at once, and they don't end up doing the same job: compare the heads in block 0, which read raw characters, against those in block 1, which read whatever block 0 already assembled.

any string from your alphabet
train it first
Finding · three

Before training these are close to flat: every head spreads its weight evenly over everything the mask allows. Training is what puts structure in them.

04
Experiment four of four · the gradients

Every gradient in that, derived by hand

A derivative is defined as a limit of a difference quotient, so it can be checked with no calculus at all: nudge a weight, see what the loss does, divide.

This runs on the model you just trained, not a smaller stand-in. For each weight it perturbs that one number by ±ε, runs two full forward passes, and compares what the loss actually did against what my hand-written backward pass claimed it would. The two sides share no code: one is algebra I did on paper, the other is arithmetic.

Both axes are on a log scale because the gradients here span five decades, and agreement has to hold at the small end as well as the large. A point off the diagonal is a derivative I got wrong. The second button removes one term from the gradient of LayerNorm, the piece that accounts for its variance depending on every coordinate at once, which is the mistake this check was written to catch.

   not run
Finding · four

Run it. Every point should sit on the diagonal, across every decade.

The package is transformer-from-scratch, about 600 lines of NumPy with no runtime dependencies. There's a notebook that does the checks a browser can't, including diffing every gradient against PyTorch's autograd, where the two agree to about 1e-15.