A language model looks like it understands you. Underneath, it is doing one thing over and over: guessing the next piece of text and getting slightly less wrong each time. Each tab below runs the real machinery in your browser — no faked plots. A live byte-pair tokenizer is trained on a built-in corpus and splits whatever you type; a tiny skip-gram model pushes words that share company toward each other in a 2-D map; scaled dot-product attention draws the arcs each token actually leans on; and a genuine softmax classifier trained by cross-entropy watches its own loss fall as its next-word guess sharpens. Press TRAIN ONE EPOCH up top and the embedding map and the next-token model both take a step. Watch the loss drop — that dropping number is the entire trick.
A model must map text to a finite list of integer IDs. Two obvious choices both fail. Whole words give a vocabulary that is enormous, never complete, and helpless against typos or new words. Single characters give a tiny vocabulary but make every sequence painfully long, forcing the model to relearn spelling from scratch every time.
Byte-pair encoding (BPE) threads the needle. Start with characters. Count every adjacent pair in the corpus, glue the single most frequent pair into a new token, and add it to the vocabulary. Repeat a few thousand times. Common letter sequences, endings, and whole frequent words become single tokens, while rare strings stay decomposable.
The payoff is coverage with compression. Any string — including one never seen in training — can always be encoded, because the alphabet remains in the vocabulary as a fallback. Yet common text is short, because "the" is one token, not three.
This choice ripples through everything. Token boundaries decide how the model "sees" numbers (often digit by digit, hurting arithmetic), how many tokens a language costs (English is cheap; some scripts are expensive), and even how prompts are billed. The tokenizer is frozen before training — the model spends its whole life speaking in exactly these pieces.
A token ID is meaningless — 4127 is no closer to 4128 than to 99. So the first thing a model does is look each ID up in an embedding table: a learned vector of a few hundred numbers. Training shapes that table so the geometry carries meaning.
The classic recipe is skip-gram with negative sampling (word2vec). For each word in the corpus, take its real context words as positive examples and a handful of random words as negatives. Nudge the word's vector to have a high dot product with its true neighbours and a low one with the random impostors.
Do this across millions of sentences and a startling structure appears. Similar words cluster; and famous directions emerge — the vector from king to queen roughly matches man to woman. Meaning has become arithmetic.
In a full transformer the embedding table is just the input layer, learned jointly with everything above it, and refined at every layer by attention. But the principle set here never leaves: a word is a point, and its neighbours define what it means. This map is the model's dictionary.
Embeddings give each token a meaning in isolation, but words live in context: "bank" by a river is not "bank" with your money. Attention is how a token gathers context.
From each token's vector the model computes three projections: a query (what am I looking for?), a key (what do I offer?), and a value (what will I hand over?). A token's query is compared, by dot product, against every key. Big dot product means "relevant." A softmax turns those scores into weights that sum to one, and the token's new representation is that weighted mix of everyone's values.
Two refinements make it work at scale. Scores are divided by √d_k so they don't explode as vectors grow — the "scaled" in scaled dot-product. And the whole thing runs in several heads in parallel, each free to specialise, their outputs concatenated.
Stack this dozens of layers deep and information routes freely across a sequence, every token continuously re-reading every other. This single operation — introduced in "Attention Is All You Need" (2017) — is the beating heart of every large language model. The causal mask is the only twist for generation: hide the future so the model must truly predict it.
Tokenization, embeddings and attention all feed one final step: a softmax over the whole vocabulary that outputs a probability for every possible next token. Training compares that predicted distribution to reality — where the "true" answer is simply the token that actually came next — and pushes the probability of the right token up.
The scorecard is cross-entropy loss: the negative log-probability the model assigned to the correct token. Guess it with probability 1 and the loss is 0. Guess it with probability near 0 and the loss is huge. Averaged over a corpus, this is exactly the model's surprise per token, measured in bits.
Gradient descent then does what it always does: nudge every weight — embeddings, attention projections, the output layer — a hair in the direction that lowers that surprise. Repeat over trillions of tokens.
Nothing in the objective mentions grammar, facts, or reasoning. Yet to predict the next token well across the whole internet, a model is forced to internalise all of it: syntax to finish sentences, world knowledge to complete facts, even arithmetic to continue a sum. Capability is a side effect of relentless next-token prediction — the single loop this tab runs in miniature.