Machine learning looks like magic and is mostly calculus rolling downhill. Each tab below builds a real, hand-rolled model in JavaScript — no libraries, no faked plots — and lets you turn its knobs. A perceptron rotates a line until two clouds of points fall apart; a ball skis a loss landscape you can only ever see in slices; a tiny two-layer network lights up its activations forward and its gradients backward; and a polynomial slides from lazy underfit to memorised noise while a held-out test curve quietly betrays it. Press TRAIN ONE EPOCH up top and every model takes a single step at once. Watch the loss fall — that falling number is the whole of learning.
A perceptron scores a point by a weighted sum s = w·x + b and predicts blue if s > 0, red otherwise. Geometrically, w·x + b = 0 is a straight line (a hyperplane in higher dimensions); w is the normal vector pointing into the blue side, and b slides the line off the origin.
Learning is pure error-correction. Present a point; if the prediction is right, do nothing. If it's wrong, add η·y·x to w. Because that pushes w·x in the direction of the correct label, the same point is now scored a little more correctly — the line rotates toward it.
Rosenblatt proved in 1958 that if a separating line exists, this loop finds one after a finite number of updates, bounded by (R/γ)² where R is the data radius and γ the margin. No line exists for problems like XOR — which is exactly why a single perceptron can't learn it, a limitation Minsky & Papert made famous in 1969.
The escape, twenty years later, was to stack perceptrons into layers and replace the hard threshold with a smooth curve so the whole thing is differentiable — then descend the error with calculus. That is tabs 2 and 3.
Every model has parameters θ and a loss L(θ) that measures how wrong it is on the data. Learning means finding θ that makes L small. The loss is a landscape over parameter space — here just 2-D so we can draw it, but in a real network it lives in millions of dimensions.
The gradient ∇L is the vector of partial derivatives; it points in the direction of steepest increase. So to go down, step the opposite way: θ ← θ − η∇L. The scalar η, the learning rate, is the stride length.
Momentum keeps a running velocity v ← βv − η∇L, then θ ← θ + v. Consistent downhill directions accumulate into speed; oscillations across a narrow valley partly cancel. It damps zig-zag and powers through flat saddle regions where the raw gradient nearly vanishes.
Because you only ever consult the local slope, gradient descent finds a nearby minimum, not necessarily the global one — the reason initialisation and randomness matter. Astonishingly, in the huge landscapes of deep networks, most local minima turn out to be nearly as good as the best.
Each neuron computes a = σ(w·x + b) — a weighted sum passed through a non-linear activation (here tanh, then a sigmoid at the output). Stacking a hidden layer lets the network carve curved decision regions, escaping the single perceptron's straight-line limit and solving XOR.
To train, define a loss (squared error, or cross-entropy) and ask: how does L change if I nudge this one weight? That's the partial derivative ∂L/∂w. Computing millions of these naively would be hopeless.
Backpropagation makes it cheap. The chain rule says a deep derivative is a product of local ones. So compute the error at the output, then propagate a "sensitivity" signal δ backward through each layer, multiplying by that layer's local derivative. Every weight's gradient falls out as δ · (its input).
One forward pass, one backward pass, and you have every gradient — then take a gradient-descent step (Tab 2) on the whole weight vector. Rumelhart, Hinton & Williams popularised this in 1986; it is the engine under essentially every modern neural network.
Fitting a degree-d polynomial means choosing coefficients that minimise squared error on the training points — a linear least-squares solve. As d grows, the model class gets more flexible: it can represent ever more wiggly curves, so it can always drive training error lower.
But low training error is not the goal. We want low error on new data drawn from the same source. That's why we hold out a test set and never fit to it — it's an honest estimate of how the model behaves in the wild.
The famous bias–variance trade-off: a too-simple model has high bias (systematically wrong, underfit); a too-complex one has high variance (wildly sensitive to the particular noise in the training sample, overfit). Total test error is roughly bias² + variance + irreducible noise — a U-shaped curve in model complexity.
The cures are more data, simpler models, and regularisation (penalising large coefficients). Every modern network — however enormous — lives or dies by this same gap between training loss and held-out loss.