Training a model is a ball rolling downhill. The loss is a surface over the parameters; the gradient is the direction of steepest ascent, so we step the opposite way. Pick a landscape — a gentle bowl, a rugged field of local minima, or the notorious Rosenbrock banana — then drop a skier and tune its learning rate and momentum. Too timid and it crawls; too greedy and it rockets uphill and diverges. Watch it stall on saddles, get trapped in the wrong valley, or find the floor.
A model has parameters $\theta$ and a loss $L(\theta)$ measuring how wrong it is. Plotting $L$ over two parameters gives the surface you see — the loss landscape. Learning means finding the $\theta$ where $L$ is smallest.
The gradient $\nabla L$ is the vector of partial derivatives; it points in the direction of steepest ascent. To go down, step against it. That is the entire algorithm — gradient descent: $$\theta_{t+1} = \theta_t - \eta\,\nabla L(\theta_t).$$
The learning rate $\eta$ scales each step. For a quadratic bowl with curvature (Hessian eigenvalues) up to $\lambda_{\max}$, descent converges only if $$0 < \eta < \tfrac{2}{\lambda_{\max}}.$$ Below that it homes in; above it, each step overshoots and the loss explodes. The skier here diverges for exactly this reason.
Momentum gives the skier inertia. Instead of stepping straight along the gradient, it accumulates a velocity: $$v_{t+1}=\beta v_t-\eta\nabla L,\quad \theta_{t+1}=\theta_t+v_{t+1}.$$ With $\beta\approx0.9$ it coasts across flat spots and saddle points (where $\nabla L=0$ but it is not a minimum) that would freeze plain descent, and it damps the zig-zag in narrow, ill-conditioned valleys like Rosenbrock.
Local minima and saddles. A non-convex landscape has many valleys. Descent is greedy — it finds the nearest one, not the deepest. This is why the starting point matters, why momentum helps, and why in practice we use stochastic, noisy gradients that can rattle a skier out of a shallow trap.
Real networks live in millions of dimensions, but the picture is the same: read the local slope, take a step, repeat. Everything else — Adam, learning-rate schedules, batch noise — is a smarter way to choose that step.