Gradient descent and the learning rate

Step 1 Understand it

The problem it solves. For most models there is no formula that gives the best weights directly, and for neural networks there certainly isn't. But you can always compute the gradient. Gradient descent turns "which way is downhill?" into an algorithm that walks to a minimum one small step at a time. Choosing the step size well is the difference between fast training, slow training and no training at all.

Watch it happen

The idea

To make a loss smaller, take a small step against the gradient (downhill), and repeat. The learning rate is the size of each step.

How it works, step by step

1. Pick a starting point: w = 4, where f(w) = w² = 16.
2. Compute the gradient there: f′(w) = 2w = 8. It is positive, so downhill is to the left.
3. Step against it, scaled by the learning rate: w ← 4 − 0.1·8 = 3.2.
4. Repeat. Each step here multiplies w by (1 − 2η) = 0.8: 4 → 3.2 → 2.56 → 2.05 → … After 50 steps w ≈ 0.00006.
5. The steps shrink by themselves near the bottom, because the gradient shrinks there.
6. With η = 0.9 the factor is 1 − 1.8 = −0.8: w flips sign every step but still shrinks, a zig-zag that converges.
7. With η = 1.1 the factor is −1.2: every step overshoots further than the last, and w explodes: 4, −4.8, 5.76, −6.91, …

Picture it

xychart-beta
  title "w over 8 steps: learning rate 0.1 vs 1.1"
  x-axis "step" [0, 1, 2, 3, 4, 5, 6, 7]
  y-axis "w" -15 --> 15
  line [4.0, 3.2, 2.56, 2.05, 1.64, 1.31, 1.05, 0.84]
  line [4.0, -4.8, 5.76, -6.91, 8.29, -9.95, 11.94, -14.33]

Analogy

Walking down a mountain in thick fog. You feel the slope under your feet and step downhill.

Where the analogy breaks

In fog you might stop in a small valley that is not the lowest one. In very high dimensions, flat 'saddle' areas are usually the bigger problem.

Worked example

f(w) = w², so the gradient is 2w. Start at w = 4 with learning rate 0.1: 4 → 3.2 → 2.56 → … → close to 0. With learning rate 1.1: 4 → −4.8 → 5.76 → … it explodes.

w, lr = 4.0, 0.1
for step in range(50):
    grad = 2 * w
    w = w - lr * grad
print(w)   # very close to 0

Must-know points

• Update rule: w ← w − η·∇L(w), for every weight at once.
• η too small: slow. Too large: oscillates or diverges. There is a sweet spot in between.
• For f(w) = w², each step multiplies w by (1 − 2η), so it converges only when 0 < η < 1.
• Stochastic gradient descent (SGD) estimates the gradient from a small random batch: noisier, but far cheaper per step.
• Gradient descent finds a local minimum. For convex functions, that is the global one.
• In practice, try learning rates spaced by factors of 3 to 10 (0.1, 0.03, 0.01, …) and watch the loss curve.

Common mistakes

• Picking one learning rate and never looking at the loss curve.
• Blaming the model when the real problem is a learning rate far too big or too small.
• Leaving features on wildly different scales. The loss surface becomes a long, thin valley and gradient descent zig-zags. Scale features first.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  Neural Networks, Ch 2: Gradient descent, how neural networks learn

  3Blue1Brown

• Watch

  Gradient Descent, Step-by-Step

  StatQuest (Josh Starmer)

• Read

  Mathematics for Machine Learning, Ch 7 (optimisation with gradient descent)

  Deisenroth, Faisal, Ong · free

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   In three sentences, tell a friend why the learning rate matters.

   Show answer

   The learning rate is how big a step the model takes downhill each time it learns. Too small, and training crawls: it may take forever, or stop before it gets anywhere good. Too big, and it overshoots the bottom and lands further up the other side each time, so the loss jumps around or explodes into nan.

2. DeriveWrite the maths, or the steps, from memory.

   f(w) = (w − 3)². Start at w = 0 with learning rate 0.1. Do 3 steps on paper. Where is w going, and why?

   Show answer

   The gradient of f(w) = (w − 3)² is 2(w − 3). The update is w ← w − 0.1·2(w − 3).

   Step 1: w = 0 − 0.1·2·(0 − 3) = 0 + 0.6 = 0.6
   Step 2: w = 0.6 − 0.1·2·(0.6 − 3) = 0.6 + 0.48 = 1.08
   Step 3: w = 1.08 − 0.1·2·(1.08 − 3) = 1.08 + 0.384 = 1.464

   w is moving towards 3, the minimum, where the gradient is 0.

   Why, exactly: subtract 3 from both sides of the update and you get (w − 3) ← (w − 3)·(1 − 2·0.1) = 0.8·(w − 3). So each step removes 20% of the remaining distance: 3 → 2.4 → 1.92 → 1.536 from the target. It gets closer and closer, with steps that shrink as the slope flattens.

3. BuildCode it from scratch in NumPy, with no ML library.

   Write gradient descent for f(w) and plot w over 50 steps. Then use it for linear regression on real data, and plot the loss.

   Show answer

   The same loop works for any differentiable loss: compute the gradient, step against it, repeat. On (w − 3)², w rises quickly towards 3 and flattens out.

   For linear regression on real data, the diabetes dataset (from scikit-learn, only to load it) predicts disease progression from body mass index. Scale the feature first; then gradient descent with a learning rate of 0.1 brings the loss down smoothly, and the weights match the exact least-squares answer from np.polyfit.

   pythonCopy

   import numpy as np
   import matplotlib.pyplot as plt
   from sklearn.datasets import load_diabetes       # only to load the data
   
   w, ws = 0.0, []
   for _ in range(50):
       ws.append(w)
       w -= 0.1 * 2 * (w - 3)
   
   d = load_diabetes(scaled=False)
   x = d.data[:, 2]                                  # body mass index
   x = (x - x.mean()) / x.std()                      # scale the feature
   y = d.target
   a, b, losses = 0.0, 0.0, []
   for _ in range(200):
       pred = a * x + b
       losses.append(np.mean((pred - y) ** 2))
       a -= 0.1 * np.mean(2 * (pred - y) * x)
       b -= 0.1 * np.mean(2 * (pred - y))
   print(round(a, 2), round(b, 2), np.polyfit(x, y, 1).round(2))
   
   fig, axes = plt.subplots(1, 2, figsize=(10, 4))
   axes[0].plot(ws); axes[0].set_xlabel("step"); axes[0].set_ylabel("w")
   axes[1].plot(losses); axes[1].set_xlabel("step"); axes[1].set_ylabel("loss (MSE)")
   plt.tight_layout(); plt.show()

4. BreakShow when it fails, and why.

   Try learning rates 0.01, 0.5, 0.9 and 1.1 on f(w) = (w − 3)². Which is slow, which is fastest, which swings from side to side, and which explodes? Use the update rule to explain why.

   Show answer

   The update gives (w − 3) ← (w − 3)·(1 − 2η), so everything depends on the factor 1 − 2η:

   • η = 0.01: factor 0.98. Each step removes only 2% of the distance. Slow.
   • η = 0.5: factor 0. It jumps straight to w = 3 in one step. The fastest possible here.
   • η = 0.9: factor −0.8. The sign flips each step, so w jumps to the other side of 3, but each jump is shorter. It swings from side to side and still settles.
   • η = 1.1: factor −1.2. It swings to the other side and lands further away each time: 0 → 6.6 → −1.32 → 8.18 … It explodes.

   The general rule for this kind of bowl: the steps shrink only when |1 − 2η| < 1, so η must be between 0 and 1. For real losses the limit depends on how curved the loss is, which is why a learning rate that works on one problem can blow up another.

   pythonCopy

   for lr in [0.01, 0.5, 0.9, 1.1]:
       w, path = 0.0, []
       for _ in range(5):
           path.append(round(w, 2))
           w -= lr * 2 * (w - 3)
       print(lr, path)
   # 0.01 [0.0, 0.06, 0.12, 0.18, 0.23]
   # 0.5 [0.0, 3.0, 3.0, 3.0, 3.0]
   # 0.9 [0.0, 5.4, 1.08, 4.54, 1.77]
   # 1.1 [0.0, 6.6, -1.32, 8.18, -3.22]

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes