Support vector machines: margin and hinge loss

Step 1 Understand it

The problem it solves. Many lines can separate two classes, but the one that hugs some points will misclassify slightly different new points. The SVM picks the boundary with the most room to spare, which tends to generalise well. Its maths — margins, hinge loss, the dual problem and kernels — also introduced ideas used all over ML.

Watch it happen

The idea

An SVM finds the separating line (or plane) with the widest gap, the margin, to the nearest points of each class. Those nearest points are the support vectors. It uses the hinge loss, max(0, 1 − y·score).

How it works, step by step

1. Use labels −1 and +1, and a score s = w·x + b. The boundary is where s = 0.
2. The margin lines are where s = +1 and s = −1. Their distance apart is 2/‖w‖, so a smaller ‖w‖ means a wider street.
3. Hard margin: make every point satisfy y·s ≥ 1 (on the right side, outside the street) while keeping ‖w‖ as small as possible.
4. Only the points lying on the margin lines decide the answer: the support vectors. Move any other point slightly and nothing changes.
5. Soft margin: real data overlaps, so allow violations and pay for them with the hinge loss, max(0, 1 − y·s).
6. Minimise (λ/2)‖w‖² + average hinge loss with subgradient descent. C = 1/λ: a big C punishes violations hard (narrow street); a small C allows more (wide street).
7. Kernels replace dot products with a similarity function, giving curved boundaries without building the curved features by hand.

Picture it

flowchart LR
  N["class −1 points"] --- SV1(("support<br/>vector")) --- M1["margin"] --- B["boundary<br/>w·x + b = 0"] --- M2["margin"] --- SV2(("support<br/>vector")) --- P["class +1 points"]

Analogy

Building a road between two villages. You want the widest road possible that does not touch any house.

Where the analogy breaks

Real data is messy: some houses sit on the wrong side. Soft-margin SVMs allow a few violations, at a cost set by C.

Worked example

Labels are −1 or +1. With y = +1: score 2 gives loss 0 (safe); score 0.5 gives loss 0.5 (inside the margin); score −1 gives loss 2 (wrong side).

loss = np.maximum(0, 1 - y * (X @ w + b)).mean() + lam * (w @ w)

Must-know points

• Margin width = 2/‖w‖.
• Hinge loss max(0, 1 − y·s): 0 for confident correct points, growing linearly for violations.
• Support vectors are the only points that matter for the final boundary.
• C trades margin width against training mistakes.
• The problem is convex, so there is one best answer.
• Scale features first: margins are measured in distances.

Common mistakes

• Using 0/1 labels with the ±1 formulas.
• Forgetting to scale features.
• Reading SVM scores as probabilities. They are distances, not probabilities.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  Support Vector Machines Part 1 (of 3): Main Ideas

  StatQuest (Josh Starmer)

• Watch

  Learning: Support Vector Machines

  Patrick Winston, MIT 6.034

• Read

  ISLP, Ch 9 (support vector machines)

  free book

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain the margin with the road-between-villages picture.

   Show answer

   Two villages sit on either side of a plain, and you must build a straight road between them that touches no house. Many roads would work, but the best is the widest one: the road whose edges are as far as possible from the nearest houses on each side.

   That road is the SVM's boundary, and its width is the margin. Only the houses right at the edge of the road decide where it goes; those are the support vectors. A wide road is safer: a new house built a little off-pattern is still on the right side.

2. DeriveWrite the maths, or the steps, from memory.

   True labels [+1, +1, +1, −1] and scores [2, 0.5, −0.5, −3]. Compute each hinge loss by hand.

   Show answer

   Hinge loss = max(0, 1 − y·score). It is 0 only when the point is on the right side with a margin of at least 1.

   Point 1: y = +1, score 2: 1 − 2 = −1, so max(0, −1) = 0. Correct and safely outside the margin.
   Point 2: y = +1, score 0.5: 1 − 0.5 = 0.5. Correct side, but inside the margin, so a small loss.
   Point 3: y = +1, score −0.5: 1 + 0.5 = 1.5. Wrong side: a bigger loss, growing the further it is wrong.
   Point 4: y = −1, score −3: 1 − 3 = −2, so 0. Correct and safe.

   Total 2.0, mean 0.5. Only points 2 and 3, the ones inside the margin or wrong, affect training; that is why SVMs depend only on the "support vectors".

3. BuildCode it from scratch in NumPy, with no ML library.

   Train a linear SVM with subgradient descent on the hinge loss. Plot the boundary, the margin lines, and circle the support vectors.

   Show answer

   Minimise λ/2·‖w‖² + mean(hinge loss). The hinge has a corner, so use a subgradient: for points with y·score < 1 the gradient of their loss is −y·x (and −y for the bias); for the others it is 0.

   On two blobs, the learned boundary sits in the gap, with the margin lines (score = +1 and −1) touching the closest points of each class. Those closest points, with y·score ≤ 1, are the support vectors: circle them. Moving any other point wouldn't change the boundary.

   pythonCopy

   import numpy as np
   import matplotlib.pyplot as plt
   
   rng = np.random.default_rng(0)
   X = np.vstack([rng.normal([2, 2], 0.7, (50, 2)), rng.normal([-2, -2], 0.7, (50, 2))])
   y = np.array([1] * 50 + [-1] * 50)
   w, b, lam = np.zeros(2), 0.0, 0.01
   for step in range(1, 5001):
       lr = 1 / (lam * step + 100)
       inside = y * (X @ w + b) < 1                     # in the margin, or wrong
       grad_w = lam * w - (y[inside, None] * X[inside]).sum(axis=0) / len(y)
       grad_b = -y[inside].sum() / len(y)
       w, b = w - lr * grad_w, b - lr * grad_b
   
   support = y * (X @ w + b) <= 1 + 1e-3
   print("support vectors:", support.sum(), "| margin width:", round(2 / np.linalg.norm(w), 2))
   xs = np.linspace(-4, 4, 50)
   plt.scatter(X[:, 0], X[:, 1], c=y, s=15)
   for level, style in [(0, "k-"), (1, "k--"), (-1, "k--")]:
       plt.plot(xs, (level - b - w[0] * xs) / w[1], style)
   plt.scatter(X[support, 0], X[support, 1], s=120, facecolors="none", edgecolors="r")
   plt.ylim(-5, 5); plt.show()

4. BreakShow when it fails, and why.

   Add one outlier on the wrong side. Train with a very large C and a small C. How does the boundary move?

   Show answer

   C is the weight on the errors (C is like 1/λ): a large C says "mistakes are very expensive", a small C says "a wide road matters more".

   With one outlier on the wrong side:
   • Very large C: the SVM bends over backwards to reduce the outlier's loss. The boundary shifts and tilts towards it, and the margin becomes narrow. One bad point has reshaped the whole model, which usually hurts on new data.
   • Small C: the SVM accepts the outlier as an error, and the boundary stays almost where it was, with a wide margin.

   That is the soft-margin trade-off: C controls how much the model trusts every single training point. Choose it on the validation set.

   pythonCopy

   import numpy as np
   
   def train_svm(X, y, C, steps=20_000):
       w, b = np.zeros(2), 0.0
       for step in range(1, steps + 1):
           lr = 0.01 / np.sqrt(step)
           inside = y * (X @ w + b) < 1
           w -= lr * (w - C * (y[inside, None] * X[inside]).sum(axis=0))
           b -= lr * (-C * y[inside].sum())
       return w, b
   
   rng = np.random.default_rng(0)
   X = np.vstack([rng.normal([2, 2], 0.7, (50, 2)), rng.normal([-2, -2], 0.7, (50, 2)), [[-1.5, -1.5]]])
   y = np.array([1] * 50 + [-1] * 50 + [1])            # last point: +1 deep in the -1 blob
   for C in [100.0, 0.01]:
       w, b = train_svm(X, y, C)
       print(f"C = {C}: margin width {2 / np.linalg.norm(w):.2f}, outlier score {X[-1] @ w + b:+.2f}")

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes