Metrics: precision, recall, F1, ROC-AUC, confusion matrix

Step 1 Understand it

The problem it solves. Accuracy can lie. If 1% of transactions are fraud, a model that never flags anything is 99% accurate and completely useless. Different jobs care about different mistakes: a cancer screen must not miss cases (recall), a spam filter must not bin real email (precision). The right metric is what tells you whether a model does the job.

Watch it happen

The idea

A confusion matrix counts true positives, false positives, false negatives and true negatives. Precision = TP/(TP + FP): of the items you flagged, how many were right? Recall = TP/(TP + FN): of the real ones, how many did you find? F1 balances the two. ROC-AUC measures how well scores rank positives above negatives.

How it works, step by step

1. Pick a threshold and turn scores into yes/no predictions.
2. Count the four outcomes in a confusion matrix: TP (flagged, really positive), FP (flagged, really negative), FN (missed positive), TN (correctly left alone).
3. Precision = TP / (TP + FP): of everything flagged, how much was right?
4. Recall = TP / (TP + FN): of everything really positive, how much did we find?
5. F1 = 2·P·R / (P + R): one number that is only high when both are high.
6. Move the threshold: lower it and recall rises while precision usually falls; raise it and the reverse happens.
7. ROC curve: plot recall (true positive rate) against false positive rate for every threshold. The area under it (AUC) is the chance a random positive scores above a random negative: 0.5 is guessing, 1.0 is perfect.

Picture it

flowchart TD
  ALL["10,000 transactions<br/>100 are fraud"] --> F["model flags 80"]
  F --> TP["60 really fraud<br/>true positives"]
  F --> FP["20 innocent<br/>false positives"]
  ALL --> FN["40 frauds missed<br/>false negatives"]
  TP --> PR["precision = 60 / 80 = 0.75"]
  TP --> RE["recall = 60 / 100 = 0.60"]

Analogy

Fishing with a net. Precision: how much of your catch is fish, not old boots? Recall: how many of the fish in the lake did you catch?

Where the analogy breaks

In fishing you can see your catch. In ML, the threshold you choose changes both numbers, so always say which threshold you used.

Worked example

10,000 transactions, 100 are fraud. The model flags 80, and 60 of those are fraud. Precision = 60/80 = 0.75. Recall = 60/100 = 0.60. F1 ≈ 0.67.

tp = np.sum((pred == 1) & (y == 1))
fp = np.sum((pred == 1) & (y == 0))
fn = np.sum((pred == 0) & (y == 1))
precision, recall = tp / (tp + fp), tp / (tp + fn)

Must-know points

• Accuracy = (TP + TN) / everything. Misleading when classes are imbalanced.
• Precision: "when it says yes, is it right?" Recall: "does it find them all?"
• There is always a trade-off between them, controlled by the threshold.
• ROC-AUC doesn't depend on the threshold. With very rare positives, the precision–recall curve is more telling.
• Always report the threshold you used, and compare against a dumb baseline.
• Choose the metric from the cost of each mistake, before you train.

Common mistakes

• Reporting accuracy on imbalanced data.
• Quoting precision without recall, or the other way round.
• Choosing the threshold on the test set.
• Mixing up FP and FN. Write out the 2 × 2 table every time.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  ROC and AUC, Clearly Explained

  StatQuest (Josh Starmer)

• Interactive

  ROC & AUC

  MLU-Explain (Amazon) · animated, interactive

  Worth it: Drag the threshold and watch the confusion matrix and the ROC curve change together. The trade-off becomes something you can feel.

• Interactive

  Precision & Recall

  MLU-Explain (Amazon) · animated, interactive

• Read

  Classification: accuracy, recall, precision and related metrics

  Google ML Crash Course

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain precision and recall with the fishing-net picture.

   Show answer

   Fishing with a net. Precision asks: of everything in your net, how much is actually fish, and not old boots? Recall asks: of all the fish in the lake, how many did you catch?

   A huge net catches nearly every fish (high recall) but also lots of boots (low precision). A small, careful net catches only fish (high precision) but misses most of them (low recall). You choose the net by what a mistake costs: for a disease screening test, missing a sick person is worse, so you want high recall; for a spam filter, losing a real email is worse, so you want high precision.

2. DeriveWrite the maths, or the steps, from memory.

   From TP = 40, FP = 10, FN = 20, TN = 930, compute accuracy, precision, recall and F1 by hand.

   Show answer

   TP = 40, FP = 10, FN = 20, TN = 930, so 1,000 cases in total.

   Accuracy = (TP + TN) / total = (40 + 930) / 1,000 = 0.97.
   Precision = TP / (TP + FP) = 40 / 50 = 0.80. Of the 50 flagged, 40 were right.
   Recall = TP / (TP + FN) = 40 / 60 ≈ 0.667. Of the 60 real positives, 40 were found.
   F1 = 2·P·R / (P + R) = 2·0.8·0.667 / 1.467 ≈ 0.727. (Or directly: 2TP / (2TP + FP + FN) = 80 / 110.)

   Notice that accuracy, 97%, looks far better than the model really is: it misses a third of the positives. Accuracy is flattered by the 930 easy negatives.

3. BuildCode it from scratch in NumPy, with no ML library.

   Write precision, recall, F1 and ROC-AUC from scratch. For AUC, sort by score and use the trapezoid rule.

   Show answer

   Precision, recall and F1 come straight from the counts. For ROC-AUC, sort by score from high to low and walk down the list, lowering the threshold one example at a time: each positive moves the curve up (true positive rate), each negative moves it right (false positive rate). The area under that curve, by the trapezoid rule, is the AUC.

   Check it with its meaning: AUC is the chance that a random positive gets a higher score than a random negative, which you can count directly by comparing every pair. Both give the same number. (Ties in scores need care; this version assumes distinct scores.)

   pythonCopy

   import numpy as np
   
   def precision(y, pred):
       tp, fp = np.sum((pred == 1) & (y == 1)), np.sum((pred == 1) & (y == 0))
       return tp / (tp + fp) if tp + fp else 0.0
   
   def recall(y, pred):
       tp, fn = np.sum((pred == 1) & (y == 1)), np.sum((pred == 0) & (y == 1))
       return tp / (tp + fn) if tp + fn else 0.0
   
   def f1(y, pred):
       p, r = precision(y, pred), recall(y, pred)
       return 2 * p * r / (p + r) if p + r else 0.0
   
   def roc_auc(y, scores):
       y = y[np.argsort(-scores)]                        # highest score first
       tpr = np.concatenate([[0], np.cumsum(y) / y.sum()])
       fpr = np.concatenate([[0], np.cumsum(1 - y) / (1 - y).sum()])
       return np.trapezoid(tpr, fpr)
   
   rng = np.random.default_rng(0)
   y = rng.integers(0, 2, 500)
   scores = y + rng.normal(0, 0.8, 500)                  # noisy but useful scores
   pairs = np.mean(scores[y == 1][:, None] > scores[y == 0][None, :])
   print(round(roc_auc(y, scores), 4), round(pairs, 4))   # the same
   pred = (scores > 0.5).astype(int)
   print(round(precision(y, pred), 3), round(recall(y, pred), 3), round(f1(y, pred), 3))

4. BreakShow when it fails, and why.

   Make a 'model' that always predicts 'not fraud' on data with 1% fraud. What are its accuracy, precision and recall? Which metric exposes it?

   Show answer

   On 10,000 transactions with 100 frauds, "always not fraud" gets:
   • accuracy 99%, which looks excellent;
   • recall 0: it finds none of the 100 frauds;
   • precision undefined (0 / 0, because it flags nothing), usually reported as 0.

   Recall exposes it immediately, and so do F1 (0) and the confusion matrix, which shows TP = 0. Accuracy hides it because 99% of the answers are "not fraud" anyway.

   The rule: on imbalanced data, never report accuracy alone. Report precision and recall for the rare class, or the precision–recall curve.

   pythonCopy

   import numpy as np
   
   y = np.array([1] * 100 + [0] * 9900)          # 1% fraud
   pred = np.zeros_like(y)                       # always "not fraud"
   tp = np.sum((pred == 1) & (y == 1))
   print("accuracy", (pred == y).mean())          # 0.99
   print("recall", tp / y.sum())                  # 0.0
   print("flagged", pred.sum())                   # 0: precision is 0 / 0

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes