Entropy, cross-entropy and KL divergence

Step 1 Understand it

The problem it solves. A classifier outputs probabilities, and you need a loss that rewards putting high probability on the right answer and punishes confident mistakes hard. Cross-entropy is that loss, and nearly every classifier and every language model is trained on it. KL divergence measures how far apart two distributions are, and it shows up in VAEs, knowledge distillation and RLHF.

Watch it happen

The idea

Entropy measures uncertainty, or 'surprise', in a distribution. Cross-entropy measures your surprise when you use your model q to predict data that really comes from p. KL divergence is the extra surprise: KL(p‖q) = cross-entropy − entropy.

How it works, step by step

1. The surprise of an event with probability q is −log q. Certain events (q = 1) cost 0; rare ones cost a lot.
2. Entropy is the average surprise under the true distribution: H(p) = −Σ p log p. A coin that lands heads 70% of the time has about 0.88 bits; a certain outcome has 0.
3. Cross-entropy uses the model's probabilities for the surprise but the true distribution for the averaging: H(p, q) = −Σ p log q.
4. With a one-hot true label, every term but one vanishes: the loss is −log of the probability the model gave the right class. For 0.7 that is 0.357.
5. If the model had said 0.99, the loss would be about 0.01; if it had said 0.05, about 3.0. Confident mistakes are punished hardest.
6. KL(p ‖ q) = H(p, q) − H(p). It is never negative, and it is 0 only when q = p. H(p) doesn't depend on the model, so minimising cross-entropy also minimises KL.

Picture it

xychart-beta
  title "Cross-entropy loss: −log(probability given to the right answer)"
  x-axis "probability of the right answer" [0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99]
  y-axis "loss" 0 --> 3.2
  bar [3.0, 2.3, 1.2, 0.69, 0.36, 0.11, 0.01]

Analogy

Packing for a trip. With the true weather forecast (p), you pack exactly right. With a wrong forecast (q), you pack badly. KL is the extra, wasted luggage.

Where the analogy breaks

In the analogy, swapping the two forecasts feels like the same mistake. KL is not symmetric: KL(p‖q) is usually different from KL(q‖p).

Worked example

The true label is class 1 of 3, so p = [0, 1, 0]. The model says q = [0.2, 0.7, 0.1]. Cross-entropy = −log(0.7) ≈ 0.357. If the model said 0.99, the loss would be about 0.01. Here the entropy of p is 0, so KL equals the cross-entropy.

q = np.array([0.2, 0.7, 0.1])
-np.log(q[1])   # ≈ 0.357

Must-know points

• Surprise = −log(probability). The natural log gives nats; log base 2 gives bits.
• Entropy H(p) = −Σ p log p: highest for a uniform distribution, zero for a certain one.
• Cross-entropy H(p, q) = −Σ p log q, and it is always at least H(p).
• KL(p ‖ q) = H(p, q) − H(p) ≥ 0, and it is not symmetric.
• Classification loss = −log(probability of the true class), averaged over examples.
• Language models are trained on exactly this: −log P(the next token).

Common mistakes

• Taking the log of a probability that is exactly 0: −log 0 is infinite. Work with logits and log-softmax instead.
• Treating KL as a distance. KL(p ‖ q) ≠ KL(q ‖ p).
• Swapping the roles in cross-entropy: the true distribution does the averaging, and the model supplies the logs.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  Entropy (for data science) Clearly Explained

  StatQuest (Josh Starmer)

• Watch

  A Short Introduction to Entropy, Cross-Entropy and KL-Divergence

  Aurélien Géron

• Read

  Visual Information Theory

  Chris Olah · illustrated

  Worth it: Pictures that turn entropy, cross-entropy and KL into areas you can see. The best explanation of why the formulas look the way they do.

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain KL divergence with the packing-for-a-trip picture.

   Show answer

   You are packing for a trip. With the true weather forecast, p, you pack exactly what you need. With a wrong forecast, q, you pack badly: an umbrella you never use, and no coat for the cold.

   KL divergence, KL(p‖q), is the extra, wasted luggage: how much worse you do on average by planning with q when the weather really follows p. If the forecasts agree, there is no waste, so KL is 0. It can never be negative: a wrong forecast can't help you on average. And it isn't symmetric: packing for sunshine and getting snow costs differently from packing for snow and getting sunshine.

2. DeriveWrite the maths, or the steps, from memory.

   By hand, using log base 2: the entropy of a fair coin, of a coin with p = 0.9, and of a fair die. Which is most uncertain?

   Show answer

   Entropy H(p) = −Σ pᵢ·log₂ pᵢ, measured in bits.

   Fair coin: −(0.5·log₂ 0.5 + 0.5·log₂ 0.5) = −(0.5·(−1) + 0.5·(−1)) = 1 bit.

   Coin with p = 0.9: −(0.9·log₂ 0.9 + 0.1·log₂ 0.1) = −(0.9·(−0.152) + 0.1·(−3.322)) = 0.137 + 0.332 ≈ 0.47 bits. Less uncertain: you can usually guess heads.

   Fair die: −6·(1/6)·log₂(1/6) = log₂ 6 ≈ 2.585 bits.

   The fair die is the most uncertain: it has the most equally likely outcomes. For n equally likely outcomes, entropy is log₂ n, the most any distribution over n outcomes can have.

3. BuildCode it from scratch in NumPy, with no ML library.

   Write entropy(p), cross_entropy(p, q) and kl(p, q). Check that kl = cross-entropy − entropy, and that kl(p, p) = 0.

   Show answer

   entropy(p) = −Σ p·log p, cross_entropy(p, q) = −Σ p·log q, and kl(p, q) = Σ p·log(p/q). Skip terms where p is 0, because 0·log 0 counts as 0.

   Both checks pass: kl equals cross-entropy minus entropy, and kl(p, p) is 0. KL is the extra cost of using q when the truth is p, and there is none when they are the same.

   pythonCopy

   import numpy as np
   
   def entropy(p):
       p = np.asarray(p, float)
       nz = p > 0
       return -np.sum(p[nz] * np.log2(p[nz]))
   
   def cross_entropy(p, q):
       p, q = np.asarray(p, float), np.asarray(q, float)
       nz = p > 0
       return -np.sum(p[nz] * np.log2(q[nz]))
   
   def kl(p, q):
       p, q = np.asarray(p, float), np.asarray(q, float)
       nz = p > 0
       return np.sum(p[nz] * np.log2(p[nz] / q[nz]))
   
   p, q = [0.5, 0.3, 0.2], [0.4, 0.4, 0.2]
   assert np.isclose(kl(p, q), cross_entropy(p, q) - entropy(p))
   assert np.isclose(kl(p, p), 0.0)
   print(entropy([0.5, 0.5]), round(entropy([0.9, 0.1]), 3), round(entropy([1 / 6] * 6), 3))
   # 1.0 0.469 2.585

4. BreakShow when it fails, and why.

   Compute kl(p, q) where q has a 0 where p does not. What happens? Then compare kl(p, q) with kl(q, p) for two distributions.

   Show answer

   KL becomes infinite. Where p says "this happens" but q says "probability 0", the term p·log(p/0) is infinite. The meaning is real: q claims something is impossible, and then it happens, which is infinitely surprising. It is the same reason a classifier that gives the right class probability 0 gets infinite cross-entropy loss, and why probabilities are clipped away from 0 in practice.

   KL is not symmetric. For p = [0.9, 0.1] and q = [0.5, 0.5], kl(p, q) ≈ 0.53 bits but kl(q, p) ≈ 0.74 bits. They measure different things: kl(p, q) is the cost of using q when the truth is p. So KL is not a true distance, and which way round you use it changes what a model learns.

   pythonCopy

   import numpy as np
   
   def kl(p, q):
       p, q = np.asarray(p, float), np.asarray(q, float)
       nz = p > 0
       with np.errstate(divide="ignore"):
           return np.sum(p[nz] * np.log2(p[nz] / q[nz]))
   
   print(kl([0.5, 0.5], [1.0, 0.0]))            # inf
   p, q = [0.9, 0.1], [0.5, 0.5]
   print(round(kl(p, q), 3), round(kl(q, p), 3))   # 0.531 0.737

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes