Eigenvalues and eigenvectors

Step 1 Understand it

The problem it solves. A matrix moves vectors in a confusing mix of turning and stretching. Eigenvectors find the special directions where it acts like plain multiplication, which makes the matrix easy to understand. They give the main directions of variation in data (PCA), tell you whether repeated multiplication explodes or fades (RNNs, Markov chains), and explain how fast gradient descent converges.

Watch it happen

The idea

An eigenvector is a direction that a matrix does not turn. It only stretches or shrinks it. The eigenvalue is the stretch factor: Av = λv.

How it works, step by step

1. Apply A = [[2, 1], [1, 2]] to many directions and see which ones stay on their own line. Most are knocked off: [1, 0] becomes [2, 1], a different direction.
2. [1, 1] becomes [3, 3] = 3·[1, 1]: same direction, 3 times longer. That is an eigenvector with eigenvalue 3.
3. [1, −1] becomes [1, −1]: unchanged. An eigenvector with eigenvalue 1.
4. To find them by hand, solve det(A − λI) = 0. Here (2 − λ)² − 1 = 0, so λ = 3 or λ = 1.
5. For each λ, solve (A − λI)v = 0 to get its direction. Any non-zero multiple of v is also an eigenvector.
6. In the eigenvector directions, A just stretches: 3× along [1, 1] and 1× along [1, −1]. Applying A ten times stretches by 3¹⁰ and 1¹⁰, so repeated use is easy to predict.

Picture it

flowchart LR
  V1["v = [1, 1]"] -->|"A = [[2, 1], [1, 2]]"| R1["[3, 3] = 3 × v<br/>same direction → eigenvector, λ = 3"]
  V2["v = [1, 0]"] -->|"A"| R2["[2, 1]<br/>direction turned → not an eigenvector"]

Analogy

A spinning globe. Every point moves, except the points on the axis. The axis is an eigenvector, with λ = 1.

Where the analogy breaks

A 2D rotation has no real eigenvectors at all: it turns every direction.

Worked example

A = [[2, 1], [1, 2]]. For v = [1, 1], Av = [3, 3] = 3v, so λ = 3. For v = [1, −1], Av = [1, −1], so λ = 1. In PCA, the main directions of the data are the eigenvectors of the covariance matrix.

A = np.array([[2., 1.], [1., 2.]])
vals, vecs = np.linalg.eigh(A)   # eigh is for symmetric matrices
vals                              # [1., 3.]

Must-know points

• Av = λv: the direction is unchanged and the length is multiplied by λ (a negative λ flips it).
• Find λ from det(A − λI) = 0, then v from (A − λI)v = 0.
• Symmetric matrices, like covariance matrices, have real eigenvalues and orthogonal eigenvectors. Use np.linalg.eigh for them.
• The eigenvalues multiply to the determinant and add up to the trace (the sum of the diagonal).
• Repeated multiplication: |λ| > 1 grows, |λ| < 1 fades. This is the root of exploding and vanishing gradients.
• PCA's main directions are the eigenvectors of the data's covariance matrix, ordered by eigenvalue.

Common mistakes

• Expecting every matrix to have real eigenvectors. A 2-D rotation turns every direction, so its eigenvalues are complex.
• Expecting eigenvectors to have a particular length. Any multiple works; libraries return length 1.
• Using np.linalg.eig on a symmetric matrix. eigh is faster, more accurate, and returns real, sorted values.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  Essence of Linear Algebra, Ch 14: Eigenvectors and eigenvalues

  3Blue1Brown

  Worth it: Watching vectors get knocked off their lines while the eigenvectors stay put is the moment eigenvectors make sense. Worth it before PCA.

• Interactive

  Eigenvectors and Eigenvalues, explained visually

  Setosa · drag the vectors yourself

• Read

  Mathematics for Machine Learning, Ch 4 (eigenvalues, eigendecomposition)

  Deisenroth, Faisal, Ong · free

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain eigenvectors with the spinning-globe picture, then say where the picture breaks.

   Show answer

   Spin a globe. Every point on the surface moves, except the points on the axis, which stay exactly where they are. The axis is an eigenvector: a direction the transformation doesn't turn. Its eigenvalue says how much things along it are stretched; for a spin, that is 1, no stretch at all.

   Where the picture breaks: most matrices don't spin, they stretch. For them, an eigenvector stays on its own line but gets longer or shorter (λ = 3 triples it, λ = 0.5 halves it), and a negative λ flips it to point the other way. Also, a flat 2D rotation has no axis at all, so it has no real eigenvectors. And a matrix can have several eigenvectors that aren't at right angles to each other, unlike the one axis of a globe.

2. DeriveWrite the maths, or the steps, from memory.

   By hand, find the eigenvalues and eigenvectors of [[4, 1], [2, 3]], using det(A − λI) = 0.

   Show answer

   Solve det(A − λI) = 0:
   (4 − λ)(3 − λ) − 1·2 = λ² − 7λ + 12 − 2 = λ² − 7λ + 10 = (λ − 5)(λ − 2) = 0.
   So the eigenvalues are λ = 5 and λ = 2.

   For λ = 5: A − 5I = [[−1, 1], [2, −2]]. The first row says −x + y = 0, so y = x. Eigenvector [1, 1]. Check: A·[1, 1] = [5, 5] = 5·[1, 1] ✓.

   For λ = 2: A − 2I = [[2, 1], [2, 1]], so 2x + y = 0 and y = −2x. Eigenvector [1, −2]. Check: A·[1, −2] = [4 − 2, 2 − 6] = [2, −4] = 2·[1, −2] ✓.

   Any multiple of an eigenvector is also an eigenvector, so [2, 2] or [0.707, 0.707] are equally right.

3. BuildCode it from scratch in NumPy, with no ML library.

   Write power iteration to find the largest eigenvalue and its eigenvector. Compare with np.linalg.eig.

   Show answer

   Power iteration: start with a random vector, multiply by A, rescale to length 1, and repeat. Each multiplication stretches the part along the biggest eigenvector the most, so after a few dozen steps only that direction is left. The eigenvalue is then v·Av (the Rayleigh quotient).

   For [[4, 1], [2, 3]] it finds 5 and [0.707, 0.707], the same as np.linalg.eig.

   pythonCopy

   import numpy as np
   
   def power_iteration(A, steps=100, seed=0):
       v = np.random.default_rng(seed).normal(size=A.shape[0])
       for _ in range(steps):
           v = A @ v
           v = v / np.linalg.norm(v)
       return v @ A @ v, v
   
   A = np.array([[4., 1.], [2., 3.]])
   value, vector = power_iteration(A)
   print(round(value, 6), vector.round(3))        # 5.0 [0.707 0.707]
   values, vectors = np.linalg.eig(A)
   print(values, vectors[:, np.argmax(values)].round(3))

4. BreakShow when it fails, and why.

   Run power iteration on the rotation matrix [[0, −1], [1, 0]]. What happens, and why does it never settle?

   Show answer

   It never settles. Each multiplication turns the vector by another 90°, so it goes round and round and comes back to where it started every 4 steps.

   Why: a rotation leaves no direction unchanged, so it has no real eigenvectors. Its eigenvalues are the complex numbers i and −i. Power iteration also needs one eigenvalue that is strictly bigger in size than all the others, so one direction can win. Here both have size 1, so neither wins.

   The lesson: power iteration only works when there is a single dominant eigenvalue, and it slows down when the top two are close in size.

   pythonCopy

   import numpy as np
   
   R = np.array([[0., -1.], [1., 0.]])
   v = np.array([1., 0.])
   for step in range(5):
       print(step, v)
       v = R @ v                    # [1,0] → [0,1] → [-1,0] → [0,-1] → [1,0]
   print(np.linalg.eig(R)[0])       # [0.+1.j 0.-1.j]: complex, both size 1

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes