Dot product, projection and orthogonality

Step 1 Understand it

The problem it solves. ML constantly asks "how similar are these two things?": two documents in search, a query and a key in attention, an input and a neuron's weights. The dot product answers with one cheap number. Projection, built from the dot product, keeps only the part of a vector that lies along a direction, which is the idea behind least squares and PCA.

Watch it happen

The idea

The dot product multiplies matching entries and adds them up. It measures how much two vectors point the same way. A dot product of 0 means they are at 90° (orthogonal). A projection is the 'shadow' of one vector on another.

How it works, step by step

1. Multiply matching entries and add: for a = [3, 4] and b = [1, 0], a·b = 3·1 + 4·0 = 3.
2. The same number in geometry: a·b = |a|·|b|·cos θ, where θ is the angle between them. |a| = 5 and |b| = 1, so cos θ = 3/5 and θ ≈ 53°.
3. The sign gives the direction: positive means the angle is under 90°, zero means exactly 90° (orthogonal), negative means over 90°.
4. The projection of a onto b is a's shadow on the line through b: (a·b / b·b)·b = (3/1)·[1, 0] = [3, 0].
5. What is left over, a − projection = [0, 4], is orthogonal to b: [0, 4]·[1, 0] = 0. Projection splits a vector into "along b" plus "at right angles to b".
6. Cosine similarity divides out the lengths: cos θ = a·b / (|a||b|). It runs from −1 (opposite) to 1 (same direction) and ignores length, which is why search uses it.

Picture it

flowchart LR
  A["a = [3, 4]"] --> D["a · b = 3×1 + 4×0 = 3"]
  B["b = [1, 0]"] --> D
  D --> P["shadow of a on b<br/>= [3, 0]"]
  O["[1, 2] · [2, −1] = 0"] --> R["at 90°: orthogonal"]

Analogy

Pulling a box with a rope at an angle. Only the part of your pull along the floor moves the box. That part is the projection.

Where the analogy breaks

Your pull is a physical force in 2D or 3D. A dot product works for any vectors, even word embeddings with 768 numbers.

Worked example

a = [3, 4], b = [1, 0]: a·b = 3, and the shadow of a on b is [3, 0]. [1, 2]·[2, −1] = 2 − 2 = 0, so these two are orthogonal. Cosine similarity, used in search, is the dot product of two vectors after scaling them to length 1.

a = np.array([3., 4.]); b = np.array([1., 0.])
proj = (a @ b) / (b @ b) * b    # [3., 0.]

Must-know points

• a·b = Σ aᵢbᵢ = |a||b| cos θ.
• A dot product of 0 means orthogonal: at 90°.
• a·a = |a|², so a vector's length is √(a·a).
• Projection of a onto b: (a·b / b·b)·b. The leftover part is orthogonal to b.
• Cosine similarity is the dot product after scaling both vectors to length 1.
• A matrix–vector product is a list of dot products: each output is one row of the matrix dotted with the vector.

Common mistakes

• Comparing similarity with the raw dot product when lengths differ a lot. Long vectors win just for being long; use cosine similarity.
• Forgetting to divide by b·b in the projection when b is not length 1.
• Dividing by a zero-length vector in cosine similarity.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Watch

  Essence of Linear Algebra, Ch 9: Dot products and duality

  3Blue1Brown

• Watch

  18.06 lecture: Projections onto subspaces

  Gilbert Strang (MIT)

• Read

  Mathematics for Machine Learning, Ch 3 (inner products, orthogonality, projections)

  Deisenroth, Faisal, Ong · free

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain why two vectors at 90° have a dot product of 0, using the rope-and-box picture.

   Show answer

   The dot product measures how much of one vector goes along the other: a·b = |a|·|b|·cos(angle).

   Pull a box with a rope. If the rope is flat along the floor, all your pull moves the box. Raise the rope and less of the pull goes along the floor. Pull straight up, at 90° to the floor, and none of the pull goes along the floor: the box doesn't slide at all. That "along the floor" part is the projection, and at 90° it is 0, because cos 90° = 0. So two vectors at right angles have a dot product of 0: neither has any part along the other.

2. DeriveWrite the maths, or the steps, from memory.

   By hand, project a = [2, 3] onto b = [1, 1]. Then show that a minus its projection is orthogonal to b.

   Show answer

   The projection of a onto b is (a·b / b·b)·b.

   a·b = 2·1 + 3·1 = 5 and b·b = 1 + 1 = 2, so proj = (5/2)·[1, 1] = [2.5, 2.5].

   What is left over: a − proj = [2 − 2.5, 3 − 2.5] = [−0.5, 0.5].

   Is it orthogonal to b? [−0.5, 0.5]·[1, 1] = −0.5 + 0.5 = 0 ✓. That is the point of a projection: it splits a into a part along b and a part at right angles to b.

3. BuildCode it from scratch in NumPy, with no ML library.

   Write cosine_similarity(u, v) and project(a, b). Use them to find the most similar pair among five small word vectors you make up.

   Show answer

   cosine_similarity(u, v) = u·v / (|u|·|v|): 1 means the same direction, 0 means at right angles, −1 means opposite. project(a, b) = (a·b / b·b)·b.

   With made-up vectors whose three numbers mean roughly "royalty, food, vehicle", the most similar pair is king and queen, and apple and banana come next. That is exactly how semantic search ranks results, with real embeddings instead of three hand-written numbers.

   pythonCopy

   import itertools
   import numpy as np
   
   def cosine_similarity(u, v):
       return u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
   
   def project(a, b):
       return (a @ b) / (b @ b) * b
   
   words = {                       # made-up: royalty, food, vehicle
       "king":   np.array([0.9, 0.1, 0.0]),
       "queen":  np.array([0.85, 0.15, 0.05]),
       "apple":  np.array([0.0, 0.9, 0.1]),
       "banana": np.array([0.05, 0.95, 0.0]),
       "car":    np.array([0.1, 0.0, 0.95]),
   }
   pairs = sorted(itertools.combinations(words, 2),
                  key=lambda p: -cosine_similarity(words[p[0]], words[p[1]]))
   for a, b in pairs[:2]:
       print(a, b, round(cosine_similarity(words[a], words[b]), 3))
   print(project(np.array([2., 3.]), np.array([1., 1.])))   # [2.5 2.5]

4. BreakShow when it fails, and why.

   Call cosine_similarity with a zero vector. What happens? Handle it properly and add a test.

   Show answer

   With a zero vector, the bottom of the fraction is 0, so NumPy returns nan with a "invalid value encountered" warning. A zero vector has no direction, so "the angle between them" means nothing.

   Handle it on purpose rather than letting nan spread silently through your results. Two reasonable choices: raise an error (best when a zero vector means a bug upstream), or return 0.0, "not similar to anything" (common in search, where an empty document shouldn't crash anything). Pick one and test it.

   pythonCopy

   import numpy as np
   import pytest
   
   def cosine_similarity(u, v):
       nu, nv = np.linalg.norm(u), np.linalg.norm(v)
       if nu == 0 or nv == 0:
           raise ValueError("cosine similarity is undefined for a zero vector")
       return u @ v / (nu * nv)
   
   def test_zero_vector_raises():
       with pytest.raises(ValueError):
           cosine_similarity(np.zeros(3), np.ones(3))
   
   def test_same_direction_is_one():
       assert np.isclose(cosine_similarity(np.array([1., 2.]), np.array([2., 4.])), 1.0)

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes