Sampling: greedy, temperature, top-k, top-p

Step 1 Understand it

The problem it solves. The same model can sound dull and repetitive or creative and unhinged depending only on how you pick tokens. Greedy decoding tends to loop; pure sampling sometimes picks absurd words from the long tail. Decoding settings are the cheapest, most-used control you have over an LLM's output.

Watch it happen

The idea

The model gives a probability for every possible next token. Greedy decoding always picks the top one. Temperature divides the logits by T before softmax: T < 1 makes choices sharper, T > 1 makes them flatter. Top-k keeps only the k best tokens. Top-p keeps the smallest set whose probabilities add up to p.

How it works, step by step

1. The model gives logits for every token in the vocabulary. Softmax turns them into probabilities.
2. Greedy: always take the most likely token. Deterministic, but prone to repetition.
3. Temperature T: divide the logits by T before the softmax. T < 1 sharpens the distribution (safer); T > 1 flattens it (more varied); T → 0 becomes greedy.
4. Top-k: keep only the k most likely tokens, set the rest to zero, renormalise, then sample.
5. Top-p (nucleus): sort tokens by probability and keep the smallest set whose probabilities add up to at least p (such as 0.9). The set is small when the model is sure and larger when it isn't.
6. Sample from what is left. A fixed random seed makes sampling repeatable.
7. Typical settings: T around 0.7 with top-p 0.9 for chat; T = 0 for extraction, classification and code that must be exact.

Picture it

xychart-beta
  title "Next-token probabilities for logits [2, 1, 0]"
  x-axis "token" ["A", "B", "C"]
  y-axis "probability" 0 --> 1
  line [0.87, 0.12, 0.02]
  line [0.67, 0.24, 0.09]
  line [0.51, 0.31, 0.19]

Analogy

Ordering at a restaurant. Greedy: always the most popular dish. High temperature: feeling adventurous. Top-k: only choose from the top 5 dishes.

Where the analogy breaks

One restaurant order does not change the next one. In text, each choice changes everything after it, so one odd pick can send the text off track.

Worked example

Logits [2, 1, 0]. At T = 1: about [0.67, 0.24, 0.09]. At T = 0.5: about [0.87, 0.12, 0.02].

p = softmax(logits / T)
next_id = np.random.choice(len(p), p=p)

Must-know points

• Temperature changes the shape of the distribution; top-k and top-p change which tokens are allowed.
• Low temperature = focused and repetitive; high = diverse and error-prone.
• Top-p adapts to the model's confidence; top-k uses a fixed count.
• Beam search keeps several high-probability sequences; good for translation, often bland for chat.
• One unlucky pick changes everything after it, so errors compound.

Common mistakes

• Using high temperature for tasks that need exact, repeatable answers.
• Using T = 0 for creative writing and getting loops.
• Setting both a tiny top-k and a high temperature, then wondering why the settings seem to fight.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Read

  How to generate text: using different decoding methods

  Hugging Face blog

• Code

  Build a Large Language Model (From Scratch), Ch 5 (decoding strategies)

  Sebastian Raschka · code free

• Paper

  The Curious Case of Neural Text Degeneration (2019)

  Holtzman et al. · arXiv · optional

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain temperature with the restaurant-ordering picture.

   Show answer

   Ordering at a restaurant. Temperature 0 means you always order the most popular dish: reliable, and boring after a while. A low temperature means you usually pick from the favourites. A high temperature means you feel adventurous and might order anything on the menu, including the dish nobody ever orders. Turn it up too far and you are choosing at random.

   The model's "menu" is every possible next token with a probability. Temperature reshapes those probabilities before choosing: low makes the favourite even more likely, high flattens them so long shots get picked more often. Top-k and top-p are like crossing the worst dishes off the menu first.

2. DeriveWrite the maths, or the steps, from memory.

   For probabilities [0.5, 0.2, 0.15, 0.1, 0.05], which tokens are kept with top-k = 2, and which with top-p = 0.8?

   Show answer

   Probabilities, already sorted: [0.5, 0.2, 0.15, 0.1, 0.05].

   Top-k = 2: keep the 2 most likely tokens, 0.5 and 0.2. Renormalise so they add up to 1: 0.5/0.7 ≈ 0.714 and 0.2/0.7 ≈ 0.286.

   Top-p = 0.8: keep the smallest set of top tokens whose probabilities add up to at least 0.8. Running total: 0.5, then 0.7 (not enough), then 0.85 (enough). So keep the first three: 0.5, 0.2, 0.15. Renormalise by 0.85: about 0.588, 0.235 and 0.176.

   The difference: top-k always keeps exactly k tokens; top-p keeps a number that adapts. When the model is confident, a few tokens pass 0.8; when it is unsure, many do.

3. BuildCode it from scratch in NumPy, with no ML library.

   Add temperature, top-k and top-p to your generate(). Generate 5 samples with each setting and compare.

   Show answer

   generate() gets three knobs. Temperature divides the logits before softmax. Top-k sets every logit outside the k largest to −infinity. Top-p sorts the probabilities, keeps the smallest prefix whose total passes p (always at least the top token), and zeroes the rest. Then renormalise and sample as before.

   Comparing five short samples per setting from the trained Shakespeare GPT: temperature 1.0 is varied, with some made-up words; temperature 0.5 gives safer, more repetitive text with more common words; top-k = 10 and top-p = 0.9 cut off the rarest characters, so there are fewer odd letter combinations while keeping variety. For this tiny character model the differences are modest; for large models, these settings change the feel of the output a lot.

   pythonCopy

   import torch
   import torch.nn.functional as F
   from gpt import GPT, train, decode, encode
   
   @torch.no_grad()
   def generate(model, idx, n, temperature=1.0, top_k=None, top_p=None):
       model.eval()
       for _ in range(n):
           logits = model(idx[:, -model.c.block_size:])[0][:, -1] / temperature
           if top_k is not None:
               kth = torch.topk(logits, top_k).values[:, -1, None]
               logits = logits.masked_fill(logits < kth, float("-inf"))
           probs = F.softmax(logits, dim=-1)
           if top_p is not None:
               sorted_p, order = probs.sort(descending=True)
               drop = sorted_p.cumsum(-1) - sorted_p > top_p           # drop once the total before it passes p
               sorted_p[drop] = 0
               probs = torch.zeros_like(probs).scatter(-1, order, sorted_p)
               probs = probs / probs.sum(-1, keepdim=True)
           idx = torch.cat([idx, torch.multinomial(probs, 1)], dim=1)
       return idx
   
   torch.manual_seed(0)
   model = train(GPT(), steps=1500, log_every=0)
   for name, kwargs in [("T=1.0", {}), ("T=0.5", {"temperature": 0.5}),
                        ("top-k 10", {"top_k": 10}), ("top-p 0.9", {"top_p": 0.9})]:
       torch.manual_seed(1)
       start = torch.tensor([encode("ROMEO:")] * 5)
       out = generate(model, start, 50, **kwargs)
       print(f"--- {name}")
       for row in out:
           print(repr(decode(row.tolist())))

4. BreakShow when it fails, and why.

   Generate with temperature 0.01 and with 3.0. Describe the failure at each end.

   Show answer

   Temperature 0.01 is effectively greedy: the most likely next character every time. The text quickly falls into a loop, the same few words or the same line repeated over and over ("the the the", or one speaker's line again and again), because once the most likely continuation leads back to a state it has seen, it will repeat forever. It is also bland: it never takes the slightly less likely word that makes text interesting.

   Temperature 3.0 flattens the probabilities so much that rare characters are almost as likely as common ones. The text becomes random-looking characters: mixed capitals, punctuation in odd places, and no words.

   The useful range is in between, often 0.7 to 1.0 for creative text and close to 0 for factual or structured answers, usually combined with top-p to cut off the junk tail.

   pythonCopy

   import torch
   from gpt import GPT, train, decode, encode
   
   torch.manual_seed(0)
   model = train(GPT(), steps=1500, log_every=0)
   for temperature in [0.01, 3.0]:
       torch.manual_seed(1)
       out = model.generate(torch.tensor([encode("ROMEO:")]), 150, temperature=temperature)
       print(f"--- temperature {temperature}\n{decode(out[0].tolist())}\n")

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes