Transformers and LLMs

Topics, in order

1. Mark "Tokenisation and byte-pair encoding (BPE)" as done

   4.1 Tokenisation and byte-pair encoding (BPE)

   • Watch

     Let's build the GPT Tokenizer

     Andrej Karpathy

   • Interactive

     Tiktokenizer

     interactive · type any text

     Worth it: Type any sentence, number or emoji and instantly see how GPT-style tokenizers split it, colour-coded, with the token count. It makes tokens concrete in seconds.

   • Read

     Byte-Pair Encoding tokenization

     Hugging Face LLM Course

   • Code

     Build a Large Language Model (From Scratch), Ch 2 (working with text data)

     Sebastian Raschka · code free

2. Mark "Token embeddings and positional encoding" as done

   4.2 Token embeddings and positional encoding

   • Read

     The Illustrated Transformer (positional encoding section)

     Jay Alammar

   • Read

     Rotary Embeddings: A Relative Revolution

     EleutherAI blog

   • Code

     Build a Large Language Model (From Scratch), Ch 2 (token and positional embeddings)

     Sebastian Raschka · code free

3. Mark "Self-attention: queries, keys and values" as done

   4.3 Self-attention: queries, keys and values

   • Watch

     Attention in transformers, step-by-step

     3Blue1Brown

     Worth it: The clearest moving-picture explanation of queries, keys and values anywhere: you watch word meanings shift as attention mixes them.

   • Interactive

     Transformer Explainer

     Georgia Tech Polo Club · a live GPT-2 in your browser

   • Read

     The Illustrated Transformer

     Jay Alammar

   • Code

     Build a Large Language Model (From Scratch), Ch 3 (coding attention mechanisms)

     Sebastian Raschka · code free

4. Mark "Multi-head attention and the transformer block" as done

   4.4 Multi-head attention and the transformer block

   • Watch

     Let's build GPT: from scratch, in code, spelled out

     Andrej Karpathy

   • Interactive

     Transformer Explainer

     Georgia Tech Polo Club · a live GPT-2 in your browser

     Worth it: Type your own text and click through a real GPT-2: every head, every block, every probability, animated. The best way to see a transformer block from the inside.

   • Interactive

     LLM Visualization

     Brendan Bycroft · a 3-D walk through a GPT

   • Read

     The Annotated Transformer

     Harvard NLP

   • Code

     Build a Large Language Model (From Scratch), Ch 4 (implementing a GPT model)

     Sebastian Raschka · code free

5. Mark "GPT-style models and next-token prediction" as done

   4.5 GPT-style models and next-token prediction

   • Watch

     Transformers, the tech behind LLMs (But what is a GPT?)

     3Blue1Brown

     Worth it: A visual tour of a GPT from tokens to probabilities, with the shapes of every step drawn out. The best single overview of how these models work.

   • Interactive

     LLM Visualization

     Brendan Bycroft · a 3-D walk through a GPT

   • Read

     The Illustrated GPT-2

     Jay Alammar

   • Code

     Build a Large Language Model (From Scratch), Ch 5 (pretraining on unlabelled data)

     Sebastian Raschka · code free

6. Mark "Encoder models (BERT) vs decoder models (GPT)" as done

   4.6 Encoder models (BERT) vs decoder models (GPT)

   • Watch

     Decoder-Only Transformers, ChatGPT's specific Transformer, Clearly Explained

     StatQuest (Josh Starmer)

   • Read

     The Illustrated BERT, ELMo, and co.

     Jay Alammar

   • Paper

     BERT (2018)

     Devlin et al. · arXiv · optional

7. Mark "Vision transformers and CLIP" as done

   4.7 Vision transformers and CLIP

   • Watch

     Vision Transformer explained, with code

     Umar Jamil

   • Read

     Understanding Deep Learning, Ch 12 (transformers for images)

     Simon Prince · free

   • Code

     OpenCLIP

     GitHub

   • Paper

     An Image is Worth 16x16 Words (ViT, 2020)

     Dosovitskiy et al. · arXiv · optional

   • Paper

     Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021)

     Radford et al. · arXiv · optional

8. Mark "Sampling: greedy, temperature, top-k, top-p" as done

   4.8 Sampling: greedy, temperature, top-k, top-p

   • Read

     How to generate text: using different decoding methods

     Hugging Face blog

   • Code

     Build a Large Language Model (From Scratch), Ch 5 (decoding strategies)

     Sebastian Raschka · code free

   • Paper

     The Curious Case of Neural Text Degeneration (2019)

     Holtzman et al. · arXiv · optional

9. Mark "KV cache and the cost of generation" as done

   4.9 KV cache and the cost of generation

   • Watch

     LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm

     Umar Jamil

   • Read

     Understanding and Coding the KV Cache in LLMs from Scratch

     Sebastian Raschka

10. Mark "Pretraining data and scaling laws" as done

    4.10 Pretraining data and scaling laws

    • Watch

      Deep Dive into LLMs like ChatGPT (the pretraining part)

      Andrej Karpathy

    • Read

      FineWeb: decanting the web for the finest text data at scale

      Hugging Face

    • Paper

      Training Compute-Optimal Large Language Models (Chinchilla, 2022)

      Hoffmann et al. · arXiv · optional

11. Mark "Mixture of experts" as done

    4.11 Mixture of experts

    • Read

      A Visual Guide to Mixture of Experts

      Maarten Grootendorst · dozens of diagrams

      Worth it: Colourful step-by-step diagrams of routers, experts, top-k selection and load balancing. Makes MoE intuitive in one read.

    • Read

      Mixture of Experts Explained

      Hugging Face blog

    • Paper

      Outrageously Large Neural Networks: the Sparsely-Gated MoE Layer (2017)

      Shazeer et al. · arXiv · optional

    • Paper

      Mixtral of Experts (2024)

      Jiang et al. · arXiv · optional

12. Mark "Fine-tuning and LoRA" as done

    4.12 Fine-tuning and LoRA

    • Watch

      LoRA explained, with code

      Umar Jamil

    • Code

      Build a Large Language Model (From Scratch), Appendix E (LoRA)

      Sebastian Raschka · code free

    • Paper

      LoRA: Low-Rank Adaptation of Large Language Models (2021)

      Hu et al. · arXiv · optional

13. Mark "Instruction tuning and preference tuning (RLHF, DPO)" as done

    4.13 Instruction tuning and preference tuning (RLHF, DPO)

    • Watch

      Deep Dive into LLMs like ChatGPT

      Andrej Karpathy

      Worth it: A general-audience tour of the whole pipeline — pretraining, SFT, RLHF, reasoning — by someone who built these systems. Long, but the best big-picture view there is.

    • Read

      Illustrating Reinforcement Learning from Human Feedback (RLHF)

      Hugging Face blog

    • Code

      Build a Large Language Model (From Scratch), Ch 7 (fine-tuning to follow instructions)

      Sebastian Raschka · code free

    • Paper

      Direct Preference Optimization (DPO, 2023)

      Rafailov et al. · arXiv · optional

14. Mark "Reasoning models and test-time compute" as done

    4.14 Reasoning models and test-time compute

    • Watch

      Deep Dive into LLMs like ChatGPT (the 'thinking' and RL sections)

      Andrej Karpathy

    • Read

      Understanding Reasoning LLMs

      Sebastian Raschka

    • Paper

      Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)

      Wei et al. · arXiv · optional

    • Paper

      Self-Consistency Improves Chain of Thought Reasoning (2022)

      Wang et al. · arXiv · optional

    • Paper

      DeepSeek-R1: Incentivizing Reasoning Capability via RL (2025)

      DeepSeek-AI · arXiv · optional