Systems and scale

Topics, in order

1. Mark "How a GPU works: compute vs memory" as done

   6.1 How a GPU works: compute vs memory

   • Watch

     GPU MODE lectures

     GPU MODE

   • Read

     Making Deep Learning Go Brrrr From First Principles

     Horace He

     Worth it: The clearest explanation of compute-bound, memory-bound and overhead-bound code, with memorable pictures. After it you will know where your training time goes.

   • Read

     GPU Glossary

     Modal

2. Mark "Mixed precision: fp16 and bf16" as done

   6.2 Mixed precision: fp16 and bf16

   • Watch

     Let's reproduce GPT-2 (124M): the mixed-precision section

     Andrej Karpathy

   • Read

     Automatic Mixed Precision recipe

     PyTorch tutorials

   • Paper

     Mixed Precision Training (2017)

     Micikevicius et al. · arXiv · optional

3. Mark "Training memory: weights, gradients, optimiser states, activations" as done

   6.3 Training memory: weights, gradients, optimiser states, activations

   • Read

     Transformer Math 101

     EleutherAI blog

     Worth it: All the back-of-the-envelope formulas for compute and memory in one short page: weights, optimiser states, activations, and how to estimate them for any model.

   • Read

     The Ultra-Scale Playbook (memory usage section)

     Hugging Face

   • Paper

     ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019)

     Rajbhandari et al. · arXiv · optional

4. Mark "Parallelism: data, tensor and pipeline (the ideas)" as done

   6.4 Parallelism: data, tensor and pipeline (the ideas)

   • Read

     The Ultra-Scale Playbook

     Hugging Face · interactive diagrams

     Worth it: A long but beautifully illustrated guide to every kind of parallelism, with interactive memory and speed calculators, from people who train large models.

   • Read

     Getting Started with Distributed Data Parallel

     PyTorch tutorials

   • Paper

     Megatron-LM (2019)

     Shoeybi et al. · arXiv · optional

5. Mark "Inference speed: batching, quantisation, FlashAttention" as done

   6.5 Inference speed: batching, quantisation, FlashAttention

   • Read

     A Visual Guide to Quantization

     Maarten Grootendorst · many diagrams

     Worth it: Dozens of clear diagrams of number formats, scales, outliers and the main quantisation methods. Makes a fiddly topic visual and intuitive.

   • Paper

     FlashAttention (2022)

     Dao et al. · arXiv · optional

   • Paper

     PagedAttention / vLLM (2023)

     Kwon et al. · arXiv · optional

6. Mark "Profiling PyTorch code" as done

   6.6 Profiling PyTorch code

   • Read

     PyTorch Profiler recipe

     PyTorch tutorials

   • Read

     Performance Tuning Guide

     PyTorch tutorials

7. Mark "GPU kernels with Triton" as done

   6.7 GPU kernels with Triton

   • Watch

     GPU MODE lectures on Triton

     GPU MODE

   • Code

     GPU Puzzles

     Sasha Rush · GitHub · interactive notebook

     Worth it: Small, visual puzzles that teach how GPU kernels index data, one idea at a time, in a notebook. The friendliest way into thinking like a GPU.

   • Code

     Triton tutorials: vector addition, fused softmax, matrix multiplication

     triton-lang.org

8. Mark "Pruning and distillation" as done

   6.8 Pruning and distillation

   • Read

     Pruning Tutorial

     PyTorch tutorials

   • Paper

     Distilling the Knowledge in a Neural Network (2015)

     Hinton, Vinyals, Dean · arXiv · optional

   • Paper

     The Lottery Ticket Hypothesis (2018)

     Frankle and Carbin · arXiv · optional

9. Mark "Speculative decoding" as done

   6.9 Speculative decoding

   • Read

     Assisted Generation: a new direction toward low-latency text generation

     Hugging Face blog · with animations

     Worth it: Animated diagrams of a draft model proposing and a big model checking, plus measured speed-ups. Makes the idea click quickly.

   • Paper

     Fast Inference from Transformers via Speculative Decoding (2022)

     Leviathan, Kalman, Matias · arXiv · optional