Hamropedia

Phase 3 of 7 · 16 topics

Deep learning foundations

Build neural networks from a single neuron up. Write backprop by hand first. Then you can trust PyTorch, because you know what it does.

A neural network is many small logistic regressions stacked in layers and trained together with backprop. Phase 2 made the bricks. Phase 3 builds the wall.

0 of 16 done

Core resources for this phase

Topics, in order

  1. 3.1 The neuron and the multi-layer perceptron

  2. 3.2 Activation functions, and why non-linearity matters

  3. 3.3 Backpropagation by hand

  4. 3.4 Autograd and computational graphs

  5. 3.5 Softmax and cross-entropy loss

  6. 3.6 Optimisers: SGD, momentum, Adam; learning-rate schedules

  7. 3.7 Weight initialisation; vanishing and exploding gradients

  8. 3.8 Batch norm and layer norm

  9. 3.9 Regularisation: dropout, weight decay, augmentation, early stopping

  10. 3.10 PyTorch: nn.Module, DataLoader and the training loop

  11. 3.11 Convolutional networks: convolution, pooling, receptive field

  12. 3.12 Residual connections (ResNet)

  13. 3.13 RNNs, and why long sequences are hard

  14. 3.14 Embeddings: representing things as vectors

  15. 3.15 Debugging training: overfit one batch, read loss curves

  16. 3.16 LSTMs and GRUs: gates for long memory

Go deeper (optional)