Phase 7 of 7 · 4 topics

Generative models and reinforcement learning

Models that create new data, and models that learn by trial and error. These ideas power image generators, and the reinforcement learning behind RLHF.

Every model so far learned to predict a label. Generative models learn what the data itself looks like, so they can make new examples. Reinforcement learning has no labels at all, only rewards, so the model must discover good actions by trying.

0 of 4 done

Core resources for this phase

Book
Understanding Deep Learning, Chapters 17–19 (Simon Prince) Free

VAEs, diffusion models and reinforcement learning, in the same clear style as the rest of the book.
Blog
Lilian Weng's blog Free

Careful, well-illustrated derivations of VAEs, diffusion and policy gradients. Read each post after the book chapter.
Course
Spinning Up in Deep RL (OpenAI) Free

The clearest practical introduction to deep reinforcement learning, with code.

Topics, in order

Mark "Autoencoders and VAEs (the ELBO)" as done
7.1 Autoencoders and VAEs (the ELBO)
- Watch
  
  Variational Autoencoders
  
  Arxiv Insights
- Read
  
  Understanding Deep Learning, Ch 17 (variational autoencoders)
  
  Simon Prince · free
  
  Worth it: Exceptionally clear figures of the encoder, the latent space, the ELBO and the reparameterisation trick, building up one step at a time.
- Read
  
  From Autoencoder to Beta-VAE
  
  Lilian Weng
- Paper
  
  Auto-Encoding Variational Bayes (2013)
  
  Kingma and Welling · arXiv · optional
Mark "Diffusion models" as done
7.2 Diffusion models
- Read
  
  The Illustrated Stable Diffusion
  
  Jay Alammar · illustrated
  
  Worth it: Colourful step-by-step pictures of noising, denoising, latent space and text conditioning. The clearest picture of how image generators work.
- Read
  
  Understanding Deep Learning, Ch 18 (diffusion models)
  
  Simon Prince · free
- Read
  
  The Annotated Diffusion Model
  
  Hugging Face blog · code walkthrough
- Read
  
  What are Diffusion Models?
  
  Lilian Weng
- Paper
  
  Denoising Diffusion Probabilistic Models (DDPM, 2020)
  
  Ho, Jain, Abbeel · arXiv · optional
Mark "Reinforcement learning basics: states, actions, rewards, values" as done
7.3 Reinforcement learning basics: states, actions, rewards, values
- Watch
  
  RL Course, Lecture 1: Introduction to Reinforcement Learning
  
  David Silver
- Interactive
  
  Reinforcement Learning
  
  MLU-Explain (Amazon) · animated grid world
  
  Worth it: An animated agent learns a grid world as you scroll, with its Q-values and policy updating live. This topic, in moving pictures.
- Read
  
  Spinning Up, Part 1: Key Concepts in RL
  
  OpenAI
- Read
  
  Reinforcement Learning: An Introduction, Ch 3 and 6
  
  Sutton and Barto · free
Mark "Policy gradients and PPO" as done
7.4 Policy gradients and PPO
- Read
  
  Spinning Up, Part 3: Intro to Policy Optimization
  
  OpenAI
  
  Worth it: A short, careful derivation of the policy gradient with runnable code, from the people who popularised PPO. The best bridge from idea to implementation.
- Read
  
  Policy Gradient Algorithms
  
  Lilian Weng
- Read
  
  Understanding Deep Learning, Ch 19 (reinforcement learning)
  
  Simon Prince · free
- Paper
  
  Proximal Policy Optimization Algorithms (2017)
  
  Schulman et al. · arXiv · optional

Go deeper (optional)

Book
Reinforcement Learning: An Introduction (Sutton and Barto) Free

The standard RL textbook. Chapters 3 and 6 cover the basics used here.
Video
David Silver's RL course (UCL / DeepMind) Free

Ten classic lectures. Watch lectures 1 to 5 for the foundations.

Generative models and reinforcement learning

Core resources for this phase

Topics, in order

7.1 Autoencoders and VAEs (the ELBO)

7.2 Diffusion models

7.3 Reinforcement learning basics: states, actions, rewards, values

7.4 Policy gradients and PPO

Go deeper (optional)