Table of Contents
Preface ix
Part I Introduction to Generative Deep Learning
1 Generative Modeling 1
What Is Generative Modeling? 1
Generative Versus Discriminative Modeling 2
Advances in Machine Learning 4
The Rise of Generative Modeling 5
The Generative Modeling Framework 7
Probabilistic Generative Models 10
Hello Wrodl! 13
Your First Probabilistic Generative Model 14
Naive Bayes 17
Hello Wrodl! Continued 20
The Challenges of Generative Modeling 22
Representation Learning 23
Setting Up Your Environment 27
Summary 29
2 Deep Learning 31
Structured and Unstructured Data 31
Deep Neural Networks 33
Keras and TensorFlow 34
Your First Deep Neural Network 35
Loading the Data 35
Building the Model 37
Compiling the Model 41
Training the Model 43
Evaluating the Model 44
Improving the Model 46
Convolutional Layers 46
Batch Normalization 51
Dropout Layers 54
Putting It All Together 55
Summary 59
3 Variational Autoencoders 61
The Art Exhibition 61
Autoencoders 64
Your First Autoencoder 66
The Encoder 66
The Decoder 68
Joining the Encoder to the Decoder 71
Analysis of the Autoencoder 72
The Variational Art Exhibition 75
Building a Variational Autoencoder 78
The Encoder 78
The Loss Function 84
Analysis of the Variational Autoencoder 85
Using VAEs to Generate Faces 86
Training the VAE 87
Analysis of the VAE 91
Generating New Faces 92
Latent Space Arithmetic 93
Morphing Between Faces 94
Summary 95
4 Generative Adversarial Networks 97
Ganimals 97
Introduction to GANs 99
Your First GAN 100
The Discriminator 101
The Generator 103
Training the GAN 107
GAN Challenges 112
Oscillating Loss 112
Mode Collapse 113
Uninformative Loss 114
Hyperparameters 114
Tackling the GAN Challenges 115
Wasserstein GAN 115
Wasserstein Loss 115
The Lipschitz Constraint 117
Weight Clipping 118
Training the WGAN 119
Analysis of the WGAN 120
WGAN-GP 121
The Gradient Penalty Loss 121
Analysis of WGAN-GP 125
Summary 127
Part II Teaching Machines to Paint, Write, Compose, and Play
5 Paint 131
Apples and Organges 132
Cycle GAN 135
Your First CycleGAN 137
Overview 137
The Generators (U-Net) 139
The Discriminators 142
Compiling the CycleGAN 144
Training the CycleGAN 146
Analysis of the CycleGAN 147
Creating a CycleGAN to Paint Like Monet 149
The Generators (ResNet) 150
Analysis of the CycleGAN 151
Neural Style Transfer 153
Content Loss 154
Style Loss 156
Total Variance Loss 160
Running the Neural Style Transfer 160
Analysis of the Neural Style Transfer Model 161
Summary 162
6 Write 165
The Literary Society for Troublesome Miscreants 166
Long Short-Term Memory Networks 167
Your First LSTM Network 168
Tokenization 168
Building the Dataset 171
The LSTM Architecture 172
The Embedding Layer 172
The LSTM Layer 174
The LSTM Cell 176
Generating New Text 179
RNN Extensions 183
Stacked Recurrent Networks 183
Gated Recurrent Units 185
Bidirectional Cells 187
Encoder-Decoder Models 187
A Question and Answer Generator 190
A Question-Answer Dataset 191
Model Architecture 192
Inference 196
Model Results 198
Summary 200
7 Compose 201
Preliminaries 202
Musical Notation 202
Your First Music-Generating RNN 205
Attention 206
Building an Attention Mechanism in Keras 208
Analysis of the RNN with Attention 213
Attention in Encoder-Decoder Networks 217
Generating Polyphonic Music 221
The Musical Organ 221
Your First MuseGAN 223
The MuseGAN Generator 226
Chords, Style, Melody, and Groove 227
The Bar Generator 229
Putting It All Together 230
The Critic 232
Analysis of the MuseGAN 233
Summary 235
8 Play 237
Reinforcement Learning 238
OpenAI Gym 239
World Model Architecture 241
The Variational Autoencoder 242
The MDN-RNN 243
The Controller 243
Setup 244
Training Process Overview 245
Collecting Random Rollout Data 246
Training the VAE 248
The VAE Architecture 249
Exploring the VAE 252
Collecting Data to Train the RNN 255
Training the MDN-RNN 257
The MDN-RNN Architecture 258
Sampling the Next z and Reward from the MDN-RNN 259
The MDN-RNN Loss Function 259
Training the Controller 261
The Controller Architecture 262
CMA-ES 262
Parallelizing CMA-ES 265
Output from the Controller Training 267
In-Dream Training 268
In-Dream Training the Controller 270
Challenges of In-Dream Training 272
Summary 273
9 The Future of Generative Modeling 275
Five Years of Progress 275
The Transformer 277
Positional Encoding 279
Multihead Attention 280
The Decoder 283
Analysis of the Transformer 283
BERT 285
GPT-2 285
MuseNet 286
Advances in Image Generation 287
ProGAN 287
Self-Attention GAN (SAGAN) 289
BigGAN 291
StyleGAN 292
Applications of Generative Modeling 296
AI Art 296
AI Music 297
10 Conclusion 299
Index 303