Programming PyTorch for Deep Learning: Creating and Deploying Deep Learning Applications

by Ian Pointer

Overview

Take the next steps toward mastering deep learning, the machine learning method that's transforming the world around us by the second. In this practical book, you'll get up to speed on key ideas using Facebook's open source PyTorch framework and gain the latest skills you need to create your very own neural networks.

Ian Pointer shows you how to set up PyTorch on a cloud-based environment, then walks you through the creation of neural architectures that facilitate operations on images, sound, text, and more through deep dives into each element. He also covers the critical concepts of applying transfer learning to images, debugging models, and PyTorch in production.

• Learn how to deploy deep learning models to production
• Explore PyTorch use cases from several leading companies
• Learn how to apply transfer learning to images
• Apply cutting-edge NLP techniques using a model trained on Wikipedia
• Use PyTorch's torchaudio library to classify audio data with a convolutional-based model
• Debug PyTorch models using TensorBoard and flame graphs
• Deploy PyTorch applications in production in Docker containers and Kubernetes clusters running on Google Cloud

Product Details

ISBN-13:	9781492045359
Publisher:	O'Reilly Media, Incorporated
Publication date:	10/08/2019
Pages:	217
Product dimensions:	6.90(w) x 9.10(h) x 0.60(d)

About the Author

Ian Pointer is a data engineer, specializing in machine learning solutions (including deep learning techniques) for multiple Fortune 100 clients. Ian is currently at Lucidworks, where he works on cutting-edge NLP applications and engineering.



He immigrated to the United States from the United Kingdom in 2011 and became an American citizen in 2017.

Table of Contents

Preface ix

1 Getting Started with PyTorch 1

Building a Custom Deep Learning Machine 1

GPU 2

CPU/Motherboard 2

RAM 2

Storage 2

Deep Learning in the Cloud 3

Google Colaboratory 3

Cloud Providers 5

Which Cloud Provider Should I Use? 7

Using Jupyter Notebook 7

Installing PyTorch from Scratch 8

Download CUDA 8

Anaconda 9

Finally, PyTorch! (and Jupyter Notebook) 9

Tensors 10

Tensor Operations 11

Tensor Broadcasting 13

Conclusion 14

Further Reading 14

2 Image Classification with PyTorch 15

Our Classification Problem 15

Traditional Challenges 17

But First, Data 17

PyTorch and Data Loaders 18

Building a Training Dataset 18

Building Validation and Test Datasets 20

Finally, a Neural Network! 21

Activation Functions 22

Creating a Network 22

Loss Functions 23

Optimizing 24

Training 26

Making It Work on the GPU 27

Putting It All Together 27

Making Predictions 28

Model Saving 29

Conclusion 30

Further Reading 31

3 Convolutional Neural Networks 33

Our First Convolutional Model 33

Convolutions 34

Pooling 37

Dropout 38

History of CNN Architectures 39

AlexNet 39

Inception/GoogLeNet 40

VGG 41

ResNet 43

Other Architectures Are Available! 43

Using Pretrained Models in PyTorch 44

Examining a Model's Structure 44

BatchNorm 47

Which Model Should You Use? 48

One-Stop Shopping for Models: PyTorch Hub 48

Conclusion 49

Further Reading 49

4 Transfer Learning and Other Tricks 51

Transfer Learning with ResNet 51

Finding That Learning Rate 53

Differential Learning Rates 56

Data Augmentation 57

Torchvision Transforms 58

Color Spaces and Lambda Transforms 63

Custom Transform Classes 64

Start Small and Get Bigger! 65

Ensembles 66

Conclusion 67

Further Reading 67

5 Text Classification 69

Recurrent Neural Networks 69

Long Short-Term Memory Networks 71

Gated Recurrent Units 73

biLSTM 73

Embeddings 74

Torchtext 76

Getting Our Data: Tweets! 77

Denning Fields 78

Building a Vocabulary 80

Creating Our Model 82

Updating the Training Loop 83

Classifying Tweets 84

Data Augmentation 84

Random Insertion 85

Random Deletion 85

Random Swap 86

Back Translation 86

Augmentation and torchtext 87

Transfer Learning? 88

Conclusion 88

Further Reading 89

6 A Journey into Sound 91

Sound 91

The ESC-50 Dataset 93

Obtaining the Dataset 93

Playing Audio in Jupyter 93

Exploring ESC-50 94

SoX and LibROSA 95

Torchaudio 95

Building an ESC-50 Dataset 96

A CNN Model for ESC-50 98

This Frequency Is My Universe 100

Mel Spectrograms 100

A New Dataset 102

A Wild ResNet Appears 104

Finding a Learning Rate 105

Audio Data Augmentation 107

Torchaudio Transforms 107

SoX Effect Chains 107

SpecAugment 109

Further Experiments 113

Conclusion 113

Further Reading 114

7 Debugging PyTorch Models 115

It's 3 a.m. What Is Your Data Doing? 115

TensorBoard 116

Installing TensorBoard 116

Sending Data to TensorBoard 117

PyTorch Hooks 120

Plotting Mean and Standard Deviation 121

Class Activation Mapping 122

Flame Graphs 125

Installing py-spy 127

Reading Flame Graphs 128

Fixing a Slow Transformation 129

Debugging GPU Issues 132

Checking Your GPU 132

Gradient Checkpointing 134

Conclusion 136

Further Reading 136

8 PyTorch in Production 137

Model Serving 137

Building a Flask Service 138

Setting Up the Model Parameters 140

Building the Docker Container 141

Local Versus Cloud Storage 144

Logging and Telemetry 145

Deploying on Kubernetes 147

Setting Up on Google Kubernetes Engine 147

Creating a k8s Cluster 148

Scaling Services 149

Updates and Cleaning Up 149

TorchScript 150

Tracing 150

Scripting 153

TorchScript Limitations 154

Working with libTorch 156

Obtaining libTorch and Hello World 156

Importing a TorchScript Model 158

Conclusion 159

Further Reading 160

9 PyTorch in the Wild 161

Data Augmentation: Mixed and Smoothed 161

Mixup 161

Label Smoothing 165

Computer, Enhance! 166

Introduction to Super-Resolution 167

An Introduction to GANs 169

The Forger and the Critic 170

Training a GAN 171

The Dangers of Mode Collapse 172

ESRGAN 173

Further Adventures in Image Detection 173

Object Detection 173

Faster R-CNN and Mask R-CNN 175

Adversarial Samples 177

Black-Box Attacks 180

Defending Against Adversarial Attacks 180

More Than Meets the Eye: The Transformer Architecture 181

Paying Attention 181

Attention Is All You Need 182

BERT 183

FastBERT 183

GPT-2 185

Generating Text with GPT-2 185

ULMFiT 187

What to Use? 189

Conclusion 190

Further Reading 190

Index 193