Table of Contents

Introduction 1

About This Book 1

Foolish Assumptions 2

Icons Used in This Book 2

Beyond the Book 3

Where to Go from Here 4

Part 1: Getting to Know Tensorflow 5

Chapter 1: Introducing Machine Learning with TensorFlow 7

Understanding Machine Learning 7

The Development of Machine Learning 8

Statistical regression 9

Reverse engineering the brain 10

Steady progress 11

The computing revolution 12

The rise of big data and deep learning 12

Machine Learning Frameworks 13

Torch 14

Theano 14

Caffe 14

Keras 15

TensorFlow 15

Chapter 2: Getting Your Feet Wet 17

Installing TensorFlow 17

Python and pip/pip3 18

Installing on Mac OS 19

Installing on Linux 20

Installing on Windows 20

Exploring the TensorFlow Installation 21

Running Your First Application 22

Exploring the example code 23

Launching Hello TensorFlow! 23

Setting the Style 24

Chapter 3: Creating Tensors and Operations 27

Creating Tensors 27

Creating Tensors with Known Values 28

The constant function 30

zeros, ones, and fill 30

Creating sequences 31

Creating Tensors with Random Values 31

Transforming Tensors.33

Creating Operations 35

Basic math operations 35

Rounding and comparison 37

Exponents and logarithms 38

Vector and matrix operations 39

Putting Theory into Practice 42

Chapter 4: Executing Graphs in Sessions 45

Forming Graphs 46

Accessing graph data 47

Creating GraphDefs 49

Creating and Running Sessions 51

Creating sessions 51

Executing a session 52

Interactive sessions 53

Writing Messages to the Log 54

Visualizing Data with TensorBoard 56

Running TensorBoard 57

Generating summary data 57

Creating custom summaries 59

Writing summary data 59

Putting Theory into Practice 62

Chapter 5: Training 65

Training in TensorFlow 66

Formulating the Model 66

Looking at Variables 67

Creating variables 68

Initializing variables 68

Determining Loss 69

Minimizing Loss with Optimization 70

The Optimizer class 70

The GradientDescentOptimizer 71

The MomentumOptimizer 75

The AdagradOptimizer 76

The AdamOptimizer 77

Feeding Data into a Session 78

Creating placeholders 79

Defining the feed dictionary 79

Stochasticity 80

Monitoring Steps, Global Steps, and Epochs 80

Saving and Restoring Variables 82

Saving variables 82

Restoring variables 83

Working with SavedModels 84

Saving a SavedModel 85

Loading a SavedModel 86

Putting Theory into Practice 86

Visualizing the Training Process 89

Session Hooks 90

Creating a session hook 91

Creating a MonitoredSession 93

Putting theory into practice 94

Part 2: Implementing Machine Learning 97

Chapter 6: Analyzing Data with Statistical Regression 99

Analyzing Systems Using Regression 100

Linear Regression: Fitting Lines to Data 100

Polynomial Regression: Fitting Polynomials to Data 103

Binary Logistic Regression: Classifying Data into Two Categories 105

Setting up the problem 105

Defining models with the logistic function 106

Computing loss with maximum likelihood estimation 107

Putting theory into practice 108

Multinomial Logistic Regression: Classifying Data into Multiple Categories 110

The Modified National Institute of Science and Technology (MNIST) Dataset 110

Defining the model with the softmax function 113

Computing loss with cross entropy 114

Putting theory into practice 115

Chapter 7: Introducing Neural Networks and Deep Learning 117

From Neurons to Perceptrons 117

Neurons 118

Perceptrons 119

Improving the Model 121

Weights 121

Bias 122

Activation functions 123

Layers and Deep Learning 127

Layers 128

Deep learning 129

Training with Backpropagation 129

Implementing Deep Learning 131

Tuning the Neural Network 133

Input standardization 134

Weight initialization 135

Batch normalization 136

Regularization 139

Managing Variables with Scope 141

Variable scope 141

Retrieving variables from collections 142

Scopes for names and arguments 143

Improving the Deep Learning Process 143

Creating tuned layers 144

Putting theory into practice 145

Chapter 8: Classifying Images with Convolutional Neural Networks (CNNs) 149

Filtering Images 149

Convolution 150

Averaging Filter 151

Filters and features 152

Feature detection analogy 153

Setting convolution parameters 153

Convolutional Neural Networks (CNNs) 155

Creating convolution layers 156

Creating pooling layers 158

Putting Theory into Practice 160

Processing CIFAR images 160

Classifying CIFAR images in code 162

Performing Image Operations 166

Converting images 166

Color processing 169

Rotating and mirroring 170

Resizing and cropping 172

Convolution 174

Putting Theory into Practice 175

Chapter 9: Analyzing Sequential Data with Recurrent Neural Networks (RNNs) 179

Recurrent Neural Networks (RNNs) 180

RNNs and recursive functions 181

Training RNNs 182

Creating RNN Cells 183

Creating a basic RNN 185

Predicting text with RNNs 188

Creating multilayered cells 190

Creating dynamic RNNs 191

Long Short-Term Memory (LSTM) Cells 192

Creating LSTMs in code 194

Predicting text with LSTMs 196

Gated Recurrent Units (GRUs) 196

Creating GRUs in code 197

Predicting text with GRUs 198

Part 3: Simplifying and Accelerating Tensorflow 199

Chapter 10: Accessing Data with Datasets and Iterators 201

Datasets 201

Creating datasets 202

Processing datasets 208

Iterators 213

One-shot iterators 213

Initializable iterators 215

Reinitializable iterators 216

Feedable iterators 217

Putting Theory into Practice 218

Bizarro Datasets 221

Loading data from CSV files 222

Loading the Iris and Boston datasets 223

Chapter 11: Using Threads, Devices, and Clusters 225

Executing with Multiple Threads 226

Configuring a new session 226

Configuring a running session 228

Configuring Devices 229

Building TensorFlow from source 229

Assigning operations to devices 235

Configuring GPU usage 237

Executing TensorFlow in a Cluster 238

Creating a ClusterSpec 239

Creating a server 240

Specifying jobs and tasks 241

Running a simple cluster 244

Chapter 12: Developing Applications with Estimators 247

Introducing Estimators 248

Training an Estimator 248

Testing an Estimator 250

Running an Estimator 250

Creating Input Functions 251

Configuring an Estimator 252

Using Feature Columns 253

Creating and Using Estimators 256

Linear regressors 257

DNN classifiers 260

Combined linear-DNN classifiers 262

Wide and deep learning 263

Analyzing census data 264

Running Estimators in a Cluster 269

Accessing Experiments 270

Creating an experiment 271

Methods of the experiment class 272

Running an experiment 273

Putting theory into practice 274

Chapter 13: Running Applications on the Google Cloud Platform (GCP) 277

Overview 278

Working with GCP projects 278

Creating a new project 279

Billing 279

Accessing the machine learning engine 280

The Cloud Software Development Kit (SDK) 280

The gcloud Utility 281

Google Cloud Storage 283

Buckets 283

Objects and virtual hierarchy 285

The gsutil utility 286

Preparing for Deployment 290

Receiving arguments 290

Packaging TensorFlow code 291

Executing Applications with the Cloud SDK 293

Local execution 294

Deploying to the cloud 295

Configuring a Cluster in the Cloud 299

Setting the training input 300

Obtaining the training output 303

Setting the prediction input 304

Obtaining the prediction output 305

Part 4: The Part of Tens 307

Chapter 14: The Ten Most Important Classes 309

Tensor 309

Operation 310

Graph 310

Session 311

Variable 311

Optimizer 312

Estimator 312

Dataset 312

Iterator 313

Saver 313

Chapter 15: Ten Recommendations for Training Neural Networks 315

Select a Representative Dataset 315

Standardize Your Data 316

Use Proper Weight Initialization 316

Start with a Small Number of Layers 316

Add Dropout Layers 317

Train with Small, Random Batches 317

Normalize Batch Data 317

Try Different Optimization Algorithms 318

Set the Right Learning Rate 318

Check Weights and Gradients 318

Index 319