Practical Simulations for Machine Learning: Using Synthetic Data for AI

by Paris Buttfield-Addison, Mars Buttfield-Addison, Tim Nugent, Jon Manning

Overview

Simulation and synthesis are core parts of the future of AI and machine learning. Consider: programmers, data scientists, and machine learning engineers can create the brain of a self-driving car without the car. Rather than use information from the real world, you can synthesize artificial data using simulations to train traditional machine learning models. That's just the beginning.

With this practical book, you'll explore the possibilities of simulation- and synthesis-based machine learning and AI, concentrating on deep reinforcement learning and imitation learning techniques. AI and ML are increasingly data driven, and simulations are a powerful, engaging way to unlock their full potential.

You'll learn how to:

• Design an approach for solving ML and AI problems using simulations with the Unity engine
• Use a game engine to synthesize images for use as training data
• Create simulation environments designed for training deep reinforcement learning and imitation learning models
• Use and apply efficient general-purpose algorithms for simulation-based ML, such as proximal policy optimization
• Train a variety of ML models using different approaches
• Enable ML tools to work with industry-standard game development tools, using PyTorch, and the Unity ML-Agents and Perception Toolkits

Product Details

ISBN-13:	9781492089926
Publisher:	O'Reilly Media, Incorporated
Publication date:	07/12/2022
Pages:	331
Product dimensions:	7.00(w) x 9.19(h) x 0.91(d)

About the Author

Dr. Paris Buttfield-Addison is cofounder of Secret Lab (https://www.secretlab.com.au and @TheSecretLab on Twitter), a game development studio based in beautiful Hobart, Australia. Secret Lab builds games and game development tools, including the multi-award winning ABC Play School iPad games, Night in the Woods, the Qantas airlines Joey Playbox games, and the Yarn Spinner narrative game framework. Paris formerly worked as mobile product manager for Meebo (acquired by Google), has a degree in medieval history, a PhD in Computing, and writes technical books on mobile and game development (more than 20 so far) for O’Reilly Media. Paris particularly enjoys game design, statistics, law, machine learning, and human-centred technology research. He can be found on Twitter at @parisba and online at http://paris.id.au.

Mars Buttfield-Addison is a computer science and machine learning researcher, as well as freelance creator of STEM educational materials. She is currently working toward her PhD in computer engineering at the University of Tasmania, collaborating with CSIRO's Data61 to investigate how large radio telescope arrays can be adapted to identify and track space debris and satellites in the near field while simultaneously performing deep space observations for astronomy. Mars can be found on Twitter @TheMartianLife and online at https://themartianlife.com.

Dr. Tim Nugent pretends to be a mobile app developer, game designer, tools builder, researcher, and tech author. When he isn’t busy avoiding being found out as a fraud, he spends most of his time designing and creating little apps and games that he won’t let anyone see. Tim spent a disproportionately long time writing this tiny little bio, most of which was spent trying to stick a witty sci-fi reference in, before he simply gave up. Tim can be found on Twitter at @The_McJones, and online at http://lonely.coffee.

Dr. Jon Manning is the cofounder of Secret Lab, an independent game development studio. He’s written a whole bunch of books for O’Reilly Media about Swift, iOS development, and game development, and has a doctorate about jerks on the internet. He’s currently working on Button Squid, a top-down puzzler, and on the critically acclaimed award winning adventure game Night in the Woods, which includes his interactive dialogue system Yarn Spinner. Jon can be found on Twitter at @desplesda, and online at http://desplesda.net.

Table of Contents

Preface ix

Part I The Basics of Simulation and Synthesis

1 Introducing Synthesis and Simulation 3

A Whole New World of ML 4

The Domains 4

Simulation 5

Synthesis 5

The Tools 6

Unity 6

PyTorch via Unity ML-Agents 8

Unity ML-Agents Toolkit 8

Unity Perception 9

The Techniques 9

Reinforcement Learning 10

Imitation Learning 11

Hybrid Learning 12

Summary of Techniques 13

Projects 13

Simulation Projects 14

Synthesis Projects 14

Summary and Next Steps 15

2 Creating Your First Simulation 17

Everybody Remembers Their First Simulation 17

Our Simulation 18

Setting Up 19

Creating the Unity Project 22

Packages All the Way Down 25

The Environment 26

The Floor 26

The Target 28

The Agent 29

Starting and Stopping the Agent 32

Letting the Agent Observe the Environment 35

Letting the Agent Take Actions in the Environment 36

Giving the Agent Rewards for Its Behavior 37

Finishing Touches for the Agent 38

Providing a Manual Control System for the Agent 40

Training with the Simulation 42

Monitoring the Training with TensorBoard 45

When the Training Is Complete 46

What's It All Mean? 48

Coming Up Next 52

3 Creating Your First Synthesized Data 53

Unity Perception 53

The Process 54

Using Unity Perception 55

Creating the Unity Project 56

Creating a Scene 62

Getting the Dice Models 62

A Very Simple Scene 63

Preparing for Synthesis 68

Testing the Scenario 72

Setting Up Our Labels 73

Checking the Labels 75

What's Next? 76

Part II Simulating Worlds for Fun and Profit

4 Creating a More Advanced Simulation 81

Setting Up the Block Pusher 82

Creating the Unity Project 82

The Environment 82

The Floor 83

The Walls 85

The Block 88

The Goal 89

The Agent 92

The Environment 98

Training and Testing 105

5 Creating a Self-Driving Car 107

Creating the Environment 108

The Track 109

The Car 114

Setting Up for ML 117

Training the Simulation 127

Training 128

When the Training Is Complete 130

6 Introducing Imitation Learning 133

Simulation Environment 134

Creating the Ground 135

Creating the Goal 136

The Name's Ball, Agent Ball 140

The Camera 141

Building the Simulation 142

Agent Components 143

Adding Heuristic Controls 146

Observations and Goals 148

Generating Data and Training 149

Creating Training Data 149

Configuring for Training 150

Begin Training 152

Running with Our Trained Model 153

Understanding and Using Imitation Learning 153

7 Advanced Imitation Learning 155

Meet GAIL 155

Do What I Say and Do 157

A GAIL Scenario 157

Modifying the Agent's Actions 160

Modifying the Observations 162

Resetting the Agent 163

Updating the Agent Properties 164

Demonstration Time 164

Training with GAIL 165

Running It and Beyond 167

8 Introducing Curriculum Learning 169

Curriculum Learning in ML 170

A Curriculum Learning Scenario 172

Building in Unity 172

Creating the Ground 174

Creating the Target 174

The Agent 175

Building the Simulation 175

Making the Agent an Agent 176

Actions 177

Observations 181

Heuristic Controls for Humans 182

Creating the Curriculum 184

Resetting the Environment 184

Curriculum Config 185

Training 189

Running It 190

Curriculum Versus Other Approaches 191

What's Next? 193

9 Cooperative Learning 195

A Simulation for Cooperation 195

Building the Environment in Unity 196

Coding the Agents 205

Coding the Environment Manager 208

Coding the Blocks 214

Finalizing the Environment and Agents 216

Training for Cooperation 222

Cooperative Agents or One Big Agent 224

10 Using Cameras in Simulations 225

Observations and Camera Sensors 225

Building a Camera-Only Agent 227

Coding the Camera-Only Agent 228

Adding a New Camera for the Agent 232

Seeing What the Agents Camera Sees 234

Training the Camera-Based Agent 240

Cameras and You 241

11 Working with Python 243

Python All the Way Down 243

Experimenting with an Environment 244

What Can Be Done with Python? 250

Using Your Own Environment 251

Completely Custom Training 255

What's the Point of Python? 257

12 Under the Hood and Beyond 259

Hyper parameters (and Just Parameters) 260

Parameters 260

Reward Parameters 261

Hyperparameters 263

Algorithms 264

Unity Inference Engine and Integrations 266

Using the ML-Agents Gym Wrapper 267

Side Channels 270

Part III Synthetic Data, Real Results

13 Creating More Advanced Synthesized Data 275

Adding Random Elements to the Scene 275

Randomizing the Floor Color 276

Randomizing the Camera Position 278

What's Next? 282

14 Synthetic Shopping 283

Creating the Unity Environment 283

A Perception Camera 287

Faking It Until You Make It 300

Using Synthesized Data 302

Index 305