Programming Pig: Dataflow Scripting with Hadoop

by Alan Gates, Daniel Dai

Overview

For many organizations, Hadoop is the first step for dealing with massive amounts of data. The next step? Processing and analyzing datasets with the Apache Pig scripting platform. With Pig, you can batch-process data without having to create a full-fledged application, making it easy to experiment with new datasets.

Updated with use cases and programming examples, this second edition is the ideal learning tool for new and experienced users alike. You’ll find comprehensive coverage on key features such as the Pig Latin scripting language and the Grunt shell. When you need to analyze terabytes of data, this book shows you how to do it efficiently with Pig.

• Delve into Pig’s data model, including scalar and complex data types
• Write Pig Latin scripts to sort, group, join, project, and filter your data
• Use Grunt to work with the Hadoop Distributed File System (HDFS)
• Build complex data processing pipelines with Pig’s macros and modularity features
• Embed Pig Latin in Python for iterative processing and other advanced tasks
• Use Pig with Apache Tez to build high-performance batch and interactive data processing applications
• Create your own load and store functions to handle data formats and storage mechanisms

Product Details

ISBN-13:	9781491937099
Publisher:	O'Reilly Media, Incorporated
Publication date:	12/05/2016
Pages:	365
Product dimensions:	6.90(w) x 9.10(h) x 0.80(d)

About the Author

Alan is co-founder of Hortonworks and an original member of the engineering team that took Pig from a Yahoo! Labs research project to a successful Apache open source project. Alan also designed HCatalog and guided its adoption as an Apache Incubator project. Alan has a BS in Mathematics from Oregon State Universityand a MA in Theology from Fuller Theological Seminary. He is also the author of Programming Pig, a book from O’Reilly Press. Follow Alan on Twitter: @alanfgates.

Daniel is an Apache Pig PMC member/committer involved with Pig for 6 years at Yahoo and now at Hortonworks. He has a PhD in Computer Science from University of Central Florida, with a specialization in distributed computing, data mining and computer security.

Table of Contents

Preface; Data Addiction; Who Should Read This Book; Conventions Used in This Book; Code Examples in This Book; Using Code Examples; Safari® Books Online; How to Contact Us; Acknowledgments; Chapter 1: Introduction; 1.1 What Is Pig?; 1.2 Pig’s History; Chapter 2: Installing and Running Pig; 2.1 Downloading and Installing Pig; 2.2 Running Pig; Chapter 3: Grunt; 3.1 Entering Pig Latin Scripts in Grunt; 3.2 HDFS Commands in Grunt; 3.3 Controlling Pig from Grunt; Chapter 4: Pig’s Data Model; 4.1 Types; 4.2 Schemas; Chapter 5: Introduction to Pig Latin; 5.1 Preliminary Matters; 5.2 Input and Output; 5.3 Relational Operations; 5.4 User Defined Functions; Chapter 6: Advanced Pig Latin; 6.1 Advanced Relational Operations; 6.2 Integrating Pig with Legacy Code and MapReduce; 6.3 Nonlinear Data Flows; 6.4 Controlling Execution; 6.5 Pig Latin Preprocessor; Chapter 7: Developing and Testing Pig Latin Scripts; 7.1 Development Tools; 7.2 Testing Your Scripts with PigUnit; Chapter 8: Making Pig Fly; 8.1 Writing Your Scripts to Perform Well; 8.2 Writing Your UDF to Perform; 8.3 Tune Pig and Hadoop for Your Job; 8.4 Using Compression in Intermediate Results; 8.5 Data Layout Optimization; 8.6 Bad Record Handling; Chapter 9: Embedding Pig Latin in Python; 9.1 Compile; 9.2 Bind; 9.3 Run; 9.4 Utility Methods; Chapter 10: Writing Evaluation and Filter Functions; 10.1 Writing an Evaluation Function in Java; 10.2 Algebraic Interface; 10.3 Accumulator Interface; 10.4 Python UDFs; 10.5 Writing Filter Functions; Chapter 11: Writing Load and Store Functions; 11.1 Load Functions; 11.2 Store Functions; Chapter 12: Pig and Other Members of the Hadoop Community; 12.1 Pig and Hive; 12.2 Cascading; 12.3 NoSQL Databases; 12.4 Metadata in Hadoop; Built-in User Defined Functions and Piggybank; Built-in UDFs; Piggybank; Overview of Hadoop; MapReduce; Hadoop Distributed File System; Colophon;