Apache Oozie: The Workflow Scheduler for Hadoop

by Mohammad Islam, Aravind Srinivasan

Overview

Get a solid grounding in Apache Oozie, the workflow scheduler system for managing Hadoop jobs. With this hands-on guide, two experienced Hadoop practitioners walk you through the intricacies of this powerful and flexible platform, with numerous examples and real-world use cases.

Once you set up your Oozie server, you’ll dive into techniques for writing and coordinating workflows, and learn how to write complex data pipelines. Advanced topics show you how to handle shared libraries in Oozie, as well as how to implement and manage Oozie’s security capabilities.

• Install and configure an Oozie server, and get an overview of basic concepts
• Journey through the world of writing and configuring workflows
• Learn how the Oozie coordinator schedules and executes workflows based on triggers
• Understand how Oozie manages data dependencies
• Use Oozie bundles to package several coordinator apps into a data pipeline
• Learn about security features and shared library management
• Implement custom extensions and write your own EL functions and actions
• Debug workflows and manage Oozie’s operational details

Product Details

ISBN-13:	9781449369927
Publisher:	O'Reilly Media, Incorporated
Publication date:	05/24/2015
Pages:	269
Product dimensions:	7.00(w) x 9.20(h) x 0.60(d)

About the Author

Mohammad Kamrul Islam is currently working at Uber in data engineering team as a Staff Software Engineer. Previously, he worked at Linkedin for more than two years as Staff Software Engineer in the Hadoop development team. Before that, he worked at Yahoo for nearly five years as an Oozie architect/technical lead. His fingerprints can be found all over Oozie and is a respected voice in the Oozie community. He has been intimately involved with the Apache Hadoop ecosystem since 2009. Mohammad has a Ph.D. in Computer Science with a specialization in parallel job scheduling from Ohio State University. He received his MSCS degree from Wright State University, Ohio and
BSCS from Bangladesh Universityof Engineering and Technology (BUET). He is a Project Management Committee (PMC) member of both Apache Oozie and Apache TEZ and frequently contributes to Apache YARN/MapReduce and Apache Hive. He was elected as the PMC chair and Vice-President of Oozie as part of the Apache Software Foundation from 2013 through 2015.

Aravind Srinivasan has been involved with Hadoop in general and Oozie in particular since 2008. He is currently a Lead Application Architect at Altiscale, a Hadoop-as-a-service company, where he helps customers with Hadoop application design and architecture. His association with Big Data and Hadoop started during his time at Yahoo, where he spent almost six years working on various data pipelines for advertising systems. He has extensive experience building complicated, low latency data pipelines and also in porting legacy pipelines to Oozie. He drove a lot of Oozie’s requirements as a customer in its early days of adoption inside Yahoo and later spent some time as a Product Manager in Yahoo’s Hadoop team where he contributed further to Oozie’s roadmap. He also spent a year after Yahoo at Think Big Analytics, a Hadoop consulting firm, where he got to consult on some interesting and challenging Big Data integration projects at Facebook. He has a Masters in Computer Science from Arizona State and lives in Silicon Valley.