In the second edition of this practical book, four Cloudera data scientists present a set of self-contained patterns for performing large-scale data analysis with Spark. The authors bring Spark, statistical methods, and real-world data sets together to teach you how to approach analytics problems by example. Updated for Spark 2.1, this edition acts as an introduction to these techniques and other best practices in Spark programming.
You’ll start with an introduction to Spark and its ecosystem, and then dive into patterns that apply common techniques—including classification, clustering, collaborative filtering, and anomaly detection—to fields such as genomics, security, and finance.
If you have an entry-level understanding of machine learning and statistics, and you program in Java, Python, or Scala, you’ll find the book’s patterns useful for working on your own data applications.
With this book, you will:
Familiarize yourself with the Spark programming model
Become comfortable within the Spark ecosystem
Learn general approaches in data science
Examine complete implementations that analyze large public data sets
Discover which machine learning tools make sense for particular problems
Acquire code that can be adapted to many uses
Table of Contents
Chapter 1 Analyzing Big Data
Chapter 2 Introduction to Data Analysis with Scala and Spark
Chapter 3 Recommending Music and the Audioscrobbler Data Set
Chapter 4 Predicting Forest Cover with Decision Trees
Chapter 5 Anomaly Detection in Network Traffic with K-means Clustering
Chapter 6 Understanding Wikipedia with Latent Semantic Analysis
Chapter 7 Analyzing Co-occurrence Networks with GraphX
Chapter 8 Geospatial and Temporal Data Analysis on the New York City Taxi Trip Data
Chapter 9 Estimating Financial Risk through Monte Carlo Simulation
Chapter 10 Analyzing Genomics Data and the BDG Project
Chapter 11 Analyzing Neuroimaging Data with PySpark and Thunder
Forward:
Ever since we started the Spark project at Berkeley, I’ve been excited about not just building fast parallel systems, but helping more and more people make use of large- scale computing. This is why I’m very happy to see this book, written by four experts in data science, on advanced analytics with Spark. Sandy, Uri, Sean, and Josh have been working with Spark for a while, and have put together a great collection of con‐ tent with equal parts explanations and examples.
The thing I like most about this book is its focus on examples, which are all drawn from real applications on real-world data sets. It’s hard to find one, let alone 10, examples that cover big data and that you can run on your laptop, but the authors have managed to create such a collection and set everything up so you can run them in Spark. Moreover, the authors cover not just the core algorithms, but the intricacies of data preparation and model tuning that are needed to really get good results. You should be able to take the concepts in these examples and directly apply them to your own problems.
Big data processing is undoubtedly one of the most exciting areas in computing today, and remains an area of fast evolution and introduction of new ideas. I hope that this book helps you get started in this exciting new field.
— Matei Zaharia, CTO at Databricks and Vice President, Apache Spark