Learning Spark by Holden Karau O'Reilly

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Table of Contents
Preface................................................................................................................................................ 5
Audience................................................................................................................................................5
How This Book is Organized.............................................................................................................. 6
Supporting Books.................................................................................................................................6
Code Examples..................................................................................................................................... 7
Early Release Status and Feedback................................................................................................... 7
Chapter 1. Introduction to Data Analysis with Spark......................................................8
What is Apache Spark?....................................................................................................................... 8
A Unified Stack.....................................................................................................................................8
Who Uses Spark, and For What?......................................................................................................11
A Brief History of Spark.................................................................................................................... 13
Spark Versions and Releases............................................................................................................ 13
Spark and Hadoop............................................................................................................................. 14
Chapter 2. Downloading and Getting Started...................................................................15
Downloading Spark............................................................................................................................15
Introduction to Spark’s Python and Scala Shells.......................................................................... 16
Introduction to Core Spark Concepts.............................................................................................20
Standalone Applications...................................................................................................................23
Conclusion.......................................................................................................................................... 25
Chapter 3. Programming with RDDs................................................................................... 26
RDD Basics......................................................................................................................................... 26
Creating RDDs................................................................................................................................... 28
RDD Operations................................................................................................................................ 28
Passing Functions to Spark.............................................................................................................. 32
Common Transformations and Actions......................................................................................... 36
Persistence (Caching)........................................................................................................................46
Conclusion.......................................................................................................................................... 48
Chapter 4. Working with Key-Value Pairs.........................................................................49
4
Motivation.......................................................................................................................................... 49
Creating Pair RDDs........................................................................................................................... 49
Transformations on Pair RDDs....................................................................................................... 50
Actions Available on Pair RDDs......................................................................................................60
Data Partitioning................................................................................................................................61
Conclusion.......................................................................................................................................... 70
Chapter 5. Loading and Saving Your Data.......................................................................... 71
Motivation........................................................................................................................................... 71
Choosing a Format............................................................................................................................. 71
Formats............................................................................................................................................... 72
File Systems........................................................................................................................................88
Compression.......................................................................................................................................89
Databases............................................................................................................................................ 91
Conclusion.......................................................................................................................................... 93
About the Authors.....................................................................................................................

本帖隐藏的内容

Learning Spark.pdf (1.19 MB, 需要: 80 个论坛币)

2015-3-2 23:09:10 上传

需要: 80 个论坛币  [购买]

注意：这个是pre-release版本

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享1 收藏8 回帖

关键词：Learning earning Reilly Holden Spark Media

1. /**
   
2. * Illustrates a simple aggregate in scala to compute the average of an RDD
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. import org.apache.spark.rdd.RDD
   
8. 
   
9. object BasicAvg {
   
10.   def main(args: Array[String]) {
    
11.     val master = args.length match {
    
12.       case x: Int if x > 0 => args(0)
    
13.       case _ => "local"
    
14.     }
    
15.     val sc = new SparkContext(master, "BasicAvg", System.getenv("SPARK_HOME"))
    
16.     val input = sc.parallelize(List(1,2,3,4))
    
17.     val result = computeAvg(input)
    
18.     val avg = result._1 / result._2.toFloat
    
19.     println(result)
    
20.   }
    
21.   def computeAvg(input: RDD[Int]) = {
    
22.     input.aggregate((0, 0))((x, y) => (x._1 + y, x._2 + 1),
    
23.       (x,y) => (x._1 + y._1, x._2 + y._2))
    
24.   }
    
25. }

复制代码

spark is trendy, a must-read book

1. /**
   
2. * Illustrates loading a simple text file
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. 
   
8. object BasicAvgFromFile {
   
9.     def main(args: Array[String]) {
   
10.       if (args.length < 2) {
    
11.         println("Usage: [sparkmaster] [inputfile]")
    
12.         exit(1)
    
13.       }
    
14.       val master = args(0)
    
15.       val inputFile = args(1)
    
16.       val sc = new SparkContext(master, "BasicAvg", System.getenv("SPARK_HOME"))
    
17.       val input = sc.textFile(inputFile)
    
18.       val result = input.map(_.toInt).aggregate((0, 0))(
    
19.         (acc, value) => (acc._1 + value, acc._2 + 1),
    
20.         (acc1, acc2) => (acc1._1 + acc2._1, acc1._2 + acc2._2))
    
21.       val avg = result._1 / result._2.toFloat
    
22.       println(result)
    
23.     }
    
24. }

复制代码

1. /**
   
2. * Illustrates loading a directory of files
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. import org.apache.spark.SparkContext._
   
8. 
   
9. object BasicAvgFromFiles {
   
10.     def main(args: Array[String]) {
    
11.       if (args.length < 3) {
    
12.         println("Usage: [sparkmaster] [inputdirectory] [outputdirectory]")
    
13.         exit(1)
    
14.       }
    
15.       val master = args(0)
    
16.       val inputFile = args(1)
    
17.       val outputFile = args(2)
    
18.       val sc = new SparkContext(master, "BasicAvgFromFiles", System.getenv("SPARK_HOME"))
    
19.       val input = sc.wholeTextFiles(inputFile)
    
20.       val result = input.mapValues{y =>
    
21.         val nums = y.split(" ").map(_.toDouble)
    
22.         nums.sum / nums.size.toDouble
    
23.       }
    
24.       result.saveAsTextFile(outputFile)
    
25.     }
    
26. }

复制代码

1. /**
   
2. * Illustrates mapPartitions in scala
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. 
   
8. object BasicAvgMapPartitions {
   
9.   case class AvgCount(var total: Int = 0, var num: Int = 0) {
   
10.     def merge(other: AvgCount): AvgCount = {
    
11.       total += other.total
    
12.       num += other.num
    
13.       this
    
14.     }
    
15.     def merge(input: Iterator[Int]): AvgCount = {
    
16.       input.foreach{elem =>
    
17.         total += elem
    
18.         num += 1
    
19.       }
    
20.       this
    
21.     }
    
22.     def avg(): Float = {
    
23.       total / num.toFloat;
    
24.     }
    
25.   }
    
26. 
    
27.   def main(args: Array[String]) {
    
28.     val master = args.length match {
    
29.       case x: Int if x > 0 => args(0)
    
30.       case _ => "local"
    
31.     }
    
32.     val sc = new SparkContext(master, "BasicAvgMapPartitions", System.getenv("SPARK_HOME"))
    
33.     val input = sc.parallelize(List(1, 2, 3, 4))
    
34.     val result = input.mapPartitions(partition =>
    
35.       // Here we only want to return a single element for each partition, but mapPartitions requires that we wrap our return in an Iterator
    
36.       Iterator(AvgCount(0, 0).merge(partition)))
    
37.       .reduce((x,y) => x.merge(y))
    
38.     println(result)
    
39.   }
    
40. }

复制代码

1. /**
   
2. * Illustrates a simple fold in scala
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. 
   
8. object BasicAvgWithKryo {
   
9.     def main(args: Array[String]) {
   
10.       val master = args.length match {
    
11.         case x: Int if x > 0 => args(0)
    
12.         case _ => "local"
    
13.       }
    
14.       val conf = new SparkConf().setMaster(master).setAppName("basicAvgWithKryo")
    
15.       conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    
16.       val sc = new SparkContext(conf)
    
17.       val input = sc.parallelize(List(1,2,3,4))
    
18.       val result = input.aggregate((0, 0))((x, y) => (x._1 + y, x._2 + 1),
    
19.         (x,y) => (x._1 + y._1, x._2 + y._2))
    
20.       val avg = result._1 / result._2.toFloat
    
21.       println(result)
    
22.     }
    
23. }

复制代码

1. /**
   
2. * Illustrates filtering and union to extract lines with "error" or "warning"
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. import org.apache.spark.SparkContext._
   
8. 
   
9. object BasicFilterUnionCombo {
   
10.     def main(args: Array[String]) {
    
11.       val conf = new SparkConf
    
12.       conf.setMaster(args(0))
    
13.       val sc = new SparkContext(conf)
    
14.       val inputRDD = sc.textFile(args(1))
    
15.       val errorsRDD = inputRDD.filter(_.contains("error"))
    
16.       val warningsRDD = inputRDD.filter(_.contains("warn"))
    
17.       val badLinesRDD = errorsRDD.union(warningsRDD)
    
18.       println(badLinesRDD.collect().mkString("\n"))
    
19.     }
    
20. }

复制代码

1. /**
   
2. * Illustrates intersection by key
   
3. */
   
4. package com.oreilly.learningsparkexamples.scala
   
5. 
   
6. import org.apache.spark._
   
7. import org.apache.spark.rdd.PairRDDFunctions
   
8. import org.apache.spark.rdd.RDD
   
9. import org.apache.spark.SparkContext._
   
10. 
    
11. import scala.reflect.ClassTag
    
12. 
    
13. object BasicIntersectByKey {
    
14. 
    
15.   def intersectByKey[K: ClassTag, V: ClassTag](rdd1: RDD[(K, V)], rdd2: RDD[(K, V)]): RDD[(K, V)] = {
    
16.     rdd1.cogroup(rdd2).flatMapValues{
    
17.       case (Nil, _) => None
    
18.       case (_, Nil) => None
    
19.       case (x, y) => x++y
    
20.     }
    
21.   }
    
22. 
    
23.   def main(args: Array[String]) {
    
24.     val master = args.length match {
    
25.       case x: Int if x > 0 => args(0)
    
26.       case _ => "local"
    
27.     }
    
28.     val sc = new SparkContext(master, "BasicIntersectByKey", System.getenv("SPARK_HOME"))
    
29.     val rdd1 = sc.parallelize(List((1, "panda"), (2, "happy")))
    
30.     val rdd2 = sc.parallelize(List((2, "pandas")))
    
31.     val iRdd = intersectByKey(rdd1, rdd2)
    
32.     val panda: List[(Int, String)] = iRdd.collect().toList
    
33.     panda.map(println(_))
    
34.     sc.stop()
    
35.   }
    
36. }

复制代码