【Reading Notes】Learning Spark: Lightning-Fast Big Data Analysis

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

本帖隐藏的内容

https://github.com/gaoxuesong/learning-spark-lightning-fast-big-data-analysis

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Learning Analysis Big data earning reading

1. Learning Spark: Lightning-Fast Big Data Analysis reading notes
   
2. 
   
3. Reading notes for the book of Learning Spark: Lightning-Fast Big Data Analysis is only for spark developer educational purposes. Reading notes in Github: https://github.com/gaoxuesong/learning-spark-lightning-fast-big-data-analysis
   
4. 
   
5. 《Learning Spark: Lightning-Fast Big Data Analysis》的中文读书笔记纯属个人对于Spark的兴趣，仅供学习。 读书笔记分享于Github: https://github.com/gaoxuesong/learning-spark-lightning-fast-big-data-analysis
   
6. 
   
7. About the Author
   
8. 
   
9. Holden Karau is a software development engineer at Databricks and is active in open source. She is the author of an earlier Spark book. Prior to Databricks she worked on a variety of search and classification problems at Google, Foursquare, and Amazon. She graduated from the University of Waterloo with a Bachelors of Mathematics in Computer Science. Outside of software she enjoys playing with fire, welding, and hula hooping.
   
10. 
    
11. Most recently, Andy Konwinski co-founded Databricks. Before that he was a PhD student and then postdoc in the AMPLab at UC Berkeley, focused on large scale distributed computing and cluster scheduling. He co-created and is a committer on the Apache Mesos project. He also worked with systems engineers and researchers at Google on the design of Omega, their next generation cluster scheduling system. More recently, he developed and led the AMP Camp Big Data Bootcamps and first Spark Summit, and has been contributing to the Spark project.
    
12. 
    
13. Patrick Wendell is an engineer at Databricks as well as a Spark Committer and PMC member. In the Spark project, Patrick has acted as release manager for several Spark releases, including Spark 1.0. Patrick also maintains several subsystems of Spark's core engine. Before helping start Databricks, Patrick obtained an M.S. in Computer Science at UC Berkeley. His research focused on low latency scheduling for large scale analytics workloads. He holds a B.S.E in Computer Science from Princeton University
    
14. 
    
15. Matei Zaharia is the creator of Apache Spark and CTO at Databricks. He holds a PhD from UC Berkeley, where he started Spark as a research project. He now serves as its Vice President at Apache. Apart from Spark, he has made research and open source contributions to other projects in the cluster computing area, including Apache Hadoop (where he is a committer) and Apache Mesos (which he also helped start at Berkeley).

复制代码

1. """
   
2. >>> from pyspark.context import SparkContext
   
3. >>> sc = SparkContext('local', 'test')
   
4. >>> b = sc.parallelize([1, 2, 3, 4])
   
5. >>> avg(b)
   
6. 2.5
   
7. """
   
8. 
   
9. import sys
   
10. 
    
11. from pyspark import SparkContext
    
12. 
    
13. 
    
14. def partitionCtr(nums):
    
15.     """Compute sumCounter for partition"""
    
16.     sumCount = [0, 0]
    
17.     for num in nums:
    
18.         sumCount[0] += num
    
19.         sumCount[1] += 1
    
20.     return [sumCount]
    
21. 
    
22. 
    
23. def combineCtrs(c1, c2):
    
24.     return (c1[0] + c2[0], c1[1] + c2[1])
    
25. 
    
26. 
    
27. def basicAvg(nums):
    
28.     """Compute the avg"""
    
29.     sumCount = nums.mapPartitions(partitionCtr).reduce(combineCtrs)
    
30.     return sumCount[0] / float(sumCount[1])
    
31. 
    
32. if __name__ == "__main__":
    
33.     cluster = "local"
    
34.     if len(sys.argv) == 2:
    
35.         cluster = sys.argv[1]
    
36.     sc = SparkContext(cluster, "Sum")
    
37.     nums = sc.parallelize([1, 2, 3, 4])
    
38.     avg = basicAvg(nums)
    
39.     print avg

复制代码

1. """
   
2. >>> from pyspark.context import SparkContext
   
3. >>> sc = SparkContext('local', 'test')
   
4. >>> b = sc.parallelize([1, 2, 3, 4])
   
5. >>> basicAvg(b)
   
6. 2.5
   
7. """
   
8. 
   
9. import sys
   
10. 
    
11. from pyspark import SparkContext
    
12. 
    
13. 
    
14. def basicAvg(nums):
    
15.     """Compute the avg"""
    
16.     sumCount = nums.map(lambda x: (x, 1)).fold(
    
17.         (0, 0), (lambda x, y: (x[0] + y[0], x[1] + y[1])))
    
18.     return sumCount[0] / float(sumCount[1])
    
19. 
    
20. if __name__ == "__main__":
    
21.     master = "local"
    
22.     if len(sys.argv) == 2:
    
23.         master = sys.argv[1]
    
24.     sc = SparkContext(master, "Sum")
    
25.     nums = sc.parallelize([1, 2, 3, 4])
    
26.     avg = basicAvg(nums)
    
27.     print avg

复制代码

1. """
   
2. >>> from pyspark.context import SparkContext
   
3. >>> sc = SparkContext('local', 'test')
   
4. >>> b = sc.parallelize([1, 2, 3, 4])
   
5. >>> sorted(basicSquareNoOnes(b).collect())
   
6. [4, 9, 16]
   
7. """
   
8. 
   
9. import sys
   
10. 
    
11. from pyspark import SparkContext
    
12. 
    
13. 
    
14. def basicSquareNoOnes(nums):
    
15.     """Square the numbers"""
    
16.     return nums.map(lambda x: x * x).filter(lambda x: x != 1)
    
17. 
    
18. if __name__ == "__main__":
    
19.     master = "local"
    
20.     if len(sys.argv) == 2:
    
21.         master = sys.argv[1]
    
22.     sc = SparkContext(master, "BasicFilterMap")
    
23.     nums = sc.parallelize([1, 2, 3, 4])
    
24.     output = sorted(basicSquareNoOnes(nums).collect())
    
25.     for num in output:
    
26.         print "%i " % (num)

复制代码

1. """
   
2. >>> from pyspark.context import SparkContext
   
3. >>> sc = SparkContext('local', 'test')
   
4. >>> b = sc.parallelize([1, 2, 3, 4])
   
5. >>> sorted(basicSquare(b).collect())
   
6. [1, 4, 9, 16]
   
7. """
   
8. 
   
9. import sys
   
10. 
    
11. from pyspark import SparkContext
    
12. 
    
13. 
    
14. def basicSquare(nums):
    
15.     """Square the numbers"""
    
16.     return nums.map(lambda x: x * x)
    
17. 
    
18. if __name__ == "__main__":
    
19.     master = "local"
    
20.     if len(sys.argv) == 2:
    
21.         master = sys.argv[1]
    
22.     sc = SparkContext(master, "BasicMap")
    
23.     nums = sc.parallelize([1, 2, 3, 4])
    
24.     output = sorted(basicSquare(nums).collect())
    
25.     for num in output:
    
26.         print "%i " % (num)

复制代码

谢谢分享！

Got it, tks.

这书怎么样?