【Github】Frank Kane's Taming Big Data with Apache Spark and Python

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Frank Kane's Taming Big Data with Apache Spark and Python

本帖隐藏的内容

https://github.com/PacktPublishing/Frank-Kanes-Taming-Big-Data-with-Apache-Spark-and-Python

This is the code repository for Frank Kane's Taming Big Data with Apache Spark and Python, published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.

About the Book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python.

Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses.

Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease. ##Instructions and Navigation All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.

The code will look like the following:

from pyspark import SparkConf, SparkContext import collections conf = SparkConf().setMaster("local").setAppName("RatingsHistogram") sc = SparkContext(conf = conf)lines = sc.textFile("file:///SparkCourse/ml-100k/u.data") ratings = lines.map(lambda x: x.split()[2]) result = ratings.countByValue() sortedResults = collections.OrderedDict(sorted(result.items())) for key, value in sortedResults.items():     print("%s %i" % (key, value))

For this book you’ll need a Python development environment (Python 3.5 or newer), a Canopy installer, Java Development Kit, and of course Spark itself (Spark 2.0 and beyond).

We'll show you how to install this software in first chapter of the book.

This book is based on the Windows operating system, so installations are provided according to it. If you have Mac or Linux, you can follow this URL http://media.sundog-soft.com/spark-python-install.pdf, which contains written instructions on getting everything set up on Mac OS and on Linux.

Related Products

• Taming Big Data with Spark Streaming and Scala – Hands On! [Video]

• Taming Big Data with Apache Spark and Python - Hands On! [Video]

• Taming Big Data with MapReduce and Hadoop - Hands On! [Video]

  
  

Suggestions and Feedback

Click here if you have any feedback or suggestions.

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Apache Spark Big data python apache GitHub

1. from pyspark.sql import SparkSession
   
2. from pyspark.sql import Row
   
3. 
   
4. import collections
   
5. 
   
6. # Create a SparkSession (Note, the config section is only for Windows!)
   
7. spark = SparkSession.builder.config("spark.sql.warehouse.dir", "file:///C:/temp").appName("SparkSQL").getOrCreate()
   
8. 
   
9. def mapper(line):
   
10.     fields = line.split(',')
    
11.     return Row(ID=int(fields[0]), name=str(fields[1].encode("utf-8")), age=int(fields[2]), numFriends=int(fields[3]))
    
12. 
    
13. lines = spark.sparkContext.textFile("fakefriends.csv")
    
14. people = lines.map(mapper)
    
15. 
    
16. # Infer the schema, and register the DataFrame as a table.
    
17. schemaPeople = spark.createDataFrame(people).cache()
    
18. schemaPeople.createOrReplaceTempView("people")
    
19. 
    
20. # SQL can be run over DataFrames that have been registered as a table.
    
21. teenagers = spark.sql("SELECT * FROM people WHERE age >= 13 AND age <= 19")
    
22. 
    
23. # The results of SQL queries are RDDs and support all the normal RDD operations.
    
24. for teen in teenagers.collect():
    
25.   print(teen)
    
26. 
    
27. # We can also use functions instead of SQL queries:
    
28. schemaPeople.groupBy("age").count().orderBy("age").show()
    
29. 
    
30. spark.stop()

复制代码

1. from pyspark import SparkConf, SparkContext
   
2. 
   
3. conf = SparkConf().setMaster("local").setAppName("PopularMovies")
   
4. sc = SparkContext(conf = conf)
   
5. 
   
6. lines = sc.textFile("file:///SparkCourse/ml-100k/u.data")
   
7. movies = lines.map(lambda x: (int(x.split()[1]), 1))
   
8. movieCounts = movies.reduceByKey(lambda x, y: x + y)
   
9. 
   
10. flipped = movieCounts.map( lambda (x, y) : (y, x) )
    
11. sortedMovies = flipped.sortByKey()
    
12. 
    
13. results = sortedMovies.collect()
    
14. 
    
15. for result in results:
    
16.     print(result)

复制代码

Lisrelchen 发表于 2017-8-13 05:15
Frank Kane's Taming Big Data with Apache Spark and Python
**** 本内容被作者隐藏 ****
This is the c ...

谢谢分享

谢谢楼主分享！

谢谢楼主分享！

为什么都要回复才能下载呢