【Apache Spark】The Data Scientist's Guide to Apache Spark

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

The Data Scientist's Guide to Apache Spark

本帖隐藏的内容

data-scientists-guide-apache-spark-master.zip (11.24 MB)

2017-5-1 03:58:54 上传

This repo contains notebook exercises for a workshop teaching the best practices of using Spark for practicing data scientists in the context of a data scientist’s standard workflow. By leveraging Spark’s APIs for Python and R to present practical applications, the technology will be much more accessible by decreasing the barrier to entry.

Materials

For the workshop (and after) we will use a Gitter chatroom to keep the conversation going: https://gitter.im/Jay-Oh-eN/data-scientists-guide-apache-spark.

And/or please do not hesitate to reach out to me directly via email at jonathan@galvanize.com or over twitter @clearspandex

The presentation can be found on Slideshare here.

Prerequisites

Prior experience with Python and the scientific Python stack is beneficial. Also knowledge of data science models and applications is preferred. This will not be an introduction to Machine Learning or Data Science, but rather a course for people proficient in these methods on a small scale to understand how to apply that knowledge in a distributed setting with Spark.

Setup

• Local Installation
• Cluster Deployment
  

SparkR with a Notebook

• Install IRKernel
  

install.packages(c('rzmq','repr','IRkernel','IRdisplay'), repos = c('http://irkernel.github.io/', getOption('repos')))IRkernel::installspec()

• Set environment variables:
  

# Example: Set this to where Spark is installedSys.setenv(SPARK_HOME="/Users/[username]/spark")# This line loads SparkR from the installed directory.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))# if these two lines work, you are all setlibrary(SparkR)sc <- sparkR.init(master="local")Data

link = 'http://hopelessoptimism.com/static/data/airline-data'

The notebooks use a few datasets. For the DonorsChoose data, you can read the documentation here and download a zip (~0.5 gb) from: http://hopelessoptimism.com/static/data/donors_choose.zip

IPython Console Help

Q: How can I find out all the methods that are available on DataFrame?

• In the IPython console type sales.[TAB]

• Autocomplete will show you all the methods that are available.

• To find more information about a specific method, say .cov type help(sales.cov)

• This will display the API documentation for that method.

  
  

Spark Documentation

Q: How can I find out more about Spark's Python API, MLlib, GraphX, Spark Streaming, deploying Spark to EC2?

• Go to https://spark.apache.org/docs/latest

• Navigate using tabs to the following areas in particular.

• Programming Guide > Quick Start, Spark Programming Guide, Spark Streaming, DataFrames and SQL, MLlib, GraphX, SparkR.

• Deploying > Overview, Submitting Applications, Spark Standalone, YARN, Amazon EC2.

• More > Configuration, Monitoring, Tuning Guide.

  
  

ReferencesSetup

• Spark on Windows 7
  

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏1 回帖

关键词：Apache Spark Scientist apache Spark Guide

1. All Together Now
   
2. In [12]:
   
3. import nltk, string
   
4. 
   
5. def tokenize(text):
   
6.     tokens = []
   
7.    
   
8.     for word in nltk.word_tokenize(text):
   
9.         if word \
   
10.             not in nltk.corpus.stopwords.words('english') \
    
11.             and word not in string.punctuation \
    
12.             and word != '``':   
    
13.                 tokens.append(word)
    
14.    
    
15.     return tokens
    
16. In [13]:
    
17. sc.stop()
    
18. In [14]:
    
19. # run on all cores
    
20. sc = ps.SparkContext()
    
21. sc.master
    
22. Out[14]:
    
23. u'local[*]'
    
24. In [23]:
    
25. from collections import Counter
    
26. In [25]:
    
27. # parse input essay file
    
28. essay_rdd = sc.textFile('file:///Users/jonathandinu/spark-ds-applications/data/donors_choose/essay_parse.json')
    
29. row_rdd = essay_rdd.map(lambda x: json.loads(x))
    
30. 
    
31. # tokenize documents
    
32. tokenized_rdd = row_rdd.filter(lambda row: row['essay'] and row['essay'] != '') \
    
33.                        .map(lambda row: row['essay']) \
    
34.                        .map(lambda text: text.replace('\\n', '').replace('\r', '')) \
    
35.                        .map(lambda text: tokenize(text))
    
36. 
    
37. # compute term and document frequencies
    
38. term_frequency = tokenized_rdd.map(lambda terms: Counter(terms))
    
39. 
    
40. doc_frequency = term_frequency.flatMap(lambda counts: counts.keys()) \
    
41.                              .map(lambda keys: (keys, 1)) \
    
42.                              .reduceByKey(lambda a, b: a + b)
    
43. In [28]:
    
44. num_features = 10000
    
45. 
    
46. top_terms = doc_frequency.top(num_features, key=lambda a: a[1])
    
47. In [30]:
    
48. import math
    
49. 
    
50. total_docs = essay_rdd.count()
    
51. idf = map(lambda tup: (tup[0], math.log(float(total_docs) / (1 + tup[1]))), top_terms)
    
52. In [31]:
    
53. broadcast_idf = sc.broadcast(idf)
    
54. 
    
55. def vectorize(tokens):
    
56.     word_counts = Counter(tokens)
    
57.     doc_length = sum(word_counts.values())
    
58.    
    
59.     vector = [word_counts.get(word[0], 0) * word[1] / float(doc_length) for word in broadcast_idf.value]
    
60.     return np.array(vector)
    
61. In [32]:
    
62. bag_of_words = tokenized_rdd.filter(lambda x: len(x) > 0).map(vectorize)
    
63. In [33]:
    
64. bag_of_words.persist()
    
65. Out[33]:
    
66. PythonRDD[30] at RDD at PythonRDD.scala:43
    
67. In [34]:
    
68. top_n = 10
    
69. summary = bag_of_words.map(lambda x: map(lambda idx: broadcast_idf.value[idx][0], np.argsort(x)[::-1][:top_n]))
    
70. In [35]:
    
71. summary.take(15)

复制代码

thanks.........

不错的好书