【GitHub】Spark Scala Learning Note

1. Data From S3
   
2. 
   
3. AWS Configuration
   
4. 
   
5. Select IAM Service
   
6. Create User and Copy the Access Key ID and Secret Access Key
   
7. Create Group and attach Policy(AmazonS3FullAccess).
   
8. Create S3 Bucket
   
9. Spark application in Scala
   
10. 
    
11.     val sc = new SparkContext(new SparkConf().setAppName("Recommender").setMaster("local[2]"))
    
12.     val hadoopConf = sc.hadoopConfiguration
    
13.     hadoopConf.set("fs.s3n.awsAccessKeyId", "***")
    
14.     hadoopConf.set("fs.s3n.awsSecretAccessKey", "***")
    
15.     val base = "s3n://lin-spark-sample/ch3/"
    
16.     val rawUserArtistData = sc.textFile(base + "user_artist_data.txt")

复制代码

谢谢分享

1. Reading CSV
   
2. 
   
3. Import Dependencies
   
4. 
   
5. import org.apache.spark.SparkConf
   
6. import org.apache.spark.SparkContext
   
7. import org.apache.spark.sql.SQLContext
   
8. import com.databricks.spark.csv._
   
9. Spark App in Scala
   
10. 
    
11. val conf = new SparkConf().setAppName("csvDataFrame").setMaster("local[2]")   
    
12. val sc = new SparkContext(conf)
    
13. val sqlContext = new SQLContext(sc)   
    
14. val students=sqlContext.csvFile(filePath="StudentData.csv", useHeader=true, delimiter='|')
    
15. students.printSchema
    
16. students.show(3)

复制代码

1. Using Parquet
   
2. 
   
3. Import Dependencies
   
4. 
   
5. import org.apache.spark.SparkConf
   
6. import org.apache.spark.SparkContext
   
7. import org.apache.spark.sql.SQLContext
   
8. import com.databricks.spark.csv._
   
9. Creating DataFrame from CSV
   
10. 
    
11. val conf = new SparkConf().setAppName("DataFrameToParquet").setMaster("local[2]")
    
12. val sc = new SparkContext(conf)
    
13. val sqlContext = new SQLContext(sc)
    
14. 
    
15. // Reading the CSV file StudentData.csv
    
16. val studentsDF = sqlContext.csvFile(filePath = "StudentData.csv", useHeader = true, delimiter = '|')

复制代码

1. Creating LabeledPoint
   
2. 
   
3. Firstly, take a glance to the sample data.
   
4. 7.4;0.7;0;1.9;0.076;11;34;0.9978;3.51;0.56;9.4;5
   
5. 7.8;0.88;0;2.6;0.098;25;67;0.9968;3.2;0.68;9.8;5
   
6. 7.8;0.76;0.04;2.3;0.092;15;54;0.997;3.26;0.65;9.8;5
   
7. 11.2;0.28;0.56;1.9;0.075;17;60;0.998;3.16;0.58;9.8;6
   
8. 7.4;0.7;0;1.9;0.076;11;34;0.9978;3.51;0.56;9.4;5
   
9. 7.4;0.66;0;1.8;0.075;13;40;0.9978;3.51;0.56;9.4;5
   
10. 7.9;0.6;0.06;1.6;0.069;15;59;0.9964;3.3;0.46;9.4;5
    
11. Each of our training samples has the following format: the last field is the quality of the wine, a rating from 1 to 10, and the first 11 fields are the properties of the wine. So, from our perspective, the quality of the wine is the y variable (the output) and the rest of them are x variables (input).
    
12. Loading Data into Spark RDD
    
13. 
    
14. Import
    
15. import org.apache.spark.SparkConf
    
16. import org.apache.spark.SparkContext
    
17. import org.apache.spark.sql.SQLContext
    
18. Code
    
19. 
    
20. val conf = new SparkConf().setAppName("linearRegressionWine").setMaster("local[2]")
    
21. val sc = new SparkContext(conf)
    
22. val rdd = sc.textFile("winequality-red.csv").map(line => line.split(";"))
    
23. Converting each instance into LabeledPoint
    
24. 
    
25. A LabeledPoint is a simple wrapper around the input features (our x variables) and our pre-predicted value (y variable) for these x input values:
    
26. Import
    
27. import org.apache.spark.mllib.regression.LabeledPoint
    
28. import org.apache.spark.mllib.linalg.Vectors
    
29. Code
    
30. 
    
31. val dataPoints = rdd.map(row =>
    
32.     new LabeledPoint(
    
33.           row.last.toDouble,
    
34.           Vectors.dense(row.take(row.length - 1).map(str => str.toDouble))
    
35.     )
    
36.   ).cache()

复制代码

1. Preparing the Training and Test Data
   
2. 
   
3. val dataSplit = labeledDataPoints.randomSplit(Array(0.8, 0.2))
   
4. val trainingSet = dataSplit(0)
   
5. val testSet = dataSplit(1)

复制代码

1. Summay Statistics
   
2. 
   
3. val featureVector = rdd.map(row => Vectors.dense(row.take(row.length-1).map(str => str.toDouble)))
   
4. val stats = Statistics.colStats(featureVector)
   
5. print(s"Max : ${stats.max}, Min : ${stats.min}, and Mean : ${stats.mean} and Variance : ${stats.variance}")

复制代码