【Github】Scala and Spark for Big Data Analytics

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Scala and Spark for Big Data Analytics

This is the code repository for Scala and Spark for Big Data Analytics, published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.

About the Book

This book is divided into three parts. In the first part, it will introduce you to Scala programming, helping you understand its fundamentals and be able to program with Spark. It will then move on to introducing you to Spark and the design choices beneath it and show you how to perform data analysis with it. Finally, to shake things up, the book moves on to Advanced Spark and teaches you advanced topics, such as monitoring, configuration, debugging, testing, and finally deployment.

Instructions and Navigation

All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.

The code will look like the following:

package com.chapter11.SparkMachineLearningimport org.apache.spark.mllib.feature.StandardScalerModelimport org.apache.spark.mllib.linalg.{ Vector, Vectors }import org.apache.spark.sql.{ DataFrame }import org.apache.spark.sql.SparkSession

To follow this book, you need basic to medium-level knowledge of the Java programming language. A basic knowledge of concurrency concepts is welcome too.

Related Products

• Frank Kane's Taming Big Data with Apache Spark and Python

• Neural Network Programming with Java - Second Edition

• Java 9 with JShell

  
  

Suggestions and Feedback

Click here if you have any feedback or suggestions.

https://github.com/PacktPublishing/Scala-and-Spark-for-Big-Data-Analytics

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Analytics Analytic Big data GitHub Spark

1. package com.chapter3.ScalaFP
   
2. import scala.collection._
   
3. import scala.collection.mutable.Buffer
   
4. import scala.collection.mutable.HashMap
   
5. 
   
6. object CollectionExample {
   
7.   def main(args: Array[String]) {
   
8.     val x = 10
   
9.     val y = 15
   
10.     val z = 19
    
11.    
    
12.     Traversable(1, 2, 3)
    
13.     Iterable("x", "y", "z")
    
14.     Map("x" -> 10, "y" -> 13, "z" -> 17)
    
15.     Set("Red", "Green", "Blue")
    
16.     SortedSet("Hello,", "world!")
    
17.     Buffer(x, y, z)
    
18.     IndexedSeq(0.0, 1.0, 2.0)
    
19.     LinearSeq(x, y, z)
    
20.     List(2, 6, 10)
    
21.     HashMap("x" -> 20, "y" -> 19, "z" -> 16)
    
22.    
    
23.     val list = List(1, 2, 3) map (_ + 1)
    
24.     println(list)
    
25.    
    
26.     val set = Set(1, 2, 3) map (_ * 2)
    
27.     println(set)
    
28.    
    
29.     val list2 = List(x, y, z).map(x => x * 3)
    
30.     println(list2)
    
31.   }
    
32. }

复制代码

1. package com.chapter3.ScalaFP
   
2. 
   
3. object ListScala {
   
4.     def main(args: Array[String]) {
   
5.       val eventList = List(2, 4, 6, 8, 10) // A simple list      
   
6.       val mappedList = eventList.map(x => x*2) // Mapped each value by multiplying them by 2
   
7.       println("Original list: "+ eventList)
   
8.       println("Mapped list: "+ mappedList)
   
9.       
   
10.       //Use map to return a list from function      
    
11.       def func(x: Int) = if (x > 4) Some(x) else None
    
12.       val newList = eventList.map(x=> func(x))
    
13.       println("New list: " + newList)
    
14.     }  
    
15. }

复制代码

1. package com.chapter3.ScalaFP
   
2. object MonadiacExample {
   
3.   def main(args: Array[String]) {
   
4.     //Monadiac example 1
   
5.     for (x <- 10 until (0, -2))
   
6.        yield x
   
7.     //Monadiac example 2
   
8.     for (x <- 1 to 10 if x % 2 == 0)
   
9.       yield x
   
10.     // Monodiac example 3
    
11.     for (x <- 1 to 10; y <- 1 until x)
    
12.       yield (x, y)
    
13.     (1 to 10).flatMap(i => (1 until i).map(j => (i, j)))
    
14.   }
    
15. }

复制代码

1. package com.chapter3.ScalaFP
   
2. import java.io.IOException
   
3. import java.io.FileReader
   
4. import java.io.FileNotFoundException
   
5. object TryCatch {
   
6.   def main(args: Array[String]) {
   
7.     try {
   
8.       val f = new FileReader("data/data.txt")
   
9.     } catch {
   
10.       case ex: FileNotFoundException => println("File not found exception")
    
11.       case ex: IOException => println("IO Exception")
    
12.     } finally {
    
13.       println("Finally block always executes");
    
14.     }
    
15.   }
    
16. }

复制代码

1. package com.chapter3.ScalaFP
   
2. object filterExample {
   
3.     def main(args: Array[String]) {
   
4.       val range = List.range(1, 10)
   
5.       println(range)      
   
6.       // Filter out only odd values
   
7.       val odds = range.filter( x=> x % 2 != 0)
   
8.       println("Odd values: " + odds)      
   
9.       // Filter out only even values
   
10.       val even = range.filter( x=> x % 2 == 0)
    
11.       println("Odd values: " + even)
    
12.     }
    
13. }

复制代码

1. package com.chapter3.ScalaFP
   
2. object flatMapExample {
   
3.     def main(args: Array[String]) {
   
4.       val eventList = List(2, 4, 6, 8, 10) // A simple list      
   
5.       println("Original list: "+ eventList)      
   
6.       //Use map   
   
7.       def around(x: Int) = List(x-1, x, x+1)
   
8.       val newList1 = eventList.map(x=> around(x))
   
9.       println("New list from map : " + newList1)      
   
10.       //Use flatMap     
    
11.       val newList2 = eventList.flatMap(x=> around(x))
    
12.       println("New list from flatMap: " + newList2)
    
13.     }  
    
14. }

复制代码

1. package com.chapter3.ScalaFP
   
2. object mapExample {
   
3.   implicit class MapReduceTraversable[T, N](val traversable: Traversable[(T, N)]) {
   
4.     def reduceByKey(f: (N, N) => N) = traversable.par.groupBy(_._1).mapValues(_.map(_._2)).mapValues(_.reduce(f))
   
5.   }
   
6.   def main(args: Array[String]) {
   
7.     val eventList = List(2, 4, 6, 8, 10) // A simple list      
   
8.     println("Original list: " + eventList)
   
9.     //Use map   
   
10.     val newList1 = eventList.map(x => x * 2)
    
11.     println(newList1)
    
12.     def func(x: Int) = if (x > 4) Some(x) else None
    
13.     val newList2 = eventList.map(x => func(x))
    
14.     println(newList2)
    
15. 
    
16.     val myList = List(1, 1, 1, 1, 1, 1, 1)
    
17.     val reduce = myList.reduce { (x, y) => println(s"$x+$y=${x + y}"); x + y }
    
18.     println()
    
19.     val parReduce = myList.par.reduce { (x, y) => println(s"$x+$y=${x + y}"); x + y }
    
20.     println()
    
21. 
    
22.     val fruits = List("apple", "apple", "orange", "apple", "mango", "orange", "apple", "apple", "apple", "apple")
    
23.     val reducebyKeyValue = fruits.map(f => (f, 1)).reduceByKey(_ + _)
    
24.     println(reducebyKeyValue)
    
25.   }
    
26. }

复制代码

谢谢楼主分享！