【GitBook】Spark and Scala Recipes

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Yummy!

A scratchpad notebook for assorted collection of Spark and Scala examples.

本帖隐藏的内容

https://satishlalam.gitbooks.io/spark-and-scala-recipes/content/functions.html

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Recipes Recipe SCALA Spark Park collection notebook examples

1. Quickly create 'n to m' collection of numbers
   
2. 
   
3. 100 to 200 //res0: scala.collection.immutable.Range.Inclusive = Range(100, 101, 102, 103, 104, 105..200)
   
4. loop over 'n to m' range
   
5. 
   
6. for(i <- 100 to 200) {
   
7.   println(i)
   
8. }
   
9. // prints 100 to 200
   
10. create a list and append
    
11. 
    
12. val list = List(1,2,3,4,5)
    
13. 
    
14. // append to list
    
15. list  = list :+ 6  // error since list is val.
    
16. 
    
17. var list2 = List(1,2,3)
    
18. list2 = list2 :+ 4  // now list is a new List (1,2,3,4)
    
19. list2 = 0 +: list2 // now list is a new List(0,1,2,3,4)
    
20. Yield a Collection from foreach
    
21. 
    
22. scala> val kvp = for(i <- 1 to 5) yield {
    
23.      | (i, i*i)
    
24.      | }
    
25. kvp: scala.collection.immutable.IndexedSeq[(Int, Int)] = Vector((1,1), (2,4), (3,9), (4,16), (5,25))
    
26. Sequences and Sets
    
27. 
    
28. // sequence
    
29. //    Indexed - Array
    
30. //    Linear - List, Vector (LinkedLists of 32 element arrays)
    
31. // set - no dups
    
32. Convert a Seq to Map
    
33. 
    
34. scala> val kvpMap = kvp.map(k => k._1 -> k._2).toMap
    
35. kvpMap: scala.collection.immutable.Map[Int,Int] = Map(5 -> 25, 1 -> 1, 2 -> 4, 3 -> 9, 4 -> 16)
    
36. Flatten a list of tuples
    
37. 
    
38. val a = List(1,3,5)
    
39. val b = List(2,4,6)
    
40. val listOfTuples = a.zip(b)  // List( (1,2), (3,4), (5,6))
    
41. val flatList = listOfTuples.flatMap(x => List(x._1, x._2)) // List(1,2,3,4,5,6)
    
42. Seperate Even and Odd Numbers in a list
    
43. 
    
44. val list = (1 to 10)
    
45. val g = list.groupBy(x => x % 2)
    
46. // returns: Map(1 -> Vector(1, 3, 5, 7, 9), 0 -> Vector(2, 4, 6, 8, 10))
    
47. takeWhile

复制代码

1. Convert a list of lines to a single line
   
2. 
   
3. val lines : List[String] = List("this is line 1" , "this is line 2", "this is line 3")
   
4. // concat without delimiter
   
5. lines.mkString
   
6. // output: this is line 1this is line 2this is line 3  
   
7. 
   
8. // concat with delimiter
   
9. lines.foldLeft("")((a,b)=> (a + " " + b))
   
10. // output: this is line 1 this is line 2 this is line 3
    
11. Gotcha
    
12. 
    
13. // foldLeft vs foldRight
    
14. // foldLeft is more performant due to iterative implementation. FoldRight uses recursions
    
15. 
    
16. Internal code:
    
17. def foldLeft[B](z: B)(f: (B, A) => B): B = {
    
18.     var acc = z
    
19.     var these = this
    
20.     while (!these.isEmpty) {
    
21.       acc = f(acc, these.head)
    
22.       these = these.tail
    
23.     }
    
24.     acc
    
25.   }
    
26. 
    
27. override /*IterableLike*/
    
28.   def foldRight[B](z: B)(f: (A, B) => B): B =
    
29.     if (this.isEmpty) z
    
30.     else f(head, tail.foldRight(z)(f))
    
31. Count Words in a list of lines
    
32. 
    
33. import scala.io.Source
    
34. 
    
35. val lines = Source.fromFile("myfile").getLines()
    
36. val line = lines.foldLeft("")((a,b) => (a + " " + b)).trim // a single string of all lines
    
37. line.split(' ').length
    
38. Count Characters in a list of lines
    
39. 
    
40. import scala.io.Source
    
41. 
    
42. Source.fromFile("myfile")
    
43.         .getLines()
    
44.         .foldLeft("")((a,b) => (a + " " + b))
    
45.         .filter(_ != ' ')
    
46.         .length
    
47. Count occurence of a particular word or character in a list of lines
    
48. import scala.io.Source
    
49. 
    
50. val word = "scala"
    
51. val char = "s"
    
52. 
    
53. // fullText is of type String
    
54. val fullText = Source.fromFile("myfile")
    
55.         .getLines()
    
56.         .foldLeft("")((a,b) => (a + " " + b))
    
57. 
    
58. 
    
59. // count words
    
60. fullText.split(' ').count(_ == word)
    
61. 
    
62. // count chars
    
63. fullText.count(_ == char)

复制代码

1. Read all lines from a text file
   
2. 
   
3. import scala.io.Source
   
4. Source.fromFile("myfile")).foreach(println)
   
5. Get Current Working Directory
   
6. 
   
7. val path = new java.io.File(".").getCanonicalPath

复制代码

1. Load an Xml file
   
2. 
   
3. import scala.xml.{NodeSeq, XML}
   
4. 
   
5. // from path
   
6. var xmlNode = XML.loadFile("/path/to/my.xml")
   
7. 
   
8. // from resource (ex: under  src/main/resources/my.xml)
   
9. var xmlNode = XML.load(getClass.getResourceAsStream("/my.xml"))

复制代码

1. Measure time for a block
   
2. 
   
3. def time[R](block: => R): R = {  
   
4.     val t0 = System.nanoTime()
   
5.     val result = block    // call-by-name
   
6.     val t1 = System.nanoTime()
   
7.     println("Elapsed time: " + (t1 - t0) + "ns")
   
8.     result
   
9. }
   
10. 
    
11. val result = time { // code block
    
12. }

复制代码

谢谢分享

谢谢楼主分享！