【Rindra Ramamonjison】Apache Spark Graph Processing

In-degree and out-degree of the Enron email network

1. For the Enron email network, we can confirm that there are roughly ten times more links than nodes:
   
2. 
   
3. scala> emailGraph.numEdges
   
4. 
   
5. scala> emailGraph.numVertices
   
6. 
   
7. scala> emailGraph.inDegrees.map(_._2).sum / emailGraph.numVertices
   
8. 
   
9. scala> emailGraph.outDegrees.map(_._2).sum / emailGraph.numVertices
   
10. 
    
11. def max(a: (VertexId, Int), b: (VertexId, Int)): (VertexId, Int) = {
    
12.   if (a._2 > b._2) a else b
    
13. }
    
14. 
    
15. 
    
16. scala> emailGraph.outDegrees.reduce(max)
    
17. 
    
18. 
    
19. scala> emailGraph.outDegrees.filter(_._2 <= 1).count

复制代码

Degrees in the bipartite food network

1. For the bipartite ingredient-compound graph, we can also explore which food has the largest number of compounds, or which compound is the most prevalent in our list of ingredients:
   
2. 
   
3. scala> foodNetwork.outDegrees.reduce(max)
   
4. res: (org.apache.spark.graphx.VertexId, Int) = (908,239)
   
5. 
   
6. scala> foodNetwork.vertices.filter(_._1 == 908).collect()
   
7. res: Array[(org.apache.spark.graphx.VertexId, FNNode)] = Array((908,Ingredient(black_tea,plant derivative)))
   
8. 
   
9. scala> foodNetwork.inDegrees.reduce(max)
   
10. res: (org.apache.spark.graphx.VertexId, Int) = (10292,299)
    
11. 
    
12. scala> foodNetwork.vertices.filter(_._1 == 10292).collect()
    
13. res: Array[(org.apache.spark.graphx.VertexId, FNNode)] = Array((10292,Compound(1-octanol,111-87-5)))

复制代码

Degree histogram of the social ego networks

1. Similarly, we can compute the degrees of the connections in the ego network. Let's look at the maximum and minimum degrees in the network:
   
2. 
   
3. scala> egoNetwork.degrees.reduce(max)
   
4. res91: (org.apache.spark.graphx.VertexId, Int) = (1643293729,1084)
   
5. 
   
6. scala> egoNetwork.degrees.reduce(min)
   
7. res92: (org.apache.spark.graphx.VertexId, Int) = (550756674,1)

复制代码

Visualizing the graph data

1. 1.Import the GraphStream classes with the following:
   
2. scala> import org.graphstream.graph.{Graph => GraphStream}
   
3. scala> import org.graphstream.graph.implementations._
   
4. 
   
5. 2.Load the social ego network data using Graph.fromEdges, as we did in the previous chapter. After that, we will create a SingleGraph object:
   
6. 
   
7. // Create a SingleGraph class for GraphStream visualization
   
8. val graph: SingleGraph = new SingleGraph("EgoSocial")
   
9. 
   
10. node {
    
11.     fill-color: #a1d99b;
    
12.     size: 20px;
    
13.     text-size: 12;
    
14.     text-alignment: at-right;
    
15.     text-padding: 2;
    
16.     text-background-color: #fff7bc;
    
17. }
    
18. edge {
    
19.     shape: cubic-curve;
    
20.     fill-color: #dd1c77;
    
21.     z-index: 0;
    
22.     text-background-mode: rounded-box;
    
23.     text-background-color: #fff7bc;
    
24.     text-alignment: above;
    
25.     text-padding: 2;
    
26. }
    
27. 
    
28. 3.Connect it to the SingleGraph object graph:
    
29. 
    
30. // Set up the visual attributes for graph visualization
    
31. graph.addAttribute("ui.stylesheet","url(file:.//style/stylesheet)")
    
32. graph.addAttribute("ui.quality")
    
33. graph.addAttribute("ui.antialias")
    
34. In the last two lines, we simply informed the rendering engine to favor quality instead of speed. Next, we have to reload the graph that we built in the previous chapter. To avoid repetitions, we omit the graph building part. After this, we now load VertexRDD and EdgeRDD of the social network into the GraphStream graph object, with the following code:
    
35. 
    
36. // Given the egoNetwork, load the graphX vertices into GraphStream
    
37. for ((id,_) <- egoNetwork.vertices.collect()) {
    
38. val node = graph.addNode(id.toString).asInstanceOf[SingleNode]
    
39. }
    
40. // Load the graphX edges into GraphStream edges
    
41. for (Edge(x,y,_) <- egoNetwork.edges.collect()) {
    
42. val edge = graph.addEdge(x.toString ++ y.toString, x.toString, y.toString,
    
43. true).
    
44.      asInstanceOf[AbstractEdge]
    
45. }

复制代码

Plotting the degree distribution

1. 1.Import some classes from JFreeChart and Breeze:
   
2. 
   
3. import org.jfree.chart.axis.ValueAxis
   
4. import breeze.linalg._
   
5. import breeze.plot._
   
6. 2.Define degreeHistogram function.
   
7. def degreeHistogram(net: Graph[Int, Int]): Array[(Int, Int)] =  
   
8.     net.degrees.map(t => (t._2,t._1)).
   
9.           groupByKey.map(t => (t._1,t._2.size)).
   
10.           sortBy(_._1).collect()
    
11. 3.Normalize the node degrees by the total number of nodes, so that the degree probabilities add up to one:
    
12. 
    
13. val nn = egoNetwork.numVertices
    
14. val egoDegreeDistribution = degreeHistogram(egoNetwork).map({case (d,n) => (d,n.toDouble/nn)})
    
15. 4.Display the degree distribution
    
16. val f = Figure()
    
17. val p1 = f.subplot(2,1,0)
    
18. val x = new DenseVector(egoDegreeDistribution map (_._1.toDouble))
    
19. val y = new DenseVector(egoDegreeDistribution map (_._2))
    
20. p1.xlabel = "Degrees"
    
21. p1.ylabel = "Distribution"
    
22. p1 += plot(x, y)
    
23. p1.title = "Degree distribution of social ego network"
    
24. val p2 = f.subplot(2,1,1)
    
25. val egoDegrees = egoNetwork.degrees.map(_._2).collect()
    
26. 
    
27. p1.xlabel = "Degrees"
    
28. p1.ylabel = "Histogram of node degrees"
    
29. p2 += hist(egoDegrees, 10)

复制代码

谢谢分享

谢谢分享

谢谢分享

谢谢楼主分享！