【Apache Spark】Analyzing Flight Data: A Gentle Introduction to GraphX in Spark

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

本帖隐藏的内容

Analyzing Flight Data_ A Gentle Introduction to GraphX in Spark.pdf (443.19 KB)

2017-4-30 09:23:13 上传

Basics of GraphXGraphX, as you might have guessed, is built upon this basic paradigm of graph theory. It's awesome. RequirementsNow this tutorial is only going to cover Scala because at this time, Apache Spark only has a Scala API for GraphX.

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：introduction troduction Analyzing Gentle flight

1. The Setup & Data
   
2. Let's go ahead and get set up with a dataset that I've introduced before in my Databricks. Scalaimport org.apache.spark.graphx._
   
3. import org.apache.spark.rdd.RDD
   
4. import scala.util.MurmurHash
   
5. val df_1 = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("../Downloads/2008.csv")

复制代码

1. val flightsFromTo = df_1.select($"Origin",$"Dest")
   
2. val airportCodes = df_1.select($"Origin", $"Dest").flatMap(x => Iterable(x(0).toString, x(1).toString))

复制代码

1. val airportVertices: RDD[(VertexId, String)] = airportCodes.distinct().map(x => (MurmurHash.stringHash(x), x))
   
2. val defaultAirport = ("Missing")

复制代码

1. val flightEdges = flightsFromTo.map(x =>
   
2.     ((MurmurHash.stringHash(x(0).toString),MurmurHash.stringHash(x(1).toString)), 1)).reduceByKey(_+_).map(x => Edge(x._1._1, x._1._2,x._2))

复制代码

1. val graph = Graph(airportVertices, flightEdges, defaultAirport)
   
2. graph.persist() // we're going to be using it a lot

复制代码

1. How many unique flights from airport A to airport B are there?
   
2. Scalagraph.numVertices // 305
   
3. graph.numEdges // 5366

复制代码

1. What are the top 10 flights from airport to airport?
   
2. Scalagraph.triplets.sortBy(_.attr, ascending=false).map(triplet =>
   
3.     "There were " + triplet.attr.toString + " flights from " + triplet.srcAttr + " to " + triplet.dstAttr + ".").take(10)

复制代码

1. What are the lowest 10 flights from airport to airport?
   
2. Scalagraph.triplets.sortBy(_.attr).map(triplet =>
   
3.     "There were " + triplet.attr.toString + " flights from " + triplet.srcAttr + " to " + triplet.dstAttr + ".").take(10)

复制代码

1. val ranks = graph.pageRank(0.0001).vertices

复制代码

1. val ranksAndAirports = ranks.join(airportVertices).sortBy(_._2._1, ascending=false).map(_._2._2)
   
2. ranksAndAirports.take(10)

复制代码

谢谢楼主分享！

不错的资料

非常感谢搂住分享！