Spark MLContext Programming Guide

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

本帖隐藏的内容

http://apache.github.io/incubator-systemml/spark-mlcontext-programming-guide#spark-mlcontext-programming-guide

1. Create MLContext
   
2. 
   
3. All primary classes that a user interacts with are located in the org.apache.sysml.api.mlcontext package. For convenience, we can additionally add a static import of ScriptFactory to shorten the syntax for creating Script objects. An MLContext object can be created by passing its constructor a reference to the SparkSession (spark) or SparkContext (sc). If successful, you should see a “Welcome to Apache SystemML!” message.

复制代码

1. import org.apache.sysml.api.mlcontext._
   
2. import org.apache.sysml.api.mlcontext.ScriptFactory._
   
3. val ml = new MLContext(spark)

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Programming Context Program Contex Spark

1. DataFrame Example
   
2. 
   
3. For demonstration purposes, we’ll use Spark to create a DataFrame called df of random doubles from 0 to 1 consisting of 10,000 rows and 100 columns.
   
4. 
   
5. import org.apache.spark.sql._
   
6. import org.apache.spark.sql.types.{StructType,StructField,DoubleType}
   
7. import scala.util.Random
   
8. val numRows = 10000
   
9. val numCols = 100
   
10. val data = sc.parallelize(0 to numRows-1).map { _ => Row.fromSeq(Seq.fill(numCols)(Random.nextDouble)) }
    
11. val schema = StructType((0 to numCols-1).map { i => StructField("C" + i, DoubleType, true) } )
    
12. val df = spark.createDataFrame(data, schema)

复制代码

1. RDD Example
   
2. 
   
3. Let’s take a look at an example of input matrices as RDDs in CSV format. We’ll create two 2x2 matrices and input these into a DML script. This script will sum each matrix and create a message based on which sum is greater. We will output the sums and the message.
   
4. 
   
5. For fun, we’ll write the script String to a file and then use ScriptFactory’s dmlFromFile method to create the script object based on the file. We’ll also specify the inputs using a Map, although we could have also chained together two in methods to specify the same inputs.
   
6. 
   
7. val rdd1 = sc.parallelize(Array("1.0,2.0", "3.0,4.0"))
   
8. val rdd2 = sc.parallelize(Array("5.0,6.0", "7.0,8.0"))
   
9. val sums = """
   
10. s1 = sum(m1);
    
11. s2 = sum(m2);
    
12. if (s1 > s2) {
    
13.   message = "s1 is greater"
    
14. } else if (s2 > s1) {
    
15.   message = "s2 is greater"
    
16. } else {
    
17.   message = "s1 and s2 are equal"
    
18. }
    
19. """
    
20. scala.tools.nsc.io.File("sums.dml").writeAll(sums)
    
21. val sumScript = dmlFromFile("sums.dml").in(Map("m1"-> rdd1, "m2"-> rdd2)).out("s1", "s2", "message")
    
22. val sumResults = ml.execute(sumScript)
    
23. val s1 = sumResults.getDouble("s1")
    
24. val s2 = sumResults.getDouble("s2")
    
25. val message = sumResults.getString("message")

复制代码

1. Matrix Output
   
2. 
   
3. Let’s look at an example of reading a matrix out of SystemML. We’ll create a DML script in which we create a 2x2 matrix m. We’ll set the variable n to be the sum of the cells in the matrix.
   
4. 
   
5. We create a script object using String s, and we set m and n as the outputs. We execute the script, and in the results we see we have Matrix m and Double n. The n output variable has a value of 110.0.
   
6. 
   
7. We get Matrix m and Double n as a Tuple of values x and y. We then convert Matrix m to an RDD of IJV values, an RDD of CSV values, a DataFrame, and a two-dimensional Double Array, and we display the values in each of these data structures.
   
8. 
   
9. val s =
   
10. """
    
11. m = matrix("11 22 33 44", rows=2, cols=2)
    
12. n = sum(m)
    
13. """
    
14. val scr = dml(s).out("m", "n");
    
15. val res = ml.execute(scr)
    
16. val (x, y) = res.getTuple[Matrix, Double]("m", "n")
    
17. x.toRDDStringIJV.collect.foreach(println)
    
18. x.toRDDStringCSV.collect.foreach(println)
    
19. x.toDF.collect.foreach(println)
    
20. x.to2DDoubleArray

复制代码