[Case Study]Simple Text Classification using Scala

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. // scalastyle:off println
   
2. package org.apache.spark.examples.ml
   
3. 
   
4. import scala.beans.BeanInfo
   
5. 
   
6. import org.apache.spark.{SparkConf, SparkContext}
   
7. import org.apache.spark.ml.Pipeline
   
8. import org.apache.spark.ml.classification.LogisticRegression
   
9. import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
   
10. import org.apache.spark.mllib.linalg.Vector
    
11. import org.apache.spark.sql.{Row, SQLContext}
    
12. 
    
13. @BeanInfo
    
14. case class LabeledDocument(id: Long, text: String, label: Double)
    
15. 
    
16. @BeanInfo
    
17. case class Document(id: Long, text: String)
    
18. 
    
19. /**
    
20. * A simple text classification pipeline that recognizes "spark" from input text. This is to show
    
21. * how to create and configure an ML pipeline. Run with
    
22. * {{{
    
23. * bin/run-example ml.SimpleTextClassificationPipeline
    
24. * }}}
    
25. */
    
26. object SimpleTextClassificationPipeline {
    
27. 
    
28.   def main(args: Array[String]) {
    
29.     val conf = new SparkConf().setAppName("SimpleTextClassificationPipeline")
    
30.     val sc = new SparkContext(conf)
    
31.     val sqlContext = new SQLContext(sc)
    
32.     import sqlContext.implicits._
    
33. 
    
34.     // Prepare training documents, which are labeled.
    
35.     val training = sc.parallelize(Seq(
    
36.       LabeledDocument(0L, "a b c d e spark", 1.0),
    
37.       LabeledDocument(1L, "b d", 0.0),
    
38.       LabeledDocument(2L, "spark f g h", 1.0),
    
39.       LabeledDocument(3L, "hadoop mapreduce", 0.0)))
    
40. 
    
41.     // Configure an ML pipeline, which consists of three stages: tokenizer, hashingTF, and lr.
    
42.     val tokenizer = new Tokenizer()
    
43.       .setInputCol("text")
    
44.       .setOutputCol("words")
    
45.     val hashingTF = new HashingTF()
    
46.       .setNumFeatures(1000)
    
47.       .setInputCol(tokenizer.getOutputCol)
    
48.       .setOutputCol("features")
    
49.     val lr = new LogisticRegression()
    
50.       .setMaxIter(10)
    
51.       .setRegParam(0.001)
    
52.     val pipeline = new Pipeline()
    
53.       .setStages(Array(tokenizer, hashingTF, lr))
    
54. 
    
55.     // Fit the pipeline to training documents.
    
56.     val model = pipeline.fit(training.toDF())
    
57. 
    
58.     // Prepare test documents, which are unlabeled.
    
59.     val test = sc.parallelize(Seq(
    
60.       Document(4L, "spark i j k"),
    
61.       Document(5L, "l m n"),
    
62.       Document(6L, "spark hadoop spark"),
    
63.       Document(7L, "apache hadoop")))
    
64. 
    
65.     // Make predictions on test documents.
    
66.     model.transform(test.toDF())
    
67.       .select("id", "text", "probability", "prediction")
    
68.       .collect()
    
69.       .foreach { case Row(id: Long, text: String, prob: Vector, prediction: Double) =>
    
70.         println(s"($id, $text) --> prob=$prob, prediction=$prediction")
    
71.       }
    
72. 
    
73.     sc.stop()
    
74.   }
    
75. }
    
76. // scalastyle:on println

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Case study simple cation Using SCALA import