[Case Study]Term Frequency–Inverse Document Frequency using Scala

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. // scalastyle:off println
   
2. package org.apache.spark.examples.ml
   
3. 
   
4. // $example on$
   
5. import org.apache.spark.ml.feature.{HashingTF, IDF, Tokenizer}
   
6. // $example off$
   
7. import org.apache.spark.sql.SQLContext
   
8. import org.apache.spark.{SparkConf, SparkContext}
   
9. 
   
10. object TfIdfExample {
    
11. 
    
12.   def main(args: Array[String]) {
    
13.     val conf = new SparkConf().setAppName("TfIdfExample")
    
14.     val sc = new SparkContext(conf)
    
15.     val sqlContext = new SQLContext(sc)
    
16. 
    
17.     // $example on$
    
18.     val sentenceData = sqlContext.createDataFrame(Seq(
    
19.       (0, "Hi I heard about Spark"),
    
20.       (0, "I wish Java could use case classes"),
    
21.       (1, "Logistic regression models are neat")
    
22.     )).toDF("label", "sentence")
    
23. 
    
24.     val tokenizer = new Tokenizer().setInputCol("sentence").setOutputCol("words")
    
25.     val wordsData = tokenizer.transform(sentenceData)
    
26.     val hashingTF = new HashingTF()
    
27.       .setInputCol("words").setOutputCol("rawFeatures").setNumFeatures(20)
    
28.     val featurizedData = hashingTF.transform(wordsData)
    
29.     val idf = new IDF().setInputCol("rawFeatures").setOutputCol("features")
    
30.     val idfModel = idf.fit(featurizedData)
    
31.     val rescaledData = idfModel.transform(featurizedData)
    
32.     rescaledData.select("features", "label").take(3).foreach(println)
    
33.     // $example off$
    
34.   }
    
35. }
    
36. // scalastyle:on printl

复制代码

1. https://en.wikipedia.org/wiki/Tf%E2%80%93idf

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Case study Frequency Document inverse DOCUME frequency document

不知道topic model能不能做？