[Case Study]Term Frequency-Inverse Document Frequency using Python

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. from __future__ import print_function
   
2. 
   
3. from pyspark import SparkContext
   
4. # $example on$
   
5. from pyspark.ml.feature import HashingTF, IDF, Tokenizer
   
6. # $example off$
   
7. from pyspark.sql import SQLContext
   
8. 
   
9. if __name__ == "__main__":
   
10.     sc = SparkContext(appName="TfIdfExample")
    
11.     sqlContext = SQLContext(sc)
    
12. 
    
13.     # $example on$
    
14.     sentenceData = sqlContext.createDataFrame([
    
15.         (0, "Hi I heard about Spark"),
    
16.         (0, "I wish Java could use case classes"),
    
17.         (1, "Logistic regression models are neat")
    
18.     ], ["label", "sentence"])
    
19.     tokenizer = Tokenizer(inputCol="sentence", outputCol="words")
    
20.     wordsData = tokenizer.transform(sentenceData)
    
21.     hashingTF = HashingTF(inputCol="words", outputCol="rawFeatures", numFeatures=20)
    
22.     featurizedData = hashingTF.transform(wordsData)
    
23.     idf = IDF(inputCol="rawFeatures", outputCol="features")
    
24.     idfModel = idf.fit(featurizedData)
    
25.     rescaledData = idfModel.transform(featurizedData)
    
26.     for features_label in rescaledData.select("features", "label").take(3):
    
27.         print(features_label)
    
28.     # $example off$
    
29. 
    
30.     sc.stop()

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Case study Frequency Document inverse python example future import