[Case Study]Simple Text Classification using Python

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. from_future__ import print_function
   
2. 
   
3. from pyspark import SparkContext
   
4. from pyspark.ml import Pipeline
   
5. from pyspark.ml.classification import LogisticRegression
   
6. from pyspark.ml.feature import HashingTF, Tokenizer
   
7. from pyspark.sql import Row, SQLContext
   
8. 
   
9. 
   
10. """
    
11. A simple text classification pipeline that recognizes "spark" from
    
12. input text. This is to show how to create and configure a Spark ML
    
13. pipeline in Python. Run with:
    
14.   bin/spark-submit examples/src/main/python/ml/simple_text_classification_pipeline.py
    
15. """
    
16. 
    
17. 
    
18. if __name__ == "__main__":
    
19.     sc = SparkContext(appName="SimpleTextClassificationPipeline")
    
20.     sqlContext = SQLContext(sc)
    
21. 
    
22.     # Prepare training documents, which are labeled.
    
23.     LabeledDocument = Row("id", "text", "label")
    
24.     training = sc.parallelize([(0, "a b c d e spark", 1.0),
    
25.                                (1, "b d", 0.0),
    
26.                                (2, "spark f g h", 1.0),
    
27.                                (3, "hadoop mapreduce", 0.0)]) \
    
28.         .map(lambda x: LabeledDocument(*x)).toDF()
    
29. 
    
30.     # Configure an ML pipeline, which consists of tree stages: tokenizer, hashingTF, and lr.
    
31.     tokenizer = Tokenizer(inputCol="text", outputCol="words")
    
32.     hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol="features")
    
33.     lr = LogisticRegression(maxIter=10, regParam=0.001)
    
34.     pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
    
35. 
    
36.     # Fit the pipeline to training documents.
    
37.     model = pipeline.fit(training)
    
38. 
    
39.     # Prepare test documents, which are unlabeled.
    
40.     Document = Row("id", "text")
    
41.     test = sc.parallelize([(4, "spark i j k"),
    
42.                            (5, "l m n"),
    
43.                            (6, "spark hadoop spark"),
    
44.                            (7, "apache hadoop")]) \
    
45.         .map(lambda x: Document(*x)).toDF()
    
46. 
    
47.     # Make predictions on test documents and print columns of interest.
    
48.     prediction = model.transform(test)
    
49.     selected = prediction.select("id", "text", "prediction")
    
50.     for row in selected.collect():
    
51.         print(row)
    
52. 
    
53.     sc.stop()
    
54. Status API Training Shop Blog About Pricing
    
55. © 2015 GitHub, Inc. Terms Privacy Security Contact Help

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Case study simple cation python Using examples python simple create import