[DeepNLP]Deep Learning NLP Pipeline on Tensorflow

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. Deep Learning NLP Pipeline implemented on Tensorflow. Following the 'simplicity' rule, this project aims to use the deep learning library of Tensorflow to implement new NLP pipeline. You can extend the project to train models with your own corpus/languages. Pretrained models of Chinese corpus are distributed. Free RESTful NLP API are also provided. Visit http://www.deepnlp.org/api/v1.0/pipeline for details.
   
2. 
   
3. Brief Introduction
   
4. 
   
5. Modules
   
6. Installation
   
7. Tutorial
   
8. Segmentation
   
9. POS
   
10. NER
    
11. Pipeline
    
12. Textsum
    
13. Textrank
    
14. Textcnn
    
15. Train your model
    
16. Web API Service
    
17. 中文简介
    
18. 安装说明
    
19. Reference
    
20. Modules
    
21. 
    
22. NLP Pipeline Modules:
    
23. 
    
24. Word Segmentation/Tokenization
    
25. Part-of-speech (POS)
    
26. Named-entity-recognition(NER)
    
27. textsum: automatic summarization Seq2Seq-Attention models
    
28. textrank: extract the most important sentences
    
29. textcnn: document classification
    
30. Web API: Free Tensorflow empowered web API
    
31. Planed: Parsing, Automatic Summarization
    
32. Algorithm(Closely following the state-of-Art)
    
33. 
    
34. Word Segmentation: Linear Chain CRF(conditional-random-field), based on python CRF++ module
    
35. POS: LSTM/BI-LSTM network, based on Tensorflow
    
36. NER: LSTM/BI-LSTM/LSTM-CRF network, based on Tensorflow
    
37. Textsum: Seq2Seq with attention mechanism
    
38. Texncnn: CNN
    
39. Pre-trained Model
    
40. 
    
41. Chinese: Segmentation, POS, NER (1998 china daily corpus)
    
42. English: POS (brown corpus)
    
43. For your Specific Language, you can easily use the script to train model with the corpus of your language choice.
    
44. Installation
    
45. 
    
46. Requirements
    
47. 
    
48. CRF++ (>=0.54)
    
49. Tensorflow(1.0) This project is up to date with the latest tensorflow release. For tensorflow (<=0.12.0), use deepnlp <=0.1.5 version. See RELEASE.md for more details
    
50. Pip
    
51. 
    
52.     # linux, run the script:
    
53.     pip install deepnlp

复制代码

本帖隐藏的内容

https://github.com/rockingdingo/deepnlp#segmentation

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏1 回帖

关键词：pipeline Learning earning Tensor Learn

1. Download pretrained models
   
2. 
   
3. 下载预训练模型 If you install deepnlp via pip, the pre-trained models are not distributed due to size restriction. You can download full models for 'Segment', 'POS' en and zh, 'NER' zh, 'Textsum' by calling the download function.
   
4. 
   
5. import deepnlp
   
6. # Download all the modules
   
7. deepnlp.download()
   
8. 
   
9. # Download only specific module
   
10. deepnlp.download('segment')
    
11. deepnlp.download('pos')
    
12. deepnlp.download('ner')
    
13. deepnlp.download('textsum')

复制代码

1. Segmentation
   
2. 
   
3. 分词模块
   
4. 
   
5. #coding=utf-8
   
6. from __future__ import unicode_literals
   
7. 
   
8. from deepnlp import segmenter
   
9. 
   
10. text = "我刚刚在浙江卫视看了电视剧老九门，觉得陈伟霆很帅"
    
11. segList = segmenter.seg(text)
    
12. text_seg = " ".join(segList)
    
13. 
    
14. print (text.encode('utf-8'))
    
15. print (text_seg.encode('utf-8'))
    
16. 
    
17. #Results
    
18. #我 刚刚 在 浙江卫视 看 了 电视剧 老九门 ， 觉得 陈伟霆 很 帅

复制代码

1. POS
   
2. 
   
3. 词性标注
   
4. 
   
5. #coding:utf-8
   
6. from __future__ import unicode_literals
   
7. 
   
8. import deepnlp
   
9. deepnlp.download('pos')
   
10. 
    
11. ## English Model
    
12. from deepnlp import pos_tagger
    
13. tagger = pos_tagger.load_model(lang = 'en')  # Loading English model, lang code 'en', English Model Brown Corpus
    
14. 
    
15. text = "I want to see a funny movie"
    
16. words = text.split(" ")     # unicode
    
17. print (" ".join(words).encode('utf-8'))
    
18. 
    
19. tagging = tagger.predict(words)
    
20. for (w,t) in tagging:
    
21.     str = w + "/" + t
    
22.     print (str.encode('utf-8'))
    
23.    
    
24. #Results
    
25. #I/nn want/vb to/to see/vb a/at funny/jj movie/nn
    
26. 
    
27. ## Chinese Model
    
28. from deepnlp import segmenter
    
29. from deepnlp import pos_tagger
    
30. tagger = pos_tagger.load_model(lang = 'zh') # Loading Chinese model, lang code 'zh', China Daily Corpus
    
31. 
    
32. text = "我爱吃北京烤鸭"
    
33. words = segmenter.seg(text) # words in unicode coding
    
34. print (" ".join(words).encode('utf-8'))
    
35. 
    
36. tagging = tagger.predict(words)  # input: unicode coding
    
37. for (w,t) in tagging:
    
38.     str = w + "/" + t
    
39.     print (str.encode('utf-8'))
    
40. 
    
41. #Results
    
42. #我/r 爱/v 吃/v 北京/ns 烤鸭/n

复制代码

1. NER
   
2. 
   
3. 命名实体识别
   
4. 
   
5. #coding:utf-8
   
6. from __future__ import unicode_literals
   
7. 
   
8. # Download pretrained NER model
   
9. import deepnlp
   
10. deepnlp.download('ner')
    
11. 
    
12. from deepnlp import segmenter
    
13. from deepnlp import ner_tagger
    
14. tagger = ner_tagger.load_model(lang = 'zh') # Loading Chinese NER model
    
15. 
    
16. text = "我爱吃北京烤鸭"
    
17. words = segmenter.seg(text)
    
18. print (" ".join(words).encode('utf-8'))
    
19. 
    
20. tagging = tagger.predict(words)
    
21. for (w,t) in tagging:
    
22.     str = w + "/" + t
    
23.     print (str.encode('utf-8'))
    
24. 
    
25. #Results
    
26. #我/nt 爱/nt 吃/nt 北京/p 烤鸭/nt

复制代码

1. Pipeline
   
2. 
   
3. #coding:utf-8
   
4. from __future__ import unicode_literals
   
5. 
   
6. from deepnlp import pipeline
   
7. p = pipeline.load_model('zh')
   
8. 
   
9. #Segmentation
   
10. text = "我爱吃北京烤鸭"
    
11. res = p.analyze(text)
    
12. 
    
13. print (res[0].encode('utf-8'))
    
14. print (res[1].encode('utf-8'))
    
15. print (res[2].encode('utf-8'))
    
16. 
    
17. words = p.segment(text)
    
18. pos_tagging = p.tag_pos(words)
    
19. ner_tagging = p.tag_ner(words)
    
20. 
    
21. print (pos_tagging.encode('utf-8'))
    
22. print (ner_tagging.encode('utf-8'))

复制代码

谢谢楼主分享！

Great, thanks for sharing!