A Nonsensical Language Model using Theano LSTM

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. Today we will train a nonsensical language model !
   
2. 
   
3. We will first collect some language data, convert it to numbers, and then feed it to a recurrent neural network and ask it to predict upcoming words. When we are done we will have a machine that can generate sentences from our made-up language ad-infinitum !

复制代码

本帖隐藏的内容

A Nonsensical Language Model using Theano LSTM.pdf (546.33 KB)

2017-9-11 02:45:31 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Language Theano model Using lang

1. def generate_nonsense(word = ""):
   
2.     if word.endswith("."):
   
3.         return word
   
4.     else:
   
5.         if len(word) > 0:
   
6.             word += " "
   
7.         word += samplers["stop"]()
   
8.         word += " " + samplers["noun"]()
   
9.         if random.random() > 0.7:
   
10.             word += " " + samplers["adverb"]()
    
11.             if random.random() > 0.7:
    
12.                 word += " " + samplers["adverb"]()
    
13.         word += " " + samplers["verb"]()
    
14.         if random.random() > 0.8:
    
15.             word += " " + samplers["noun"]()
    
16.             if random.random() > 0.9:
    
17.                 word += "-" + samplers["noun"]()
    
18.         if len(word) > 500:
    
19.             word += "."
    
20.         else:
    
21.             word += " " + samplers["punctuation"]()
    
22.         return generate_nonsense(word)
    
23. 
    
24. def generate_dataset(total_size, ):
    
25.     sentences = []
    
26.     for i in range(total_size):
    
27.         sentences.append(generate_nonsense())
    
28.     return sentences
    
29. 
    
30. # generate dataset               
    
31. lines = generate_dataset(100)

复制代码

1. ### Utilities:
   
2. class Vocab:
   
3.     __slots__ = ["word2index", "index2word", "unknown"]
   
4.    
   
5.     def __init__(self, index2word = None):
   
6.         self.word2index = {}
   
7.         self.index2word = []
   
8.         
   
9.         # add unknown word:
   
10.         self.add_words(["**UNKNOWN**"])
    
11.         self.unknown = 0
    
12.         
    
13.         if index2word is not None:
    
14.             self.add_words(index2word)
    
15.                
    
16.     def add_words(self, words):
    
17.         for word in words:
    
18.             if word not in self.word2index:
    
19.                 self.word2index[word] = len(self.word2index)
    
20.                 self.index2word.append(word)
    
21.                        
    
22.     def __call__(self, line):
    
23.         """
    
24.         Convert from numerical representation to words
    
25.         and vice-versa.
    
26.         """
    
27.         if type(line) is np.ndarray:
    
28.             return " ".join([self.index2word[word] for word in line])
    
29.         if type(line) is list:
    
30.             if len(line) > 0:
    
31.                 if line[0] is int:
    
32.                     return " ".join([self.index2word[word] for word in line])
    
33.             indices = np.zeros(len(line), dtype=np.int32)
    
34.         else:
    
35.             line = line.split(" ")
    
36.             indices = np.zeros(len(line), dtype=np.int32)
    
37.         
    
38.         for i, word in enumerate(line):
    
39.             indices[i] = self.word2index.get(word, self.unknown)
    
40.             
    
41.         return indices
    
42.    
    
43.     @property
    
44.     def size(self):
    
45.         return len(self.index2word)
    
46.    
    
47.     def __len__(self):
    
48.         return len(self.index2word)

复制代码

1. Create a Mapping from numbers to words
   
2. Now we can use the Vocab class to gather all the words and store an Index:
   
3. 
   
4. In [ ]:
   
5. vocab = Vocab()
   
6. for line in lines:
   
7.     vocab.add_words(line.split(" "))
   
8. To send our sentences in one big chunk to our neural network we transform each sentence into a row vector and place each of these rows into a bigger matrix that holds all these rows. Not all sentences have the same length, so we will pad those that are too short with 0s in pad_into_matrix:
   
9. 
   
10. In [168]:
    
11. def pad_into_matrix(rows, padding = 0):
    
12.     if len(rows) == 0:
    
13.         return np.array([0, 0], dtype=np.int32)
    
14.     lengths = map(len, rows)
    
15.     width = max(lengths)
    
16.     height = len(rows)
    
17.     mat = np.empty([height, width], dtype=rows[0].dtype)
    
18.     mat.fill(padding)
    
19.     for i, row in enumerate(rows):
    
20.         mat[i, 0:len(row)] = row
    
21.     return mat, list(lengths)
    
22. 
    
23. # transform into big numerical matrix of sentences:
    
24. numerical_lines = []
    
25. for line in lines:
    
26.     numerical_lines.append(vocab(line))
    
27. numerical_lines, numerical_lengths = pad_into_matrix(numerical_lines)

复制代码

1. Prediction
   
2. We have now defined our network. At each timestep we can produce a probability distribution for each input index:
   
3. 
   
4. def create_prediction(self, greedy=False):
   
5.     def step(idx, *states):
   
6.         # new hiddens are the states we need to pass to LSTMs
   
7.         # from past. Because the StackedCells also include
   
8.         # the embeddings, and those have no state, we pass
   
9.         # a "None" instead:
   
10.         new_hiddens = [None] + list(states)
    
11. 
    
12.         new_states = self.model.forward(idx, prev_hiddens = new_hiddens)
    
13.         return new_states[1:]
    
14.     ...
    
15. 
    
16. Our inputs are an integer matrix Theano symbolic variable:
    
17. 
    
18.     ...
    
19.     # in sequence forecasting scenario we take everything
    
20.     # up to the before last step, and predict subsequent
    
21.     # steps ergo, 0 ... n - 1, hence:
    
22.     inputs = self.input_mat[:, 0:-1]
    
23.     num_examples = inputs.shape[0]
    
24.     # pass this to Theano's recurrence relation function:

复制代码

谢谢楼主分享！

感谢分享

poas 8efp9w