1.? 什么是LSTM

長(zhǎng)短期記憶網(wǎng)絡(luò) LSTM（long short-term memory）是 RNN 的一種變體，其核心概念在于細(xì)胞狀態(tài)以及“門”結(jié)構(gòu)。細(xì)胞狀態(tài)相當(dāng)于信息傳輸?shù)穆窂?#xff0c;讓信息能在序列連中傳遞下去。你可以將其看作網(wǎng)絡(luò)的“記憶”。理論上講，細(xì)胞狀態(tài)能夠?qū)⑿蛄刑幚磉^(guò)程中的相關(guān)信息一直傳遞下去。因此，即使是較早時(shí)間步長(zhǎng)的信息也能攜帶到較后時(shí)間步長(zhǎng)的細(xì)胞中來(lái)，這克服了短時(shí)記憶的影響。信息的添加和移除我們通過(guò)“門”結(jié)構(gòu)來(lái)實(shí)現(xiàn)，“門”結(jié)構(gòu)在訓(xùn)練過(guò)程中會(huì)去學(xué)習(xí)該保存或遺忘哪些信息。
?

?

2. 實(shí)驗(yàn)代碼

2.1. 搭建一個(gè)只有一層RNN和Dense網(wǎng)絡(luò)的模型。

2.2.?驗(yàn)證LSTM里的邏輯

?假設(shè)我的輸入數(shù)據(jù)是x = [1,0],?

kernel = [[[2, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0],

? ? ? ? ? ? ? [1, 1, 0, 1, 1, 0, 0, 1, 1 ,0, 0, 0],]]

recurrent_kernel =?[[1, 0, 0, 1, 2,1,0,1,2,0,1,0],

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [1, 1, 0, 0, 2,1,0,1,2,2,0,0],

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [1, 0, 1, 2, 0,1,0,1,1,0,1,0]]

biase =?[3, 1, 0, 1, 1,0,0,1,0,2,0.0,0]

通過(guò)下面手算，h的結(jié)果是[0, 4,1], c 的結(jié)果是[0,4,1].? 注意無(wú)激活函數(shù)。

代碼驗(yàn)證上面的結(jié)果

def change_weight():# Create a simple Dense layerlstm_layer = LSTM(units=3, input_shape=(3, 2), activation=None, recurrent_activation=None, return_sequences=True,return_state= True)# Simulate input data (batch size of 1 for demonstration)input_data = np.array([[[1.0, 2], [2, 3], [3, 4]],[[5, 6], [6, 7], [7, 8]],[[9, 10], [10, 11], [11, 12]]])# Pass the input data through the layer to initialize the weights and biaseslstm_layer(input_data)kernel, recurrent_kernel, biases = lstm_layer.get_weights()# Print the initial weights and biasesprint("recurrent_kernel:", recurrent_kernel, recurrent_kernel.shape ) # (3,3)print('kernal:',kernel, kernel.shape) #(2,3)print('biase: ',biases , biases.shape) # (3)kernel = np.array([[2, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 0],[1, 1, 0, 1, 1, 0, 0, 1, 1 ,0, 0, 0],])recurrent_kernel = np.array([[1, 0, 0, 1, 2,1,0,1,2,0,1,0],[1, 1, 0, 0, 2,1,0,1,2,2,0,0],[1, 0, 1, 2, 0,1,0,1,1,0,1,0]])biases = np.array([3, 1, 0, 1, 1,0,0,1,0,2,0.0,0])lstm_layer.set_weights([kernel, recurrent_kernel, biases])print(lstm_layer.get_weights())# test_data = np.array([#     [[1.0, 3], [1, 1], [2, 3]]# ])test_data = np.array([[[1,0.0]]])output, memory_state, carry_state  = lstm_layer(test_data)print(output)print(memory_state)print(carry_state)
if __name__ == '__main__':change_weight()

執(zhí)行結(jié)果：

recurrent_kernel: [[-0.36744034 -0.11181469 -0.10642298  0.5450207  -0.30208975  0.54054320.09643812 -0.14983998  0.1859854   0.2336958  -0.16187981  0.11621032][ 0.07727922 -0.226477    0.1491096  -0.03933501  0.31236103 -0.129630920.10522162 -0.4815724  -0.2093935   0.34740582 -0.60979587 -0.15877807][ 0.15371156  0.01244636 -0.09840634 -0.32093546  0.06523462  0.189349320.38859126 -0.3261706  -0.05138849  0.42713478  0.49390993  0.37013963]] (3, 12)
kernal: [[-0.47606698 -0.43589187 -0.5371355  -0.07337284  0.30526626 -0.18241835-0.03675252  0.2873094   0.33218485  0.24838251  0.17765659  0.4312396 ][ 0.4007727   0.41280174  0.40750778 -0.6245315   0.6382301   0.428892250.11961156 -0.6021105  -0.43556038  0.39798307  0.6390712   0.16719025]] (2, 12)
biase:  [0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0.] (12,)
[array([[2., 1., 1., 0., 0., 0., 0., 1., 1., 0., 1., 0.],[1., 1., 0., 1., 1., 0., 0., 1., 1., 0., 0., 0.]], dtype=float32), array([[1., 0., 0., 1., 2., 1., 0., 1., 2., 0., 1., 0.],[1., 1., 0., 0., 2., 1., 0., 1., 2., 2., 0., 0.],[1., 0., 1., 2., 0., 1., 0., 1., 1., 0., 1., 0.]], dtype=float32), array([3., 1., 0., 1., 1., 0., 0., 1., 0., 2., 0., 0.], dtype=float32)]
tf.Tensor([[[0. 4. 0.]]], shape=(1, 1, 3), dtype=float32)
tf.Tensor([[0. 4. 0.]], shape=(1, 3), dtype=float32)
tf.Tensor([[0. 4. 1.]], shape=(1, 3), dtype=float32)

可以看出h=[0,4,0], c=[0,4,1]