Compare Vanilla RNN's vs Current LSTM's

[kdavis-mozilla]

No description provided.

[andi4191]

@kdavis-mozilla : I noticed there is already a branch with name issue362. Has there been some changes made for this benchmark variation algorithm in the past? Should I rename my local branch or do we need the changes made on this branch as well for this ?

[kdavis-mozilla]

@andi4191 Just deleted it. The branch name is all yours.

[andi4191]

DataSet Used: TED
Vanilla RNN:

Number of GPU instances: 4
Learning rate : 0.0001. (Optimal)
Epochs: 29 (Optimal)

Code changes: https://github.com/andi4191/DeepSpeech/commit/933506d07c818e97190e458d0168085a51514bc4

Inference time: 0:09:17 (On test set)
I job type: dev, inference time: 0:25:16
I Validation of Epoch 28 - loss: 133.591197
I job type: train, inference time: 0:25:06
I Training of Epoch 29 - loss: 84.073379
I job type: dev, inference time: 0:25:18
I Validation of Epoch 29 - loss: 135.241456
I Early stop triggered as (for last 4 steps) validation loss: 135.241456 with standard deviation: 0.586952 and mean: 134.408705
I FINISHED Optimization - training time: 2:05:50
I job type: test, inference time: 0:09:17
I Test of Epoch 30 - WER: 0.514973, loss: 94.1673003303, mean edit distance: 0.248363
I --------------------------------------------------------------------------------
I WER: 0.250000, loss: 4.074897, mean edit distance: 0.052632
I - src: "if this doesnt work"
I - res: "if this doesnt wore"
I --------------------------------------------------------------------------------
I WER: 0.333333, loss: 2.654649, mean edit distance: 0.100000
I - src: "i said don"
I - res: "i said done"
I --------------------------------------------------------------------------------
I WER: 0.333333, loss: 3.617916, mean edit distance: 0.076923
I - src: "no im serious"
I - res: "now im serious"
I --------------------------------------------------------------------------------
I WER: 0.333333, loss: 3.842691, mean edit distance: 0.090909
I - src: "yes he said"
I - res: "yes ye said"
I --------------------------------------------------------------------------------
I WER: 0.333333, loss: 4.307844, mean edit distance: 0.153846
I - src: "no im serious"
I - res: "now im serious"
I --------------------------------------------------------------------------------
I WER: 0.500000, loss: 1.293201, mean edit distance: 0.100000
I - src: "farms that"
I - res: "forms that"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 2.222308, mean edit distance: 0.500000
I - src: "so"
I - res: "o"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 2.724182, mean edit distance: 0.333333
I - src: "and"
I - res: "nd"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 3.569519, mean edit distance: 0.500000
I - src: "so"
I - res: "of"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 4.486110, mean edit distance: 0.133333
I - src: "not intensively"
I - res: "not intensive by"
I --------------------------------------------------------------------------------

Current LSTM Cell:

Number of GPU instance: 4
Number of Epochs: 10
Learning rate : 0.0001 (Optimal)
Inference time: 0:08:57 (test set)

I FINISHED Optimization - training time: 1:11:12
I Inference time: 0:08:57
I Test of Epoch 10 - WER: 0.424047, loss: 76.2531689538, mean edit distance: 0.199896
I --------------------------------------------------------------------------------
I WER: 0.166667, loss: 2.970927, mean edit distance: 0.034483
I - src: "and one of the most important"
I - res: "ad one of the most important"
I --------------------------------------------------------------------------------
I WER: 0.200000, loss: 2.782618, mean edit distance: 0.043478
I - src: "it was a beautiful fish"
I - res: "i was a beautiful fish"
I --------------------------------------------------------------------------------
I WER: 0.250000, loss: 2.090823, mean edit distance: 0.047619
I - src: "thats success he said"
I - res: "that success he said"
I --------------------------------------------------------------------------------
I WER: 0.250000, loss: 2.751168, mean edit distance: 0.047619
I - src: "thats success he said"
I - res: "thats success said"
I --------------------------------------------------------------------------------
I WER: 0.250000, loss: 2.800725, mean edit distance: 0.052632
I - src: "if this doesnt work"
I - res: "if this doesnt wok"
I --------------------------------------------------------------------------------
I WER: 0.333333, loss: 1.649019, mean edit distance: 0.037037
I - src: "and not just for those fish"
I - res: "and not just porthole fish"
I --------------------------------------------------------------------------------
I WER: 0.500000, loss: 2.078481, mean edit distance: 0.100000
I - src: "farms that"
I - res: "firms that"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 1.906928, mean edit distance: 0.500000
I - src: "so"
I - res: "o"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 2.133130, mean edit distance: 0.500000
I - src: "so"
I - res: "o"
I --------------------------------------------------------------------------------
I WER: 1.000000, loss: 2.137708, mean edit distance: 0.333333
I - src: "and"
I - res: "nd"
I --------------------------------------------------------------------------------

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.