Replies: 5 comments
-
|
>>> Bala_Murugan |
Beta Was this translation helpful? Give feedback.
-
|
>>> lissyx |
Beta Was this translation helpful? Give feedback.
-
|
>>> Bala_Murugan |
Beta Was this translation helpful? Give feedback.
-
|
>>> lissyx |
Beta Was this translation helpful? Give feedback.
-
|
>>> Bala_Murugan |
Beta Was this translation helpful? Give feedback.
-
>>> Bala_Murugan
[January 12, 2021, 7:36am]
Hi,
Usually I used to store the Training Outputs in Text file. When I compare those Training outputs with the Tensorboard Graphs. Its Mismatching. . .
Tensorboard is showing graphs which is irrelavant to the Training
Output.
See in my Training Output. . .
Loading best validating checkpoint from
/home/bala/Speech_Recognition/Training_Pappa/v1-Jan5_No_Augment/deepspeech-0.9.2-checkpoint/best_dev-1564701 slash
learning_rate : 0.000005
Epoch 0 slash | Training slash | Elapsed Time: 0:07:48 slash | Steps: 963 slash | Loss:
33.665629 slash
Epoch 0 slash | Validation slash | Elapsed Time: 0:01:14 slash | Steps: 482 slash | Loss:
32.672577 slash | Dataset:
/home/bala/Speech_Recognition/Training_Data_AWS/Dec-30_v1_Files/awsgroupdev.csv slash
I Saved new best validating model with loss 32.672577 to:
/home/bala/Speech_Recognition/Training_Pappa/v1-Jan5_No_Augment/deepspeech-0.9.2-checkpoint/best_dev-1565664
Epoch 1 slash | Training slash | Elapsed Time: 0:07:59 slash | Steps: 963 slash | Loss:
33.600953 slash
Epoch 1 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
32.527303 slash |
Epoch 2 slash | Training slash | Elapsed Time: 0:07:58 slash | Steps: 963 slash | Loss:
33.350906 slash
Epoch 2 slash | Validation slash | Elapsed Time: 0:01:12 slash | Steps: 482 slash | Loss:
32.453275 slash |
Epoch 3 slash | Training slash | Elapsed Time: 0:07:58 slash | Steps: 963 slash | Loss:
33.195642 slash
Epoch 3 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
32.487144 slash |
Epoch 4 slash | Training slash | Elapsed Time: 0:07:58 slash | Steps: 963 slash | Loss:
33.144140 slash
Epoch 4 slash | Validation slash | Elapsed Time: 0:01:12 slash | Steps: 482 slash | Loss:
32.416610
Epoch 5 slash | Training slash | Elapsed Time: 0:07:59 slash | Steps: 963 slash | Loss:
32.796107 slash
Epoch 5 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
32.416092
Epoch 6 slash | Training slash | Elapsed Time: 0:08:00 slash | Steps: 963 slash | Loss:
32.730357 slash
Epoch 6 slash | Validation slash | Elapsed Time: 0:01:12 slash | Steps: 482 slash | Loss:
32.320890
Epoch 7 slash | Training slash | Elapsed Time: 0:08:00 slash | Steps: 963 slash | Loss:
32.626243 slash
Epoch 7 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
32.322866
Epoch 8 slash | Training slash | Elapsed Time: 0:08:00 slash | Steps: 963 slash | Loss:
32.543887 slash
Epoch 8 slash | Validation slash | Elapsed Time: 0:01:12 slash | Steps: 482 slash | Loss:
32.240156 slash |
Epoch 9 slash | Training slash | Elapsed Time: 0:07:59 slash | Steps: 963 slash | Loss:
32.429364 slash
Epoch 9 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
32.168196
Epoch 10 slash | Training slash | Elapsed Time: 0:07:58 slash | Steps: 963 slash | Loss:
32.202116 slash
Epoch 10 slash | Validation slash | Elapsed Time: 0:01:12 slash | Steps: 482 slash | Loss:
32.198536
. slash
. slash
. slash
Epoch 44 slash | Training slash | Elapsed Time: 0:08:06 slash | Steps: 963 slash | Loss:
28.093275 slash
Epoch 44 slash | Validation slash | Elapsed Time: 0:01:13 slash | Steps: 482 slash | Loss:
31.266112
See My Training and Validation loss is Keep on decreasing, but the
tensorgraph showing irrelevant outputs.
(103).filename}[1366×768 128
[This is an archived TTS discussion thread from discourse.mozilla.org/t/getting-wrong-tensorboard-graphs-respective-to-training-output]
Beta Was this translation helpful? Give feedback.
All reactions