Getting inf loss during validation of dataset after certain epochs

[JRMeyer]

>>> Preetam_XD
[February 23, 2021, 9:57am]

Hello guys , I m training on 0.9.1 deepspeech model on common voice
single word dataset on Google colab. After training the data set for
8-10 epochs validation loss gets inf and if I train further the training
loss also gets inf can anyone tell me what should I do? slash
I tried to lower the learning rate from 0.0001 to 0.00001 but still same
result. slash
Wer I get is around 0.88 which is not good at all please tell how can I
improve the model

Deepspeech version 0.9.1 slash
Training slash
Python version 3.6 slash
Tensorflow-gpu-1.15.4 slash
Windows 10 slash
Google colab

Ерoch 0 slash | Training slash | Elapsed Time: 0:38:02 slash | steps: 12706 slash | Loss:
4.281445 slash
Epoch 0 slash | validat ion slash | Elapsed Tine: 0:05:05 slash | Steps: 7198 slash | Loss:
4.998548 slash | Dataset: /content/en slash
I Saved new best validating model with loss 4.998548 to:
/content/drive/MyDrive/common_voice_eng/cl

Ероch 1 slash | Training slash | Elapsed Time: 0:37:52 Steps: 12706 slash | Loss:
4.097944 slash
Epoch 1 slash | validation slash | Elapsed Time: 0:05:03 Steps: 7198 slash | Loss:
5.146969 slash | Dataset: /content/en/clips

Ерoch 2 slash |Training slash | Elapsed Time: 0:37:53 steps: 12706 slash | Loss:
4.106288 slash
Epoch 2 slash | validation slash | Elapsed Time: 0:05:03 Steps: 7198 slash | Loss:
5.551651 slash | Dataset: /content/en/clips

Epoch 3 slash |Training slash | Elapsed Time: 0:37:53 slash | Steps: 12706 slash | Loss:
4.052530 slash
Epoch 3 slash | validation Elapsed Time: 0:05:40 slash | Steps: 7198 slash | Loss:
4.784578 slash | Dataset: /cantent/en/ slash
I Saved new best validat ing model with loss 4.784578 to:
/content/drive/comon_voice_eng/checkpoints

Epoch 4 slash |Training slash | Elapsed Time: 0:37:55 slash | Steps: 12706 Loss:
4.000706 slash
Epoch 4 slash |validation slash | Elapsed Time: 0:02:28 Steps: 3287 slash | loss:
6.391232 slash | Dataset: /cantent/en/clips slash
Epoch 4 slash | validation slash | Elapsed Time: 0:03:43 Steps: 4881 slash | Loss: inf
slash | Dataset: /content/en/clips slash
Epoch 4 slash | Validation slash | Elapsed Tine: 0:04:46 Steps: 6195 slash | Loss: inf
slash | Dataset /content/en/clips slash
Epoch 4 slash | validation slash | Elapsed Tine: 0:05:51 Steps: 7198I Loss: inf slash |
Datas.: /content/en/clips

[This is an archived TTS discussion thread from discourse.mozilla.org/t/getting-inf-loss-during-validation-of-dataset-after-certain-epochs]

[JRMeyer]

>>> othiele
[February 23, 2021, 10:09am]

Follow the guidelines, but I would guess dropout of 0.3 is missing ...

[Archived Post]

[JRMeyer]

>>> Preetam_XD
[February 23, 2021, 10:21am]

I used dropout_rate as 0.3 but still same result.

[Archived Post]

[JRMeyer]

>>> othiele
[February 23, 2021, 10:37am]

Can't help without
information.

[Archived Post]