How long does it take to train the model with common voice training set

[JRMeyer]

>>> jackhuang
[December 24, 2017, 12:01pm]

I use one GPU (Tesla, 12G) to train the model and I have trained the
model with default parameters for 11 days, but the training progress（75
epoch） have not finished. slash
I have used 'ctrl + c' to interrupt the training process for 2 times and
continued the training process with the same checkpoint_dir. I wander
whether my configuration or my operation is wrong, so the program will
take much time training.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/how-long-does-it-take-to-train-the-model-with-common-voice-training-set]

[JRMeyer]

>>> lissyx
[January 5, 2018, 5:42am]

Training on our cluster (16 GPUs TITAN X, 12G) takes severals hours for
each epoch, so full Common Voice with just one GPU is not unlikely to
take a long time. How much epochs have you completed over 11 days ?
Maybe not even one ? Might not be that surprising.

[Archived Post]

[JRMeyer]

>>> jackhuang
[January 8, 2018, 6:20am]

I find that some of the data show in the 'train-other.csv' file are
defective. For example, the transcript of the data is blank.

[Archived Post]

[JRMeyer]

>>> jackhuang
[January 8, 2018, 6:22am]

Besides, when I add '--display_step 1' to my command, the training
process will be very slow. I am very curious about this.

[Archived Post]

[JRMeyer]

>>> lissyx
[January 8, 2018, 10:38am]

There's nothing to be curious about, display_step computes WER, which
is very intensive to compute.

[Archived Post]

[JRMeyer]

>>> jackhuang
[February 1, 2018, 6:43am]

Is there some ways that make the program not compute the WER report
after finishing the training process?

[Archived Post]