Which checkpoints i can delete if i have storage problem

[JRMeyer]

>>> Sushantmkarande
[May 2, 2019, 2:32pm]

I am fine-tuning deepspeech 0.4.1 model on aws and now checkpoint size
is more than 7 gb I have only 15 gb of size on aws so i am getting OOM
error. slash
so which old checkpoints can i delete.

[This is an archived TTS discussion thread from discourse.mozilla.org/t/which-checkpoints-i-can-delete-if-i-have-storage-problem]

[JRMeyer]

>>> kdavis
[May 3, 2019, 5:21am]

OOM does not mean you are running out of disk space. It means you are
running out of GPU memory.

Also we can't decide which checkpoints you can delete as we don't know
which ones are important to your particular use case.

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[May 4, 2019, 2:26pm]

i dont remember the exact
error but it said resource exhause error and also my aws disk was 100%
full when i deleted some files it started training.

now my disk is full because of all the checkpoints and now I have
deleted all files which are not mentioned in checkpoint txt

model_checkpoint_path: 'model.ckpt-228410'
all_model_checkpoint_paths: 'model.ckpt-228363'
all_model_checkpoint_paths: 'model.ckpt-228375'
all_model_checkpoint_paths: 'model.ckpt-228387'
all_model_checkpoint_paths: 'model.ckpt-228399'
all_model_checkpoint_paths: 'model.ckpt-228410'

ex- all files before 228300 and now my fine-tuning started normally.

is this the right way.

[Archived Post]

[JRMeyer]

>>> kdavis
[May 5, 2019, 5:55am]

This seems reasonable. However, again we can't decide which checkpoints
you can delete as we don't know which ones are important to your
particular use case.

[Archived Post]

[JRMeyer]

>>> Sushantmkarande
[May 6, 2019, 5:48am]

thanks, have to go with reasonable then, no other
option:thinking:![:thinking:](

[Archived Post]