Model Garden : How to continue training with tfm.core.train_lib.run_experiment()?

[Pormons]

Hi, I am new and trying the model garden tfm package. I am training my model in Google Colab, and the checkpoints are being saved to my google drive, how can I continue my training using this checkpoint? using the model gardens tfm package? tfm.core.train_lib.run_experiment()?

I have followed this https://www.tensorflow.org/tfmodels/vision/object_detection tutorial, due to google colabs limited resources I cant really complete the training in one session now I am trying to contnue the training from those checkpoints and I dont know how.

[stefan-it]

Hi @Pormons ,

I'm not vision export but I saw this checkpoint loading in other examples.

E.g. in your experiment yaml configuration, there should be the possibility to set the task.init_checkpoint variable to a e.g. Google bucket. e.g. on command line with --params_override=task.init_checkpoint='gs://folder/ckpt-33264'.

I downloaded a ResNet checkpoint from here:

$ wget https://storage.googleapis.com/tf_model_garden/vision/retinanet/retinanet-resnet50fpn.tar.gz

After unpacking the checkpoint structure looks like:

ckpt-33264.data-00000-of-00001  ckpt-33264.index

So you would use gs://folder/ckpt-33264 as value for the task.init_checkpoint option.

For your trained checkpoint you have adjust the checkpoint name.

Additionally it is also possible to pass this way in your training yaml configuration like:

task:
  init_checkpoint: 'gs://folder/ckpt-33264'

I hope this helps :)