GPU memory reduction: will this method reduce GPU memory consumption?

[bing0037]

Hi, Thanks for you excellent work on Dynamic Sparse Training. I am trying to reproduce your work and to reduce GPU memory consumption during sparse training.

I read through your code and try to implement your method on another vgg training implement to verify my understanding of your code. Here is my modification:

My modification:

bing0037/pytorch-vgg-cifar10_ITOP@6dab205

My running scripts:

# baseline: 
model=vgg16
# dense training:
CUDA_VISIBLE_DEVICES=1 python main.py  --arch=$model

# sparse training: ITOP with RigL
CUDA_VISIBLE_DEVICES=1 python main.py  --arch=$model --sparse --sparse_init ERK  --multiplier 1 --density 0.05 --update_frequency 4000 --growth gradient --death magnitude --redistribution none

Result: GPU memory consumption:

Baseline: 2639MB
ITOP with RigL: 2765MB

Question:

My run results show that ITOP with RigL consumes more or equal (it is supposed to be significantly less, right?) GPU memory than normal. Could you help me to figure out the problem of my implementation or any comment or suggestions?

Thanks.

[Shiweiliuiiiiiii]

Hi, sorry for the late response. The sparse operation in this repo is implemented with masks due to the limited support of sparsity on GPU. Thus, the GPU memory consumption of ITOP will be weight memory + mask memory, which is larger than standard training.