why do u need 2GPUs to run Qwen2.5 3B?

[rpking1107]

sorry if it's a stupid question.
But my 4090 is 24GB ram, it's more than enough to run a 3B model right? can i use Qwen2.5 3B and change the parameter to GPU=1?

thanks guys

[Superskyyy]

Training is different from inference, you need way more vram, safe to say your 4090 is not enough.

[zacksiri]

@rpking1107 i was able to get training running on my A4500 with 20GB ram. you just need to adjust the training parameters. It's going to take longer because of smaller batch size, but it does work.

[carlos-aguayo]

@zacksiri Can you share what parameters did you use? I tried lowering the batch sizes and a few others and would still run out of memory.

[zacksiri]

@carlos-aguayo You can see my parameters here.

#5 (comment)

It works with qwen2.5 1.5b, I'm not sure I have the best configuration / output yet. Need to try a few more experiments.

I'm experimenting with 3b model but I think I'm pushing my luck.