ValueError: We need an offload_dir to dispatch this model according to this device_map > ```python

[alexhmyang]

          > ```python

lora_model_path

thank you! I follow your step, it moves on, but raise new issue.

(lmflow) u20@u20:~/LMFlow/service$ python app.py
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:21<00:00, 10.81s/it]
Traceback (most recent call last):
File "/home/u20/LMFlow/service/app.py", line 36, in
model = AutoModel.get_model(model_args, tune_strategy='none', ds_config=ds_config)
File "/home/u20/LMFlow/src/lmflow/models/auto_model.py", line 14, in get_model
return HFDecoderModel(model_args, *args, **kwargs)
File "/home/u20/LMFlow/src/lmflow/models/hf_decoder_model.py", line 192, in init
self.backend_model = PeftModel.from_pretrained(
File "/home/u20/miniconda3/envs/lmflow/lib/python3.9/site-packages/peft/peft_model.py", line 177, in from_pretrained
model = dispatch_model(model, device_map=device_map)
File "/home/u20/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/big_modeling.py", line 342, in dispatch_model
raise ValueError(
ValueError: We need an offload_dir to dispatch this model according to this device_map, the following submodules need to be offloaded: base_model.model.model.layers.22, base_model.model.model.layers.23, base_model.model.model.layers.24, base_model.model.model.layers.25, base_model.model.model.layers.26, base_model.model.model.layers.27, base_model.model.model.layers.28, base_model.model.model.layers.29, base_model.model.model.layers.30, base_model.model.model.layers.31, base_model.model.model.norm, base_model.model.lm_head.

after google, the answer may be this : oobabooga/text-generation-webui#383
is it because my llama7b is original model is too large and without quantize and compressed?

do you have a solution for this? thank you

Originally posted by @alexhmyang in #67 (comment)

[research4pan]

Thanks for your attention! Currently LoRA model doesn't support ram-optimized load (which offloads parameters to disks during loading). You can try to add --use_ram_optimized_load False to disable it and check if it works. Thanks 😄

[alexhmyang]

i am not very clear whether i understand your suggestion, and it not works.

my issue is : I want to load finetuned model to provide service( i have llama original 7b model, transfer it, and download several finetuned model. ) but something seems wrong .

[research4pan]

I checked the implementation of service/app.py. You may change this line to

model_args = ModelArguments(model_name_or_path=model_name_or_path, use_ram_optimized_load=False, lora_model_path=lora_path)

to see if the problem is resolved. Thanks!

[zhangjiahuang]

这样修改之后出错了
TypeError: init() got an unexpected keyword argument 'use_ram_optimized_load'

[research4pan]

This is a bit strange. The model argument should accept this field in our latest main branch. Could you please provide the commit id of your code base so we may check that for you? Thanks~

[zhangjiahuang]

ould you please provide the commit id of your code base so we may check that for you
执行的是service/app.py
修改部分的代码如下：
model_name_or_path = 'models--pinkmanlove--llama-7b-hf'
lora_path = '../output_models/llama7b-lora-380k'
model_args = ModelArguments(model_name_or_path=model_name_or_path, use_ram_optimized_load=False, lora_model_path=lora_path)
报错信息如下：
Traceback (most recent call last):
File "/root/LMFlow/service/app.py", line 27, in
model_args = ModelArguments(model_name_or_path=model_name_or_path, use_ram_optimized_load=False, lora_model_path=lora_path)
TypeError: init() got an unexpected keyword argument 'use_ram_optimized_load'

[shizhediao]

Hi, was the issue resolved?

Thanks!

[shizhediao]

This issue has been marked as stale because it has not had recent activity. If you think this still needs to be addressed please feel free to reopen this issue. Thanks

[larawehbe]

For anyone in the future passing over this, it should be "offload_folder" instead of "offload_dir"
This is odd. Maintainers at Peft shoud fix it
anyway, as of the date of today, it can work like this :

base_model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    device_map='auto',
    offload_folder="offload/"
)

base_model.config.use_cache = False

model = PeftModel.from_pretrained(
    base_model,
    output_dir,
    offload_folder = "offload/"
    )

[pdmhuman]

ValueError: At least one of the model submodule will be offloaded to disk, please pass along an offload_folder.
I recieved this error. please help what to change and in which file have to change? using Text Generation WebUI

[jiahuiLeee]

For anyone in the future passing over this, it should be "offload_folder" instead of "offload_dir" This is odd. Maintainers at Peft shoud fix it anyway, as of the date of today, it can work like this :

base_model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    device_map='auto',
    offload_folder="offload/"
)

base_model.config.use_cache = False

model = PeftModel.from_pretrained(
    base_model,
    output_dir,
    offload_folder = "offload/"
    )

It worked!