FSDP+QLoRA微调后推理

[bgtii]

Reminder

• I have read the README and searched the existing issues.

Reproduction

微调
CUDA_VISIBLE_DEVICES=4,5 accelerate launch
--config_file /root/workspace/LLaMA-Factory/examples/accelerate/fsdp_config.yaml
/root/workspace/LLaMA-Factory/src/train_bash.py
--stage sft
--do_train
--model_name_or_path /models/llama-7b
--dataset 肾科ai生成1000问
--dataset_dir /root/workspace/LLaMA-Factory/data
--template default
--finetuning_type lora
--lora_target q_proj,v_proj
--output_dir /root/workspace/LLaMA-Factory/output_chechpoint
--overwrite_cache
--overwrite_output_dir
--cutoff_len 1024
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 8
--lr_scheduler_type cosine
--logging_steps 10
--save_steps 100
--eval_steps 100
--evaluation_strategy steps
--load_best_model_at_end
--learning_rate 5e-5
--num_train_epochs 3.0
--max_samples 3000
--val_size 0.1
--quantization_bit 4
--plot_loss
--fp16

推理
CUDA_VISIBLE_DEVICES=5 python src/cli_demo.py \
--model_name_or_path /models/llama-7b  \
--adapter_name_or_path /root/workspace/LLaMA-Factory/output_chechpoint \
--template default \
--finetuning_type lora

Expected behavior

No response

System Info

[2024-03-26 16:52:14,808] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|tokenization_utils_base.py:2082] 2024-03-26 16:52:16,469 >> loading file tokenizer.model
[INFO|tokenization_utils_base.py:2082] 2024-03-26 16:52:16,469 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2082] 2024-03-26 16:52:16,469 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2082] 2024-03-26 16:52:16,469 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2082] 2024-03-26 16:52:16,469 >> loading file tokenizer.json
[WARNING|logging.py:329] 2024-03-26 16:52:16,469 >> You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in huggingface/transformers#24565
[INFO|configuration_utils.py:724] 2024-03-26 16:52:16,555 >> loading configuration file /models/llama-7b/config.json
[INFO|configuration_utils.py:789] 2024-03-26 16:52:16,556 >> Model config LlamaConfig {
"_name_or_path": "/models/llama-7b",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 0,
"eos_token_id": 1,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 11008,
"max_position_embeddings": 2048,
"max_sequence_length": 2048,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 32,
"pad_token_id": -1,
"pretraining_tp": 1,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 10000.0,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.39.1",
"use_cache": true,
"vocab_size": 32000
}

03/26/2024 16:52:16 - INFO - llmtuner.model.patcher - Using KV cache for faster generation.
[INFO|modeling_utils.py:3280] 2024-03-26 16:52:16,677 >> loading weights file /models/llama-7b/pytorch_model.bin.index.json
[INFO|modeling_utils.py:1417] 2024-03-26 16:52:16,698 >> Instantiating LlamaForCausalLM model under default dtype torch.float16.
[INFO|configuration_utils.py:928] 2024-03-26 16:52:16,699 >> Generate config GenerationConfig {
"bos_token_id": 0,
"eos_token_id": 1,
"pad_token_id": -1
}

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████| 33/33 [03:00<00:00, 5.46s/it]
[INFO|modeling_utils.py:4024] 2024-03-26 16:55:17,446 >> All model checkpoint weights were used when initializing LlamaForCausalLM.

[INFO|modeling_utils.py:4032] 2024-03-26 16:55:17,446 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /models/llama-7b.
If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training.
[INFO|configuration_utils.py:881] 2024-03-26 16:55:17,452 >> loading configuration file /models/llama-7b/generation_config.json
[INFO|configuration_utils.py:928] 2024-03-26 16:55:17,452 >> Generate config GenerationConfig {
"bos_token_id": 0,
"eos_token_id": 1,
"pad_token_id": 0
}

03/26/2024 16:55:18 - INFO - llmtuner.model.adapter - Fine-tuning method: LoRA
Traceback (most recent call last):
File "/root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/peft/config.py", line 197, in _get_peft_type
config_file = hf_hub_download(
File "/root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 111, in _inner_fn
validate_repo_id(arg_value)
File "/root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 159, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/root/workspace/LLaMA-Factory/output_chechpoint'. Use repo_type argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/workspace/LLaMA-Factory/src/cli_demo.py", line 49, in
main()
File "/root/workspace/LLaMA-Factory/src/cli_demo.py", line 15, in main
chat_model = ChatModel()
File "/root/workspace/LLaMA-Factory/src/llmtuner/chat/chat_model.py", line 23, in init
self.engine: "BaseEngine" = HuggingfaceEngine(model_args, data_args, finetuning_args, generating_args)
File "/root/workspace/LLaMA-Factory/src/llmtuner/chat/hf_engine.py", line 33, in init
self.model, self.tokenizer = load_model_and_tokenizer(
File "/root/workspace/LLaMA-Factory/src/llmtuner/model/loader.py", line 149, in load_model_and_tokenizer
model = load_model(tokenizer, model_args, finetuning_args, is_trainable, add_valuehead)
File "/root/workspace/LLaMA-Factory/src/llmtuner/model/loader.py", line 94, in load_model
model = init_adapter(model, model_args, finetuning_args, is_trainable)
File "/root/workspace/LLaMA-Factory/src/llmtuner/model/adapter.py", line 110, in init_adapter
model: "LoraModel" = PeftModel.from_pretrained(
File "/root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/peft/peft_model.py", line 328, in from_pretrained
PeftConfig._get_peft_type(
File "/root/miniconda3/envs/llama_factory/lib/python3.10/site-packages/peft/config.py", line 203, in _get_peft_type
raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'")
ValueError: Can't find 'adapter_config.json' at '/root/workspace/LLaMA-Factory/output_chechpoint'

没有生成adapter_config.json文件

Others

No response

[bgtii]

这个要怎么推理？

[hiyouga]

更新源码重新微调