如何使用8bit model做inference

[tonyaw]

您好，我想把codellama的model做8bit的quantization，然后再用它做inference。
使用参数如下:
{
"stage": "sft",
"model_name_or_path": "/workspace/model/CodeLlama-34b-Instruct-hf",
"do_train": false,
"do_predict": true,
"dataset": "test_no_answer",
"template": "llama2",
"finetuning_type": "lora",
"quantization_bit": 8,
"bf16": true,
"lora_target": "all",
"output_dir": "/workspace/34b_baseline",
"cutoff_len": 16384,
"per_device_eval_batch_size": 1,
"max_samples": 100,
"predict_with_generate": true
}

遇到error:
File "/workspace/task_entry.py", line 30, in training_task
run_exp(args_dict)
File "/workspace/src/llmtuner/tuner/tune.py", line 65, in run_exp
run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
File "/workspace/src/llmtuner/tuner/sft/workflow.py", line 49, in run_sft
trainer = CustomSeq2SeqTrainer(
File "/usr/local/lib/python3.9/dist-packages/transformers/trainer_seq2seq.py", line 56, in init
super().init(
File "/usr/local/lib/python3.9/dist-packages/transformers/trainer.py", line 412, in init
raise ValueError(

Message:

You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more details

User error.

请问应该如何解决?
谢谢！

[hiyouga]

安装 transformers==4.33.2

[tonyaw]

请问这个要求是完全等于4.33.2，还是
transformers>=4.33.2？

[tonyaw]

且LlamaFactory的code需要update么?
我现在用的是大概两周前的snapshot。
如果需要更新，请问有个PR告诉我哪些需要改么?

[hiyouga]

完全等于
你先试试

[tonyaw]

测试结果是:

• 完全等于，没有问题。

• 但是如果是大于等于，实际安装的是4.34.1，然后是同样的error。

这意味着以后不能再升级transformers了(暂时)?

[hiyouga]

是的

[tonyaw]

您好，
我下了最新的code，又遇到这个error，发现requirements.txt:
transformers>=4.31.0,<4.35.0
请问为何不是transformers==4.33.2呢？