使用Llama3-8B-Chinese-Chat做扩展上下文长度的时候遇到的问题

[qq610396646]

Reminder

• I have read the README and searched the existing issues.

System Info

[INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[WARNING] using untested triton version (2.3.0), only 1.0.0 is known to be compatible

• llamafactory version: 0.8.2.dev0
• Platform: Linux-5.19.0-50-generic-x86_64-with-glibc2.35
• Python version: 3.10.14
• PyTorch version: 2.3.0+cu121 (GPU)
• Transformers version: 4.41.2
• Datasets version: 2.18.0
• Accelerate version: 0.31.0
• PEFT version: 0.11.1
• TRL version: 0.9.4
• GPU type: NVIDIA GeForce RTX 4060 Ti
• DeepSpeed version: 0.14.3
• Bitsandbytes version: 0.43.1
• vLLM version: 0.5.0.post1

Reproduction

export CUDA_VISIBLE_DEVICES=0,1
llamafactory-cli train \
    --stage sft \
    --do_train True \
    --model_name_or_path LLM-Research/Llama3-8B-Chinese-Chat \
    --finetuning_type lora \
    --template llama3 \
    --flash_attn auto \
    --quantization_bit 8 \
    --dataset_dir data \
    --dataset alpaca_gpt4_zh \
    --cutoff_len 32768 \
    --rope_scaling linear \
    --learning_rate 5e-05 \
    --num_train_epochs 1.0 \
    --max_samples 300 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 8 \
    --lr_scheduler_type cosine \
    --max_grad_norm 1.0 \
    --logging_steps 5 \
    --save_steps 100 \
    --warmup_steps 0 \
    --optim adamw_torch \
    --packing False \
    --report_to none \
    --output_dir saves/LLaMA3-8B-Chinese-Chat/lora/llama3_lora \
    --fp16 True \
    --lora_rank 8 \
    --lora_alpha 16 \
    --lora_dropout 0 \
    --lora_target q_proj,v_proj \
    --plot_loss True

训练的脚本
合并lora的参数配置文件

model_name_or_path: LLM-Research/Llama3-8B-Chinese-Chat

adapter_name_or_path: saves/LLaMA3-8B-Chinese-Chat/lora/llama3_lora

template: llama3

finetuning_type: lora

rope_scaling: linear

cutoff_len: 32768

export_dir: models/llama3_lora_sft

export_size: 2

export_device: cpu

export_legacy_format: false

Expected behavior

我按照 #3370 这个issue合并了lora。

进行微调的时候scaling factor是4.0

但是合并lora的时候 scaling factor是2.0

合并lora导出模型的配置，我认为。

max_position_embeddings 应该是32768。

scaling factor是4.0。

产生差异的原因是什么？

Others

No response

[hiyouga]

fixed

[qq610396646]

thx。已经解决了，