[0.6.1] llama 13b gptq the value update is not the same shape as the original. updated: (2560, 3840), original (5120, 3840)

[Slyne]

Looks like it ignores the mapping.tp_rank.

[juney-nvidia]

@Slyne

Can you share the full command sequences to reproduce the issue?

Thanks
June

[Slyne]

@Slyne

Can you share the full command sequences to reproduce the issue?

Thanks June

Step1. Get llama2 13B from meta
Step2. Convert llama2 13B with huggingface script: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py
Step3. Follow GPTQ instructions in llama example under TRTLLM to get llama-13b-4bit-gs128.safetensors
Step4. Run the below command

python build.py --model_dir /llama_hf/13B/ \
                --quant_ckpt_path ./GPTQ-for-LLaMa/llama-13b-4bit-gs128.safetensors \
                --dtype float16 \
                --remove_input_padding \
                --use_gpt_attention_plugin float16 \
                --enable_context_fmha \
                --use_gemm_plugin float16 \
                --use_inflight_batching \
                --paged_kv_cache \
                --use_rmsnorm_plugin \
                --use_weight_only \
                --weight_only_precision int4_gptq \
                --per_group \
                --world_size 2 \
                --tp_size 2 \
                --max-input-len 1900 \
                --max-output-len 64 \
                --output_dir ./tmp/llama/13B/trt_engines/int4_GPTQ/2-gpu/

[Barry-Delaney]

@Slyne thanks for the feedback. We have fixed this internally, and will update it in the future main branch.

[juney-nvidia]

@Slyne

Close it since it has already been fixed in the main branch. In case there are still things missing, pls open a new issue to track it.

Thanks
June