Failed to build Llama-2-13b-hf engine with Lora adapter

[manarshehadeh]

Running with release 0.6.1.

hitting assertion error:
AssertionError: The value updated is not the same shape as the original. Updated: (32000, 5120), original: (0, 5120)

Build command:

Callstack:

[byshiue]

It looks the vocab size does not be passed in model successfully, you could check the vocab size in build.py before passing in model.

Do you use the checkpoint we use in document?

[manarshehadeh]

Indeed, the vocab size is overridden by LoraConfig vocab size, value is 0 since the adapter config file doesn't contain a value for vocab size.
Can you give more clarity over what you mean with checkpoint in document?

[byshiue]

Indeed, the vocab size is overridden by LoraConfig vocab size, value is 0 since the adapter config file doesn't contain a value for vocab size. Can you give more clarity over what you mean with checkpoint in document?

I mentioned

https://huggingface.co/meta-llama/Llama-2-13b-hf
https://huggingface.co/hfl/chinese-llama-2-lora-13b

For different checkpoint, we don't verifiy and it might not work because they have different settings.

You could try removing

args.vocab_size = lora_config.vocab_size

in llama/build.py.

[kaiyux]

The issue should be fixed in the latest main branch, please see #754.

Closing. Please feel free to comment if you have any question, thanks!