How can I use the Lora Adapter for a model with Vocab size 40960?

[hrson-1203]

The error occurs when I call the LLMEngine object.
The error below appears.

ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 33024

Does the method using the Lora Adapter not apply to models that expand the vocab or have a vocab size larger than 32000?

[newportchen]

please support bigger vocab size like yi_6b: 64000 @Yard1

[AlphaINF]

same issue, I deployed a LLaMA structure LLLM but the vocab_size had been expanded to 130,000

[r4dm]

same problem, please add support for vocab_size over 32000. in my case need vocab_size 60256

[AlphaINF]

I found a solution which inspired by Gemma
you can find the file: vllm/vllm/model_executor/models/llama.py, then find the definition of LlamaForCausalLM (in line 278)

that's the original support lora modules

    # LoRA specific attributes
    supported_lora_modules = [
        "qkv_proj",
        "o_proj",
        "gate_up_proj",
        "down_proj",
        "embed_tokens",
        "lm_head",
    ]
    embedding_modules = {
        "embed_tokens": "input_embeddings",
        "lm_head": "output_embeddings",
    }
    embedding_padding_modules = ["lm_head"]

just remove some modules which may cause errors and finally saving like this:

    # LoRA specific attributes
    supported_lora_modules = [
        "qkv_proj",
        "o_proj",
        "gate_up_proj",
        "down_proj",
    ]
    embedding_modules = {}
    embedding_padding_modules = []

[Xingxiangrui]

+1 for this, Please add more vocab size support.

[ashgold]

Why doesn't the lora adapter load based on the vocabulary size of the origin model, even though the target_modules in adapter_config.json doesn't have embed_tokens and lm_head?

[flexorRegev]

Was also trying to run Yi with the same problem..
@Yard1 can you elaborate on what's needed to support? I'd be glad to work on this PR

[Yard1]

We'd need to modify the kernel/write a new LoRA kernel to support such a large vocabulary size. The current Punica ones cannot do that.

[flexorRegev]

@Yard1 How does the Gemma avoid this? it also has a huge vocab_size

[Yard1]

We just don't allow the embeddings layer to be modified on gemma

[newportchen]

We just don't allow the embeddings layer to be modified on gemma

The Yi_6B has 64000 vocab_size, when i modify the llama model source just like gemma，it's work fine。

but the Yi_6B_200K always throw exception:

The Yi_6B_200K has a different max_position_embeddings:200000

@Yard1

[PenutChen]

+1. Need support for a larger vocabulary size. In addition to vocabulary size, there is also a use case like LLaMA Pro block expansion fine-tuning with LoRA, which may not have LoRA weights for all layers. For example, only layers 5, 10, 15, and 20 might have corresponding LoRA weights.

[Yard1]

It should already be possible to use adapters which do not modify every layer - if that is not the case, it is a bug and should be fixed.

[flexorRegev]

Also made it work with a gemma-like adaptation to Yi and it works :)

[qZhang88]

lora finetune for Llama-3 has the same problem, however same code finetune Qwen1.5-7B-Chat, with vocab size 151643, has no such problem, why?

[Yard1]

@qZhang88 Llama 3 should be supported in latest release.

[piercefreeman]

@Yard1 I've tried llama3 on v0.4.0.post1, but this issue is still present when initializing the engine with lora adapters. The latest main code seems to get further in the initialization process (ie. no explicit crash about vocab size) but seems to stall out later in the pipeline - haven't gotten a chance to debug that yet.

But to confirm - llama3 with adapters is supposed to work on 0.4.0.post1, or were you referring to the 0.4.1 prerelease?

[Yard1]

It should be 0.4.1 - this is the commit: 1e96c33

[Techinix]

@Yard1 how can we upgrade to this release ??

[piercefreeman]

@Techinix You can either manually install the wheel from the Release page or build yourself locally from the tagged git version. On my setup, local building takes around 15-30min.

[SuperBruceJia]

For the LLaMA 3 model:

Please use the following codes for your tokenizer:

tokenizer.pad_token = tokenizer.eos_token

In this way, the vocabulary size is still 128256 .

However, if you use

tokenizer.add_special_tokens(
    {'pad_token': '[PAD]'}
)
# Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size.
model.resize_token_embeddings(len(tokenizer))

The vocabulary size is increased to 128768 , which leads to this ValueError: When using LoRA, vocab size must be 32000 > vocab_size < 128512 error.

For now, I don't know the reason. If you know the reason, please leave a message. Thank you very much in advance!

Shuyue
May 11th, 2024

[schoennenbeck]

Wouldn't it make more sense to check which layers the LoRA-adapter actually applies to? Most LoRA-frameworks don't even touch the embedding layer by default.

[AlphaINF]

hello. In some multilingual models(like qwen2 or OpenBuddy multilingual series model), the vocab_size may exceed 128512, their size are bigger than 128k but usually not larger than 200k, can you extend the vocab_size to 200k?

@qZhang88 Llama 3 should be supported in latest release.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!