ExLlamaV2: exl2 support

[pabl-o-ce]

If is possible ExLlamaV2 is a very fast and good library to Run LLM

ExLlamaV2 Repo

[mgoin]

@pabl-o-ce What specifically would you like to see? vLLM already integrates kernels from exllamav2 - see things such as the GPTQ kernels

vllm/csrc/quantization/gptq/qdq_4.cuh

Line 2 in 05af6da

Copied from https://github.com/turboderp/exllamav2

[pabl-o-ce]

Hi @mgoin thanks for the response.

So is possible to use exl2 format on vllm? or is using gptq format
sorry if this is a very n00b question

[tobiajung]

Hi,
seems like 0.3.3 has vllm support, but I'm not able to get a model up and running. I use the docker environment with the following args: --model LoneStriker/CodeFuse-DeepSeek-33B-4.0bpw-h6-exl2 --gpu-memory-utilization 0.65 --max-model-len 2048
But seems like vllm tries to allocate much more memory than the given 0.65 (of 48GB) and results in an error.

Is exl2 proper supported? How to start the docker container correctly to operate on exl2 models?

[wxupjack]

Hi @mgoin, I think this feature submitted by @chu-tianxiang in #2330 and #916 just utilize the shuffle and dequant functions from exllamav2 repo for GPTQ. But not means vllm(main branch) has been compatible with the dynamic precision exl2 format.

So it still don't support exl2 properly. @tobiajung

[sapountzis]

We are also interested in exl2 dynamic precision format. What steps would be needed to support it?

[nkeilar]

Support for this would allow dbrx exl2 support for the dbrx model quantized weights. Allowing inference of the model on a dual 24 gpu system.

[zminer123]

Hello there! Has any more thought/attention been given to the idea of exl2 support? The newest derivatives of llama3 (such as dolphin 70b) utilize it and it seems no one else is quantizing it to AWQ or GPTQ. I love vLLM regardless! Thank you guys for all the work you put in.

[saucebing]

Hi, we are also interested in EXL2 format, which is quite flexible and fast. As for flexibility, you can use 3.2, 4.5, 8.5 bpw (bit per weight) to quantize a model. And the inference speed of EXL2 is much faster than GPTQ in 8-bit precision.

[houmie]

Fully agree. Supporting exl2 would be perfect!

[belladoreai]

I would also love to see exl2 support

[mku-wedoai]

I would love to see EXL2 support in vLLM!

[DenisSergeevitch]

exl2 is needed feature 100%

[sparsh35]

I support it too

[chopin1998]

any update?
really hope to get perfect support for exl2 format ASAP.

[meditans]

Also voicing my support!

[kulievvitaly]

VLLM is great. Like to see exlv2 support too!

[Respaired]

yeah this would be great to have

[paulb-seldon]

Would love to see this!

[rjmehta1993]

+1. EXl quants is unbeatable

[fablerq]

Is there any chance to see exl2? 👀

[DaBossCoda]

can we get this? no one making awq and gtpq quants anymore :(

[javAlborz]

exl2 would be nice 😃

[alkeryn]

@javAlborz it would !

[SlapDrone]

+1!

[alkeryn]

vllm's cli is my favorite so far because it just works, also the api is better than tabby.
but god, exl2 is better than awq.

[Originalimoc]

+1. Most new models are in GGUF and EXL2.

[drexample]

Give exl2 support pls

[rsxdalv]

Although I am waiting for exl2 support myself, the amount of +1 messages really don't help.
If you truly wish to make a +1, state reasons so that the developers or forkers have a real incentive.
Yes, for inactive issues a +1 might amplify your voice and is better than nothing, but a stream of +1s really can be worse than nothing. The probability that devs have unsubscribed from this issue is fairly high.

[JohnConnor123]

we are waiting for exl2 support!