vLLM support

[MichaelJayW]

No description provided.

[Data-drone]

Are there any updates on this one?

[louisoutin]

+1

[ruidongtd]

+1

[insist93]

+1

[leonardxie]

+1

[TexasRangers86]

+1

[Lvjinhong]

So, could you provide advice so that I can make custom modifications on vLLM myself (llama2 70b)?

[RonanKMcGovern]

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

[TexasRangers86]

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

hello，how can I pass medusa model and base model args when I use medusa on tgi.

[RonanKMcGovern]

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

hello，how can I pass medusa model and base model args when I use medusa on tgi.

Just pass the medusa model repo (as you would with any other model) and then add on --speculate 2

You can try this template: https://runpod.io/gsc?template=2xpg09eenv&ref=jmfkcdio

[TexasRangers86]

Thanks a lot !!!!

[chuangzhidan]

how to use medusa based on vllm or sglang?