[Feature]: vAttention

[nivibilla]

🚀 The feature, motivation and pitch

Claim major improvements over vllm. Unfortunately no code only the paper.

arxiv.org/abs/2405.04437

Alternatives

No response

Additional context

No response

[ramyaprabhu-alt]

Hi, I'm one of the authors of this paper. Thank you for your interest in our work! We plan to release the code soon, hopefully in a few weeks.

[Jeffwan]

@ramyaprabhu-alt Just curious the code release would be a separate project or a PR against vLLM? I think it's a PR, right?

[ramyaprabhu-alt]

Our initial release will be as a separate project on a slightly older version of vLLM. But soon after, we can also raise a PR against vLLM-latest.

[apanwariisc]

Glad to share the source code of vAttention. Please check it out here: https://github.com/microsoft/vattention

[arunpatala]

is this still on VLLM roadmap to integrate? Please let us know.

Thanks

[ramyaprabhu-alt]

is this still on VLLM roadmap to integrate? Please let us know.

Thanks

If there is still interest from vLLM community, we will be happy to contribute!

[raingrey]

We(many people with no name) are eagerly awaiting the release of this feature。。。so hungry so desire。。。Please！Help！