[Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch

[liangan1]

Pytorch already support XPU device since 2.4 release and xpu is also supported in OpenAI Trition. So, it should works with the Trition attention backend in SGLang. In this PR, We add 'xpu' device into SGLang.

Blocked issue: The vllm is only compatible with 2.3 now and he binary whl support for torch xpu should be ready since torch-2.5(should be ready in the Oct/2024), so we should wait the vllm to be compatible with torch-2.5.

Status

• Both XPU & CUDA works with latency benchmark
  LLama-2-7b works for the latency benchmark.
  VLLM_TEST_COMPILE_NO_CUSTOM_OPS=1 python -m sglang.bench_latency --model-path ~/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-hf/snapshots/6fdf2e60f86ff2481f2241aaee459f85b5b0bbb9/ --device xxxx

• Both XPU & CUDA generate same outputs with launch_server
  python -m sglang.launch_server --model-path ~/models/llama7b/6fdf2e60f86ff2481f2241aaee459f85b5b0bbb9/ --port 30000 --device xxx

ToDO In other PRs:

Functionality

• Add the BKC to prepare the the xpu enabled pytorch and resolve the software combability with vllm when using the XPU enabled pytorch.
• Enable other benchmarks.
• Add UTs for the XPU device.

Performance

• Customized ops support.

[merrymercy]

@liangan1 Thanks for the contribution. Could you fix the unit tests https://github.com/sgl-project/sglang/tree/main/test?

[liangan1]

@liangan1 Thanks for the contribution. Could you fix the unit tests https://github.com/sgl-project/sglang/tree/main/test?

Sure. I will work on it and let you know when all UTs pass.

[merrymercy]

It is almost there! There are only a few remaining issues for multi-gpu test cases.

[merrymercy]

We will push some big refactor soon, starting from #1534. To prevent too many conflicts, it is better to merge this PR soon or split this PR into multiple smaller ones.

[liangan1]

We will push some big refactor soon, starting from #1534. To prevent too many conflicts, it is better to merge this PR soon or split this PR into multiple smaller ones.

Sorry, I don't have enough GPUs to reproduce this tensor parallel related UTs, do you have any comments abouts about this issues? According to the UT logs, the distributed backend and model init has been finished and the timeout occurs during the CUDAGraph initialization, but I don't need to change anything for this part.

[liangan1]

I took a look at the code and could not find the issue. Maybe you can split the PR into several smaller ones (e.g., change is_hip() to not flashinfer_is_available()) and we iterate on that together.

Now I also need to manually trigger the CI run for you because you have never contributed to this repo. If you have contributed one commit, the CI will be triggered for you automatically.

Spliting this PR into smaller ones.

• Firstly we will try to enable 'device' support in the runtime. Add device support #1607