[Feature] DeepSeek V3 optimization

[zhyncs]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Usage

User Guide for Existing System (Installation & Launch)

https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3

Please use the latest version v0.4.1.post3

Features

• Support CUDA Graph @HandH1998 @ispobock
• Use BF16 for bmm @zhyncs
• Improve the accuracy for FP8 @HandH1998 @zhyncs @ispobock
• Tuning FP8 GEMM @HandH1998 @zhyncs
• Replace moe_align_block_size @HandH1998 @zhyncs @BBuf
• FusedMoE tuning for H200 E=256,N=256,device_name=NVIDIA_H200,dtype=fp8_w8a8.json @BBuf
• TP+DP Attention @Ying1123
• Support nextn speculative decoding @merrymercy
• Support overlap scheduler with DP attention @merrymercy
• Fuse Sigmoid Gate moe_kernels.cu @ispobock
• FP8 GEMM CUTLASS implementation f8_blockwise_scaling_pr_branch @fsygd
• FP8 GEMM Composable Kernel implementation @HaiShaw
• Support Pipeline Parallelism @yizhang2077

Related resources

No response

[libratiger]

Very quick response !
I understand that the overlap scheduler is model-independent and is a general optimization that should be supported by default.
At least some special optimizations are needed?

[merrymercy]

The overlap scheduler is model-independent but has not been supported when using dp attention. We have a private branch for this and will upstream it soon.

[fengyang95]

Is the memory sufficient for an 8 gpus instance? This model size is too large.

[zhyncs]

Is the memory sufficient for an 8 gpus instance? This model size is too large.

671B works on H200 * 8 with FP8 (671 < 141 * 8)

[zhyncs]

Hi @fengyang95 You can also consider multi node.

If you do not have GPUs with large enough memory, please try multi-node tensor parallelism (help 1 help 2).

[zhyncs]

FYI Due to the tight schedule, SGLang v0.4.1 currently only provides preliminary support for DeepSeek V3. To make it run more cost-efficiently, we need to complete most of the optimizations mentioned above. If you are interested in any of the above optimizations, feel free to join the SGLang Slack for discussions or contribute a PR. We hope to complete these optimizations quickly and appreciate any discussion and contributions.

[zhyncs]

Update: SGLang v0.4.1.post1 supports CUDA Graph for DeepSeek V3, please use the latest version.

pip install "sglang[all]==0.4.1.post1" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer

[zhyncs]

Update: SGLang v0.4.1.post2 supports FP8 GEMM Tuning for DeepSeek V3, please use the latest version.

pip install "sglang[all]==0.4.1.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer

[zhyncs]

ref #2647

[CSEEduanyu]

plan to support mtp?

[zhyncs]

plan to support mtp?

It's on the roadmap and it's named nextn. We'll support it soon.

[lixiaolx]

@zhyncs @Ying1123 @merrymercy ,hello,
As you mentioned above, TP+DP,

TP+DP Attention @Ying1123

I have two questions, could you help me answer them?

1.Can we decouple TP and DP after this implementation? Can we configure the scenario where DP is not equal to TP?

2.Is there a detailed schedule for the mentioned above? Are there any related supporting design documents that can be shared?

[Mutinifni]

I had another question regarding DP attention. The sglang blog mentions that DP attention is effective because of the MLA has only 1 KV head, which causes unnecessary duplication of KV caches. DeepSeek-V3 MLA has more KV heads (16 attention, 128 KV), so do we still replicate KV caches if just using something like TP8 or TP16? I understand there might not be sufficient heads if the deployment is large.

+1 for shared design docs, if possible.

[pipul]

I had another question regarding DP attention. The sglang blog mentions that DP attention is effective because of the MLA has only 1 KV head, which causes unnecessary duplication of KV caches. DeepSeek-V3 MLA has more KV heads (16 attention, 128 KV), so do we still replicate KV caches if just using something like TP8 or TP16? I understand there might not be sufficient heads if the deployment is large.

+1 for shared design docs, if possible.

@zhyncs @Mutinifni

https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json
"num_key_value_heads": 128,
DeepSeek-V3 has 128 KV heads??

[min-xu-et]

Are there any data related to inference time batch size and token imbalance between experts? What's the total throughput like for a 8xH200 node?