the normal generation throughout reference

[ltm920716]

hi，
I deploy the v3 model with vllm on 8*H200（tp=8），and the generation throughout is round 10 tokens/s，I thinks this is some slow，so could you give me a reference about the normal generation throughout，or some method to improve the throughout，thanks！

[GeeeekExplorer]

In fact, the demo is only used to show the model structure. It is mainly limited by CPU and you can observe that the kernel launching is very slow using torch profiler. For faster speed you can try sglang or vllm.

[ltm920716]

In fact, the demo is only used to show the model structure. It is mainly limited by CPU and you can observe that the kernel launching is very slow using torch profiler. For faster speed you can try sglang or vllm.

actually，I do deploy the model with vllm/vllm-openai:v0.6.6，so I want to know how to improve the throughout

[GeeeekExplorer]

I think cudagraph would be helpful if supported.

[mowentian]

Please refer to the work vllm DeepSeek-V3 Enhancements vllm-project/vllm#11539