Add a benchmark script for in-batch prefix caching

[merrymercy]

Changes

• Add a benchmark script bench_in_batch_prefix.py from @Ying1123
• Clean up the code of delay scheduling for in-batch prefix caching introduced in [Core] in batch prefix caching by delay scheduling #2442.

Some results

Please note that there are huge variances across runs.

• Results with the old default policy

Latency of test_batch_by_batch          : 1.9585 s
Latency of test_batch_by_batch_with_hint: 1.9020 s
Latency of test_send_all                : 3.7643 s

• Results with the reverted delay scheduling from [Core] in batch prefix caching by delay scheduling #2442.

Latency of test_batch_by_batch          : 1.9868 s
Latency of test_batch_by_batch_with_hint: 1.9520 s
Latency of test_send_all                : 3.0549 s

• The goal: the Latency of test_send_all should be very close to the Latency of test_batch_by_batch_with_hint (the oracle).

[libratiger]

I was intrigued by your benchmark results, particularly noting that test_send_all showed slower performance compared to other methods. To verify this, I conducted tests on my own server, but interestingly, I observed opposite results:

Latency of test_batch_by_batch          : 2.3765 s
Latency of test_batch_by_batch_with_hint: 2.4907 s
Latency of test_send_all                : 2.2191 s

The tests were run using the following commands:

python3 -m sglang.launch_server --model Qwen/Qwen2.5-3B-Instruct
# Changed the tokenizer to Qwen
python bench_in_batch_prefix.py

Environment details:

• SGLang version: 0.4.1
• Hardware: A100 GPU

I repeated these experiments multiple times and also tested with a smaller model, but the results consistently showed the same pattern: test_send_all performed faster than the other methods.

[libratiger]

attention_backend='flashinfer', sampling_backend='flashinfer',