[Feature] support long context eval and benchmark

[zhyncs]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

as titled

Related resources

No response

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

[zhyncs]

ref https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/eval_long_context.py

[EstherBear]

I think I can try to help with this issues. Could you help assign it to me?

[zhyncs]

@EstherBear How is it going?

[EstherBear]

@EstherBear How is it going?

I don't have much update this week since I'm on a trip. How urgent is this issue? Is there a set timeline we should follow?

[zhyncs]

@EstherBear May we finish this week? This demand has always existed, but there hasn't been enough manpower to address it. I believe it's important and hope it can be completed soon, ideally within the next few days. Thank you!

[EstherBear]

@EstherBear May we finish this week? This demand has always existed, but there hasn't been enough manpower to address it. I believe it's important and hope it can be completed soon, ideally within the next few days. Thank you!

I see. I will try to make a PR in next few days.