[Bug]: Function signature of two get_tokenizer used by benchmarks/benchmark_serving.py is not aligned

[cblmemo]

Your current environment

(not related)

Model Input Dumps

No response

🐛 Describe the bug

Two functions imported by this conditional import has different signature:

vllm/benchmarks/benchmark_serving.py

Lines 46 to 49 in d14e98d

	try:
	from vllm.transformers_utils.tokenizer import get_tokenizer
	except ImportError:
	from backend_request_func import get_tokenizer

vllm/vllm/transformers_utils/tokenizer.py

Lines 119 to 127 in d14e98d

	def get_tokenizer(
	tokenizer_name: Union[str, Path],
	*args,
	tokenizer_mode: str = "auto",
	trust_remote_code: bool = False,
	revision: Optional[str] = None,
	download_dir: Optional[str] = None,
	**kwargs,
	) -> AnyTokenizer:

vllm/benchmarks/backend_request_func.py

Lines 419 to 421 in d14e98d

	def get_tokenizer(
	pretrained_model_name_or_path: str, trust_remote_code: bool
	) -> Union[PreTrainedTokenizer, PreTrainedTokenizerFast]:

Which cause this error when using benchmark:

(task, pid=5102) Traceback (most recent call last):
(task, pid=5102)   File "/home/gcpuser/sky_workdir/vllm/benchmarks/benchmark_serving.py", line 1226, in <module>
(task, pid=5102)     main(args)
(task, pid=5102)   File "/home/gcpuser/sky_workdir/vllm/benchmarks/benchmark_serving.py", line 793, in main
(task, pid=5102)     tokenizer = get_tokenizer(tokenizer_id,
(task, pid=5102) TypeError: get_tokenizer() got an unexpected keyword argument 'tokenizer_mode'

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Can you fix it?

[e1ijah1]

Can you fix it?

I've also noticed this bug. I'd be happy to help fix it if you'd like.

[robertgshaw2-redhat]

That would be great!

Can you fix it?

I've also noticed this bug. I'd be happy to help fix it if you'd like.

That would be great! Should be a quick fix

[e1ijah1]

That would be great!

Can you fix it?

I've also noticed this bug. I'd be happy to help fix it if you'd like.

That would be great! Should be a quick fix

I've submitted the fix. Please take a look 🙏