[Model] Whisper model implementation

[aurickq]

Add Whisper model implementation. Based on #5964 but heavily optimized and integrated with newer encoder/decoder support in vLLM. Currently only supports audio up to 30s. No VAD model, no beam search, no phoneme model, forced alignment, etc.

Performance overall looks OK, especially at larger batch sizes

FIX #180

[mgoin]

LGTM and verified the example locally. Last part is adding it to the model docs, feel free to do in a followup. cc @DarkLight1337 @Isotr0py for final pass

[DarkLight1337]

Can we remove the eos token in the input processor? But maybe we can improve this later via the merged multi-modal processor since I assume HF can handle this automatically.

[Isotr0py]

Seems that <|startoftranscript|> is also a special token which will be added by tokenizer by default:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("openai/whisper-small")
tokenizer = processor.tokenizer
encode = tokenizer("<|startoftranscript|>")
print(tokenizer.decode(encode["input_ids"]))

It outputs '<|startoftranscript|><|notimestamps|><|startoftranscript|><|endoftext|>', which means that we need to remove a "<|startoftranscript|>" to get the original prompt as well.

Perhaps we need to construct user prompts with non special tokens.

[aurickq]

Yes, there are other special tokens that are added as well. There are also some other challenges

1. Some tokens are not reliably mapped to the same ids. E.g. <|notimestamps|> is 50363 for whisper-small but 50364 for whisper-large-v3. This makes it awkward to try to detect these special tokens from inside the input processor.
2. From in the input processor, not sure how to reliably differentiate between the two cases
   a. User passed in text prompt which is tokenized in vLLM. We need to strip special tokens.
   b. User passed in tokens prompt_token_ids which should not be stripped.

Basically, I'm not sure how to reliably remove special tokens after it's already encoded. It seems error-prone and brittle to third-party behavior in huggingface. I understand that having the tokenizer not add special tokens in the first place involves more changes inside vLLM, but IMO it's the most reliable solution that's more robust to future changes in vLLM and huggingface.

[DarkLight1337]

Let's merge this first then.

[DarkLight1337]

Can you merge from main to fix the CI failures?

[DarkLight1337]

Thanks again for your hard work!

[ArmykOliva]

Thank you for your work. Are Rest API endpoints reachable as well? Could you add an example for that please?

[aryan1165]

Just a quick question, Is anyone implementing VAD, to support chunking and accepting audios of len more than 30s. Also how continuous batching can be supported?

[Temirulan]

Thanks for great news! Few questions here:

• Is word timestamping working using this upload?
• Does this support inflight batching?

[aurickq]

Thank you for your work. Are Rest API endpoints reachable as well? Could you add an example for that please?

I have not tested this, so not sure.

Just a quick question, Is anyone implementing VAD, to support chunking and accepting audios of len more than 30s. Also how continuous batching can be supported?

I don't have plans to implement VAD/long-form. This can also be done outside of vLLM as an outer-loop.

Thanks for great news! Few questions here:

• Is word timestamping working using this upload?

It seems to be working for me, but you need to suppress the <|notimestamps|> token from being generated with llm.generate(bad_words=["<|notimestamps|>"])

• Does this support inflight batching?

Chunked prefill is not supported if that's what you mean.

[aryan-sarvam]

Hey, I tried to use whisper in online way, using vllm.entrypoints.openai api server.py but, we can't hit Completions API (Can't send multimodal data), as mentioned in the Docs (generate and Completions API are similar), and chat_utils.py doesnt have whisper support.

Is someone working on that part. I tried some rough solutions but couldn't get to execute properly.

I will also open an issue regarding this, if this is not yet on the pipeline someone building.
Thanks

[aryan-sarvam]

Thank you for your work. Are Rest API endpoints reachable as well? Could you add an example for that please?

Hi, I dont think they are, Completions one is bound to support this but it doesn't have multi modal support, and chat doesn't have this support. Let me know if you came up with a solution?

[aryan-sarvam]

Add Whisper model implementation. Based on #5964 but heavily optimized and integrated with newer encoder/decoder support in vLLM. Currently only supports audio up to 30s. No VAD model, no beam search, no phoneme model, forced alignment, etc.

Performance overall looks OK, especially at larger batch sizes

FIX #180

Hey, I was doing benchmarking on H100, of audio len around 18s. I found out that RTFx maxes out at a really low value around 120 at batch size 96. Can you share the benchmarking parameters or flags so that we can reproduce these results.

[aurickq]

Add Whisper model implementation. Based on #5964 but heavily optimized and integrated with newer encoder/decoder support in vLLM. Currently only supports audio up to 30s. No VAD model, no beam search, no phoneme model, forced alignment, etc.
Performance overall looks OK, especially at larger batch sizes

FIX #180

Hey, I was doing benchmarking on H100, of audio len around 18s. I found out that RTFx maxes out at a really low value around 120 at batch size 96. Can you share the benchmarking parameters or flags so that we can reproduce these results.

Hi, I am running the example python examples/offline_inference/whisper.py which measures the time to run ~2K audio clips:

Duration: 57.445557594299316
RPS: 35.65114668158841

[aryan-sarvam]

Duration: 57.445557594299316
RPS: 35.65114668158841

Hi @aurickq , I build the project from source and ran the same file. On h100, with exact same file, results are :-

Duration: 309.14796566963196
RPS: 6.624659475160758

Can you tell if there are some other things to do? If possible Can you connect with me, sometime for proper setup.

Thanks.