Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for Initial Prompt and Hotwords in WhisperPipeline (similar to faster-whisper) #1150

Closed
HighBridgeDragon opened this issue Nov 6, 2024 · 3 comments · Fixed by #1378
Closed

Support for Initial Prompt and Hotwords in WhisperPipeline (similar to faster-whisper) #1150

HighBridgeDragon opened this issue Nov 6, 2024 · 3 comments · Fixed by #1378
Assignees
@as-suvorov
Labels
category: whisper Whisper pipeline

Comments

@HighBridgeDragon
Copy link

HighBridgeDragon commented Nov 6, 2024 •
edited
Loading

Hello OpenVINO team,

I'm currently using the WhisperPipeline in openvino.genai and would like to request a feature similar to that of faster-whisper (CTranslate2), where users can provide an initial_prompt or specify hotwords to influence the transcription process.

This feature would be particularly useful in cases where we want to improve transcription accuracy for specific terms or in domain-specific contexts.

Feature Request:

  • Add support for initial_prompt in the WhisperPipeline.
  • Add support for hotwords or custom vocabulary injection, allowing users to specify words/phrases that are prioritized during transcription.

Is there any existing workaround or plan to incorporate these features in openvino.genai?

Thank you for considering this feature request!
@as-suvorov
Copy link
Contributor

Hello, thanks for a request! I'll review it.

@HighBridgeDragon HighBridgeDragon mentioned this issue Nov 7, 2024
Open
@andrei-kochin andrei-kochin added the category: whisper Whisper pipeline label Nov 8, 2024
@as-suvorov as-suvorov mentioned this issue Dec 13, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 19, 2024
@as-suvorov
Whisper pipeline: implement 'initial_prompt' and 'hotwords' parameters (
0be7b3d 
#1378)

Adds:
* `initial_prompt` parameter ([faster_whisper
reference](https://github.com/SYSTRAN/faster-whisper/blob/203dddb047fd2c3ed2a520fe1416467a527e0f37/faster_whisper/transcribe.py#L732))
- injects initial prompt tokens as a previous transcription into the
first processing window
* `hotwords` parameter ([faster_whisper
reference](https://github.com/SYSTRAN/faster-whisper/blob/203dddb047fd2c3ed2a520fe1416467a527e0f37/faster_whisper/transcribe.py#L768))
- injects hotwords tokens as a previous transcription into the all
processing windows
* Whisper pipeline usage notes in samples

Closes #1150
Ticket: 156888
ScottZhang812 pushed a commit to ScottZhang812/_openvino.genai that referenced this issue Dec 23, 2024
@as-suvorov
Whisper pipeline: implement 'initial_prompt' and 'hotwords' parameter…
4a5b899 
…s (#1378)

Adds:
* `initial_prompt` parameter ([faster_whisper
reference](https://github.com/SYSTRAN/faster-whisper/blob/203dddb047fd2c3ed2a520fe1416467a527e0f37/faster_whisper/transcribe.py#L732))
- injects initial prompt tokens as a previous transcription into the
first processing window
* `hotwords` parameter ([faster_whisper
reference](https://github.com/SYSTRAN/faster-whisper/blob/203dddb047fd2c3ed2a520fe1416467a527e0f37/faster_whisper/transcribe.py#L768))
- injects hotwords tokens as a previous transcription into the all
processing windows
* Whisper pipeline usage notes in samples

Closes openvinotoolkit/openvino.genai#1150
Ticket: 156888
@HighBridgeDragon
Copy link
Author

HighBridgeDragon commented Dec 26, 2024 •
edited
Loading

@as-suvorov

Thank you so much for resolving this issue! I deeply respect and appreciate your effort in resolving this.

I’m looking forward to the release of the prebuilt binary that includes your changes.

Additionally, I’m hopeful for the resolution of issue #1164 as well.

@as-suvorov
Copy link
Contributor

@HighBridgeDragon

You could already try this changes using nightly package, installation instructions: https://docs.openvino.ai/2024/get-started/install-openvino.html?PACKAGE=OPENVINO_GENAI&VERSION=NIGHTLY&OP_SYSTEM=LINUX&DISTRIBUTION=PIP
Your feedback will be highly appreciated.

I will give you an update when I get more news regarding Beam Search for the Whisper pipeline.
Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@as-suvorov as-suvorov

Labels
category: whisper Whisper pipeline
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Whisper pipeline: implement 'initial_prompt' and 'hotwords' parameters as-suvorov/openvino.genai
3 participants
@HighBridgeDragon @as-suvorov @andrei-kochin