Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[V1] Support audio language models on V1 #11733

Merged
merged 7 commits into from
Jan 7, 2025
Merged

[V1] Support audio language models on V1 #11733

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3d60be0
qwen2 audio
ywang96 Jan 4, 2025
0a1d1e2
ultravox
ywang96 Jan 4, 2025
ed30132
use envs instead
ywang96 Jan 5, 2025
4a6d6a7
Merge branch 'vllm-project:main' into v1-audio
ywang96 Jan 6, 2025
b33fc30
fix ultravox
ywang96 Jan 7, 2025
5509132
update hf processor instead
ywang96 Jan 7, 2025
2f93c1b
remove unused attribute
ywang96 Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions vllm/model_executor/models/ultravox.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
merge_multimodal_embeddings,
merge_multimodal_embeddings_from_map)

_AUDIO_PLACEHOLDER_OVERRIDE = "<|reserved_special_token_0|>"
_AUDIO_PLACEHOLDER_TOKEN = 128002
_AUDIO_TOKENS_PER_SECOND = 6.25

Expand Down Expand Up @@ -200,8 +201,12 @@ def _get_prompt_replacements(
hf_processor_mm_kwargs: Mapping[str, Any],
out_mm_kwargs: MultiModalKwargs,
) -> list[PromptReplacement]:
hf_processor = self._get_hf_processor(**hf_processor_mm_kwargs)
placeholder = hf_processor.audio_token_replacement # type: ignore

# NOTE: Ultravox processing definition uses '<|eot_id|>' as the
# placeholder that will cause confusion with the actual end of turn
# token, thus we override placeholder with a reserved special
# token.
placeholder = _AUDIO_PLACEHOLDER_OVERRIDE
ywang96 marked this conversation as resolved.
Show resolved Hide resolved

def get_replacement_ultravox(item_idx: int):
audio_token_len = out_mm_kwargs["audio_token_len"][item_idx]
Expand Down Expand Up @@ -348,6 +353,7 @@ def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
self.multi_modal_config = multimodal_config
assert self.multi_modal_config

self.audio_token_id = config.audio_token_index
self.secondary_weights = []
self.audio_tower = ModifiedWhisperEncoder(config.audio_config)
if config.audio_model_id is not None:
Expand Down
Loading