[Model] Initialize support for Deepseek-VL2 models

[Isotr0py]

FIX #11236

• Initialize support for deepseek-vl2 series models
• Note that deepseek-ai/deepseek-vl2-tiny is not supported yet because it doesn't use MLA attention.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[csdY123]

@Isotr0py
You're awesome! You've done an amazing job.
You just need to deal with this bug a little bit to succeed，I have successfully run
[rank0]: File "vllm/vllm/model_executor/models/deepseek_v3.py", line 601, in load_weights
[rank0]: if self.config.num_nextn_predict_layers > 0:
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/configuration_utils.py", line 205, in getattribute
[rank0]: return super().getattribute(key)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'DeepseekV2Config' object has no attribute 'num_nextn_predict_layers'

[Isotr0py]

@csdY123 Added a check for num_nextn_predict_layers's existence before self.config.num_nextn_predict_layers, so the model should be able to load.

(Don't have device to test the full Deepseek-VL2 model right now, so your feedback is very valuable!) :)

[Isotr0py]

The DeepSeek-V3 based deepseek-vl2 model should also work now.

Outputs

$ python examples/offline_inference/offline_inference_vision_language.py -m deepseek_vl_v2
INFO 01-10 15:08:08 __init__.py:179] Automatically detected platform cuda.
INFO 01-10 15:08:10 config.py:285] Overriding HF config with {'architectures': ['DeepseekVLV2ForCausalLM']}
INFO 01-10 15:08:17 config.py:516] This model supports multiple tasks: {'generate', 'reward', 'score', 'classify', 'embed'}. Defaulting to 'generate'.
INFO 01-10 15:08:17 config.py:1022] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
INFO 01-10 15:08:17 llm_engine.py:234] Initializing an LLM engine (v0.1.dev3959+g8d9b672) with config: model='deepseek-ai/deepseek-vl2', speculative_config=None, tokenizer='deepseek-ai/deepseek-vl2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=fp8, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=deepseek-ai/deepseek-vl2, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[2,1],"max_capture_size":2}, use_cached_outputs=False, 
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message.
INFO 01-10 15:08:19 cuda.py:176] Cannot use FlashAttention-2 backend for FP8 KV cache.
WARNING 01-10 15:08:19 cuda.py:178] Please use FlashInfer backend with FP8 KV Cache for better performance by setting environment variable  VLLM_ATTENTION_BACKEND=FLASHINFER
INFO 01-10 15:08:19 cuda.py:213] Using XFormers backend.
INFO 01-10 15:08:27 model_runner.py:1094] Starting to load model deepseek-ai/deepseek-vl2...
INFO 01-10 15:08:36 weight_utils.py:253] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  12% Completed | 1/8 [00:01<00:07,  1.13s/it]
Loading safetensors checkpoint shards:  25% Completed | 2/8 [00:02<00:07,  1.32s/it]
Loading safetensors checkpoint shards:  38% Completed | 3/8 [00:04<00:06,  1.37s/it]
Loading safetensors checkpoint shards:  50% Completed | 4/8 [00:04<00:04,  1.07s/it]
Loading safetensors checkpoint shards:  62% Completed | 5/8 [00:05<00:03,  1.13s/it]
Loading safetensors checkpoint shards:  75% Completed | 6/8 [00:07<00:02,  1.25s/it]
Loading safetensors checkpoint shards:  88% Completed | 7/8 [00:08<00:01,  1.32s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:09<00:00,  1.10s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:09<00:00,  1.18s/it]

INFO 01-10 15:08:46 model_runner.py:1099] Loading model weights took 51.2323 GB
WARNING 01-10 15:08:46 model_runner.py:1162] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results!
Python version is above 3.10, patching the collections module.
Some kwargs in processor config are unused and will not have any effect: image_std, sft_format, downsample_ratio, normalize, candidate_resolutions, patch_size, image_token, add_special_token, ignore_id, image_mean, mask_prompt, pad_token. 
Add pad token = ['<｜▁pad▁｜>'] to the tokenizer
<｜▁pad▁｜>:2
Add image token = ['<image>'] to the tokenizer
<image>:128815
Add grounding-related tokens = ['<|ref|>', '<|/ref|>', '<|det|>', '<|/det|>', '<|grounding|>'] to the tokenizer with input_ids
<|ref|>:128816
<|/ref|>:128817
<|det|>:128818
<|/det|>:128819
<|grounding|>:128820
Add chat tokens = ['<|User|>', '<|Assistant|>'] to the tokenizer with input_ids
<|User|>:128821
<|Assistant|>:128822

Some kwargs in processor config are unused and will not have any effect: image_std, sft_format, downsample_ratio, normalize, candidate_resolutions, patch_size, image_token, add_special_token, ignore_id, image_mean, mask_prompt, pad_token. 
Add pad token = ['<｜▁pad▁｜>'] to the tokenizer
<｜▁pad▁｜>:2
Add image token = ['<image>'] to the tokenizer
<image>:128815
Add grounding-related tokens = ['<|ref|>', '<|/ref|>', '<|det|>', '<|/det|>', '<|grounding|>'] to the tokenizer with input_ids
<|ref|>:128816
<|/ref|>:128817
<|det|>:128818
<|/det|>:128819
<|grounding|>:128820
Add chat tokens = ['<|User|>', '<|Assistant|>'] to the tokenizer with input_ids
<|User|>:128821
<|Assistant|>:128822

You're using a CachedLlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.

INFO 01-10 15:08:58 worker.py:241] Memory profiling takes 12.19 seconds
INFO 01-10 15:08:58 worker.py:241] the current vLLM instance can use total_gpu_memory (79.15GiB) x gpu_memory_utilization (0.90) = 71.24GiB
INFO 01-10 15:08:58 worker.py:241] model weights take 51.23GiB; non_torch_memory takes 0.15GiB; PyTorch activation peak memory takes 1.18GiB; the rest of the memory reserved for KV Cache is 18.67GiB.
INFO 01-10 15:08:58 gpu_executor.py:76] # GPU blocks: 2549, # CPU blocks: 546
INFO 01-10 15:08:58 gpu_executor.py:80] Maximum concurrency for 4096 tokens per request: 9.96x
INFO 01-10 15:09:17 model_runner.py:1416] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00,  2.42s/it]
INFO 01-10 15:09:22 model_runner.py:1542] Graph capturing finished in 5 secs, took 0.24 GiB
INFO 01-10 15:09:22 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 36.41 seconds
Some kwargs in processor config are unused and will not have any effect: image_std, sft_format, downsample_ratio, normalize, candidate_resolutions, patch_size, image_token, add_special_token, ignore_id, image_mean, mask_prompt, pad_token. 
Add pad token = ['<｜▁pad▁｜>'] to the tokenizer
<｜▁pad▁｜>:2
Add image token = ['<image>'] to the tokenizer
<image>:128815
Add grounding-related tokens = ['<|ref|>', '<|/ref|>', '<|det|>', '<|/det|>', '<|grounding|>'] to the tokenizer with input_ids
<|ref|>:128816
<|/ref|>:128817
<|det|>:128818
<|/det|>:128819
<|grounding|>:128820
Add chat tokens = ['<|User|>', '<|Assistant|>'] to the tokenizer with input_ids
<|User|>:128821
<|Assistant|>:128822

You're using a CachedLlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00,  1.78s/it, est. speed input: 802.93 toks/s, output: 25.69 toks/s]
The image shows a view of a tall tower, likely a communications or observation tower, surrounded by cherry blossom trees in full bloom. The sky is clear and blue, providing a beautiful backdrop to the scene.
The image features a tall tower, likely a communications or observation tower, surrounded by blooming cherry blossoms. The blossoms are in the foreground, with the tower rising into the background. The sky is clear and blue, providing a vibrant backdrop.
The image shows a view of a tall tower, likely a skyscraper or observation tower, with cherry blossoms in the foreground. The tower is surrounded by a clear blue sky, and the cherry blossoms are in full bloom, creating a beautiful and vibrant scene.
The image shows a view of a tall tower with a blue sky in the background. The foreground is filled with pink cherry blossoms, creating a beautiful contrast between the natural and man-made elements.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

Otherwise LGTM. As per offline discussion, we can work on deepseek-ai/deepseek-vl2-tiny and the inner timm model in another PR.

[Swipe4057]

CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server --model /data/models/deepseek-vl2 --served-model-name deepseek-vl2 --gpu_memory_utilization 0.9 --quantization fp8 --max-model-len 4096 --disable-log-requests

Result:

Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 389, in run_mp_engine
raise e
File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 116, in from_engine_args
engine_config = engine_args.create_engine_config(usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/engine/arg_utils.py", line 1043, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/engine/arg_utils.py", line 969, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/config.py", line 342, in init
self.multimodal_config = self._init_multimodal_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/config.py", line 398, in _init_multimodal_config
if ModelRegistry.is_multimodal_model(architectures):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 429, in is_multimodal_model
model_cls, _ = self.inspect_model_cls(architectures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 384, in inspect_model_cls
for arch in architectures:
TypeError: 'NoneType' object is not iterable

[DarkLight1337]

CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server --model /data/models/deepseek-vl2 --served-model-name deepseek-vl2 --gpu_memory_utilization 0.9 --quantization fp8 --max-model-len 4096 --disable-log-requests

Result:

Process SpawnProcess-1: Traceback (most recent call last): File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 389, in run_mp_engine raise e File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 378, in run_mp_engine engine = MQLLMEngine.from_engine_args(engine_args=engine_args, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/engine/multiprocessing/engine.py", line 116, in from_engine_args engine_config = engine_args.create_engine_config(usage_context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/engine/arg_utils.py", line 1043, in create_engine_config model_config = self.create_model_config() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/engine/arg_utils.py", line 969, in create_model_config return ModelConfig( ^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/config.py", line 342, in init self.multimodal_config = self._init_multimodal_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/config.py", line 398, in _init_multimodal_config if ModelRegistry.is_multimodal_model(architectures): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 429, in is_multimodal_model model_cls, _ = self.inspect_model_cls(architectures) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 384, in inspect_model_cls for arch in architectures: TypeError: 'NoneType' object is not iterable

Can you show the full logs?

[Isotr0py]

File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 384, in inspect_model_cls
for arch in architectures:
TypeError: 'NoneType' object is not iterable

The config.json in Deepseek-VL2's model repos are all missing the architectures field, so you need to specify --hf_overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' or add architectures": ["DeepseekVLV2ForCausalLM"] to the config file manually.

[iamweiliu]

File "/data/venvs/lib/vllm/vllm/model_executor/models/registry.py", line 384, in inspect_model_cls
for arch in architectures:
TypeError: 'NoneType' object is not iterable

The config.json in Deepseek-VL2's model repos are all missing the architectures field, so you need to specify --hf_overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}' or add architectures": ["DeepseekVLV2ForCausalLM"] to the config file manually.

Save my life!

[iamweiliu]

ERROR 01-14 18:14:14 engine.py:387] AttributeError: 'DeepseekVLV2Config' object has no attribute 'hidden_size'

[Isotr0py]

@iamweiliu Can you provide the full logs? hidden_size should not be got from DeepseekVLV2Config because it doesn't have this field.

[iamweiliu]

@iamweiliu Can you provide the full logs? hidden_size should not be got from DeepseekVLV2Config because it doesn't have this field.

I already fix it. Just install https://github.com/Isotr0py/DeepSeek-VL2.