[Model] Add cogagent model support vLLM

[sixsixcoder]

1. Add THUDM/CogAgent model support vLLM
2. Added new rotation position encoding method

test demo

from PIL import Image
from vllm import LLM, SamplingParams

model_name = "THUDM/cogagent-9b-20241220"

def procress_inputs():
    task = "将邮件标记为已读"
    platform_str = "(Platform: Mac)\n"
    history_str = "\nHistory steps: "
    format_str = "(Answer in Action-Operation-Sensitive format.)"
    query = f"Task: {task}{history_str}\n{platform_str}{format_str}"
    return query

llm = LLM(model=model_name,
          tensor_parallel_size=1,
          max_model_len=8192,
          trust_remote_code=True,
          enforce_eager=True)
stop_token_ids = [151329, 151336, 151338]
sampling_params = SamplingParams(temperature=0.2,
                                 max_tokens=1024,
                                 stop_token_ids=stop_token_ids)

prompt = procress_inputs()
image = Image.open("abc.png").convert('RGB')
inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "image": image
        },
        }
outputs = llm.generate(inputs, sampling_params=sampling_params)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)

Expected Output

Action: 打开微信应用程序，以便进行后续操作。
Grounded Operation: LAUNCH(app='微信', url='None')
<<一般操作>>

Actual Output

Action: 在屏幕上找到“Action: 在屏幕上找到“Group 11574Group1574396Group 1574Group 1575Group 1576Group 1577Group 1578Group 1579Group 1580Group 1581Group 1582Group 1583Group 1584Group 1585Group 1586Group 1587Group 1588Group 1589Group 1590Group 1591Group 1592Group 1593Group 1594Group 1595Group 1596Group 1597Group 1598Group 1599Group 1600Group 1601Group 1602Group 1603Group 1604Group 1605Group 1606Group 1607Group 1608Group 1609Group 1610Group 1611Group 1612Group 1613Group 1614Group 1615Group 1616Group 1617Group 1618Group 1619Group 1620Group 1621Group 1622Group 1623Group 1624 1625Group 1626Group 1627Group 1628Group 1629Group 1630Group 1631Group 1632Group 1633Group 1634Group 1635Group 1636Group 1637Group 1638Group 1639Group 1640 16411574Group سمسمه1475Group 1476Group 1477Group 1478Group 1479Group 1480Group 1481Group 1482Group 1483Group 1484Group 1485Group 1486Group 1487Group 1488Group 1489Group 1490Group 1491Group 1492Group 1493Group 1494Group 1495Group 1496Group 1497Group 1498Group 1499Group 1500Group 1501Group 1502Group 1503Group 1504Group 1505Group 1506Group 1507Group 1508Group 1509Group 1510Group 1511Group 1512Group 1513Group 1514Group 1515Group 1516Group 1517Group 1518Group 1520Group 1521Group 1522Group 1523Group 1524 1525Group 1526Group 1527Group 1528Group 1529Group 1530Group 1531Group 1532Group 1533Group 1534Group 1535Group 1536Group 1537Group 1538 1539Group 1540 1541”控件，并对其进行左键单击，以选择相应的选项或功能。
Grounded Operation: CLICK(box=[[000,000,0C5760505401_s-830_5217435401.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0. groupGroup1574Group1575Group1576Group1577Group1578Group1579Group1580Group1581Group1582Group1583Group158

I want to add CogAgent model support vLLM. But when I encounter this problem, the actual output is inconsistent with the expected output. Can you help me?

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[sixsixcoder]

@Isotr0py can you help me? Thank you so much.

[DarkLight1337]

@jeejeelee have you started implementing merged multi-modal processor for this model yet? If so can you contribute to this PR as well?

[jeejeelee]

@jeejeelee have you started implementing merged multi-modal processor for this model yet? If so can you contribute to this PR as well?

It's just getting started, progress is very slow.

[Isotr0py]

I'm a little bit busy this week, will try to get bandwidth to start working on this ASAP :(

BTW, seems that the main difference between cogvlm and cogagent is only the MQA?

[Isotr0py]

THUDM/cogagent-9b-20241220 should work now:

Inputs

task = "搜索并选择 icloud\n"
platform_str = "(Platform: Mac)\n"
history_str = "\nHistory steps: "
format_str = "(Answer in Action-Operation-Sensitive format.)"
promtp = f"Task: {task}{history_str}\n{platform_str}{format_str}"

Image: https://github.com/THUDM/CogAgent/blob/main/assets/app_1.png

Outputs

$ python examples/offline_inference/offline_inference_vision_language.py -m glm4v
INFO 01-11 00:21:08 __init__.py:179] Automatically detected platform cuda.
INFO 01-11 00:21:13 config.py:516] This model supports multiple tasks: {'classify', 'reward', 'generate', 'embed', 'score'}. Defaulting to 'generate'.
WARNING 01-11 00:21:13 cuda.py:98] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
WARNING 01-11 00:21:13 config.py:650] Async output processing is not supported on the current platform type cuda.
INFO 01-11 00:21:13 llm_engine.py:234] Initializing an LLM engine (v0.1.dev4022+g32c9eff) with config: model='../cogagent-9b-20241220', speculative_config=None, tokenizer='../cogagent-9b-20241220', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../cogagent-9b-20241220, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[],"max_capture_size":0}, use_cached_outputs=False, 
WARNING 01-11 00:21:13 tokenizer.py:206] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 01-11 00:21:14 cuda.py:216] Using Flash Attention backend.
INFO 01-11 00:21:14 model_runner.py:1094] Starting to load model ../cogagent-9b-20241220...
INFO 01-11 00:21:14 cuda.py:200] Cannot use FlashAttention-2 backend for head size 112.
INFO 01-11 00:21:14 cuda.py:213] Using XFormers backend.
Loading safetensors checkpoint shards:   0% Completed | 0/6 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  17% Completed | 1/6 [00:01<00:05,  1.04s/it]
Loading safetensors checkpoint shards:  33% Completed | 2/6 [00:02<00:04,  1.13s/it]
Loading safetensors checkpoint shards:  50% Completed | 3/6 [00:03<00:03,  1.12s/it]
Loading safetensors checkpoint shards:  67% Completed | 4/6 [00:04<00:02,  1.12s/it]
Loading safetensors checkpoint shards:  83% Completed | 5/6 [00:05<00:01,  1.14s/it]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:06<00:00,  1.01it/s]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:06<00:00,  1.06s/it]

INFO 01-11 00:21:21 model_runner.py:1099] Loading model weights took 25.9437 GB
WARNING 01-11 00:21:22 tokenizer.py:206] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
INFO 01-11 00:21:25 worker.py:241] Memory profiling takes 3.32 seconds
INFO 01-11 00:21:25 worker.py:241] the current vLLM instance can use total_gpu_memory (47.50GiB) x gpu_memory_utilization (0.90) = 42.75GiB
INFO 01-11 00:21:25 worker.py:241] model weights take 25.94GiB; non_torch_memory takes 0.40GiB; PyTorch activation peak memory takes 1.07GiB; the rest of the memory reserved for KV Cache is 15.34GiB.
INFO 01-11 00:21:25 gpu_executor.py:76] # GPU blocks: 25133, # CPU blocks: 6553
INFO 01-11 00:21:25 gpu_executor.py:80] Maximum concurrency for 2048 tokens per request: 196.35x
INFO 01-11 00:21:27 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 5.45 seconds
INFO 01-11 00:21:27 preprocess.py:236] Your model uses the legacy input pipeline instead of the new multi-modal processor. Please note that the legacy pipeline will be removed in a future release. For more details, see: https://github.com/vllm-project/vllm/issues/10114
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:06<00:00,  1.59s/it, est. speed input: 1030.87 toks/s, output: 27.25 toks/s]
Action: 打开浏览器并访问https://www.apple.com，以便进行后续操作。
Grounded Operation: LAUNCH(app='Apple', url='https://www.apple.com')
<<一般操作>>
Action: 打开浏览器并访问网址https://www.apple.com，以便进行后续操作。
Grounded Operation: LAUNCH(app='Apple', url='https://www.apple.com')
<<一般操作>>
Action: 打开浏览器，并访问网址https://www.apple.com，以便进行后续操作。
Grounded Operation: LAUNCH(app='Apple', url='https://www.apple.com')
<<一般操作>>
Action: 打开浏览器，并访问网址https://www.icloud.com/，以便进行后续操作。
Grounded Operation: LAUNCH(app='iCloud', url='https://www.icloud.com/')
<<一般操作>>

[Isotr0py]

@jeejeelee Any updates about the merged multi-modal processor? I'm going to leave it to a following PR since this PR becomes small.