[core] further polish memory profiling #12126

youkaichao · 2025-01-16T17:42:56Z

Improve upon #11809

current main branch, vllm serve meta-llama/Llama-3.2-11B-Vision --load-format dummy --max-model-len 65536 --max-num-seqs 128 --enforce-eager will fail.

after this PR, it can work on H100-80G now:

INFO 01-16 09:40:00 worker.py:238] the current vLLM instance can use total_gpu_memory (79.22GiB) x gpu_memory_utilization (0.90) = 71.29GiB
INFO 01-16 09:40:00 worker.py:238] model weights take 19.86GiB; non_torch_memory takes 0.16GiB; PyTorch activation peak memory takes 36.63GiB; the rest of the memory reserved for KV Cache is 14.65GiB.
INFO 01-16 09:40:00 executor_base.py:95] # CUDA blocks: 6001, # CPU blocks: 1638

Signed-off-by: youkaichao <[email protected]>

github-actions · 2025-01-16T17:43:08Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

youkaichao · 2025-01-16T17:43:39Z

tests/test_utils.py


    # load weights

    weights = torch.randn(128, 1024, 1024, device='cuda', dtype=torch.float32)

-    weights_memory_in_bytes = 128 * 1024 * 1024 * 4 # 512 MiB


refactor in this PR: remote the _in_bytes in variable name to make the name shorter.

youkaichao · 2025-01-16T17:43:58Z

tests/test_utils.py

    assert abs(non_torch_ratio - 1) <= 0.05
-    assert abs(torch_peak_ratio - 1) <= 0.05


now this becomes accurate.

youkaichao · 2025-01-16T17:45:08Z

vllm/utils.py

+        # we measure the torch peak memory usage via allocated_bytes,
+        # rather than `torch.cuda.memory_reserved()` .
+        # After `torch.cuda.reset_peak_memory_stats()`,
+        # `torch.cuda.memory_reserved()` will keep growing, and only shrink
+        # when we call `torch.cuda.empty_cache()` or OOM happens.
+        self.torch_peak = torch.cuda.memory_stats().get(
+            "allocated_bytes.all.peak", 0)


this is the key change, reported by @gshtras

youkaichao · 2025-01-16T17:46:25Z

vllm/worker/worker.py

+            torch.cuda.reset_peak_memory_stats()
+            self.baseline_snapshot = MemorySnapshot()


another key change: we also measure the non-torch memory before creating the vllm instance.

Do we need to do this in v1/worker/gpu_worker.py as well?

v1 does not use this memory_profiling utility yet. welcome to port it to v1 code path!

gshtras · 2025-01-16T20:14:35Z

Ran some tests and can confirm that this fixes the original issue we had with Llama3.2 90B model peak memory jumping from 98GB to 160GB

tlrmchlsmth

Thanks for wrestling this into a better state!

youkaichao · 2025-01-18T04:24:57Z

failed tests look unrelated, merging.

Signed-off-by: youkaichao <[email protected]>

Signed-off-by: youkaichao <[email protected]> Signed-off-by: Bowen Wang <[email protected]>

Signed-off-by: youkaichao <[email protected]>

- **[Bugfix] Fix score api for missing max_model_len validation (vllm-project#12119)** - **[Bugfix] Mistral tokenizer encode accept list of str (vllm-project#12149)** - **[AMD][FP8] Using MI300 FP8 format on ROCm for block_quant (vllm-project#12134)** - **[torch.compile] disable logging when cache is disabled (vllm-project#12043)** - **[misc] fix cross-node TP (vllm-project#12166)** - **[AMD][CI/Build][Bugfix] use pytorch stale wheel (vllm-project#12172)** - **[core] further polish memory profiling (vllm-project#12126)** - **[Docs] Fix broken link in SECURITY.md (vllm-project#12175)** - **[Model] Port deepseek-vl2 processor, remove dependency (vllm-project#12169)** - **[core] clean up executor class hierarchy between v1 and v0 (vllm-project#12171)** - **[Misc] Support register quantization method out-of-tree (vllm-project#11969)** - **[V1] Collect env var for usage stats (vllm-project#12115)** - **[BUGFIX] Move scores to float32 in case of running xgrammar on cpu (vllm-project#12152)** - **[Bugfix] Fix multi-modal processors for transformers 4.48 (vllm-project#12187)** - **[torch.compile] store inductor compiled Python file (vllm-project#12182)** - **benchmark_serving support --served-model-name param (vllm-project#12109)** - **[Misc] Add BNB support to GLM4-V model (vllm-project#12184)** - **[V1] Add V1 support of Qwen2-VL (vllm-project#12128)** - **[Model] Support for fairseq2 Llama (vllm-project#11442)** - **[Bugfix] Fix num_heads value for simple connector when tp enabled (vllm-project#12074)** - **[torch.compile] fix sym_tensor_indices (vllm-project#12191)** - **Move linting to `pre-commit` (vllm-project#11975)** - **[DOC] Fix typo in docstring and assert message (vllm-project#12194)** - **[DOC] Add missing docstring in LLMEngine.add_request() (vllm-project#12195)** - **[Bugfix] Fix incorrect types in LayerwiseProfileResults (vllm-project#12196)** - **[Model] Add Qwen2 PRM model support (vllm-project#12202)** - **[Core] Interface for accessing model from `VllmRunner` (vllm-project#10353)** - **[misc] add placeholder format.sh (vllm-project#12206)** - **[CI/Build] Remove dummy CI steps (vllm-project#12208)** - **[CI/Build] Make pre-commit faster (vllm-project#12212)** - **[Model] Upgrade Aria to transformers 4.48 (vllm-project#12203)** - **[misc] print a message to suggest how to bypass commit hooks (vllm-project#12217)** - **[core][bugfix] configure env var during import vllm (vllm-project#12209)** - **[V1] Remove `_get_cache_block_size` (vllm-project#12214)** - **[Misc] Pass `attention` to impl backend (vllm-project#12218)** - **[Bugfix] Fix `HfExampleModels.find_hf_info` (vllm-project#12223)** - **[CI] Pass local python version explicitly to pre-commit mypy.sh (vllm-project#12224)** - **[Misc] Update CODEOWNERS (vllm-project#12229)** - **fix: update platform detection for M-series arm based MacBook processors (vllm-project#12227)** - **[misc] add cuda runtime version to usage data (vllm-project#12190)** - **[bugfix] catch xgrammar unsupported array constraints (vllm-project#12210)** - **[Kernel] optimize moe_align_block_size for cuda graph and large num_experts (e.g. DeepSeek-V3) (vllm-project#12222)** - **Add quantization and guided decoding CODEOWNERS (vllm-project#12228)** - **[AMD][Build] Porting dockerfiles from the ROCm/vllm fork (vllm-project#11777)** - **[BugFix] Fix GGUF tp>1 when vocab_size is not divisible by 64 (vllm-project#12230)** - **[ci/build] disable failed and flaky tests (vllm-project#12240)** - **[Misc] Rename `MultiModalInputsV2 -> MultiModalInputs` (vllm-project#12244)** - **[Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (vllm-project#12237)** - **[Misc] Remove redundant TypeVar from base model (vllm-project#12248)** - **[Bugfix] Fix mm_limits access for merged multi-modal processor (vllm-project#12252)** --------- Signed-off-by: Wallas Santos <[email protected]> Signed-off-by: Kunshang Ji <[email protected]> Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: youkaichao <[email protected]> Signed-off-by: hongxyan <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Michal Adamczyk <[email protected]> Signed-off-by: zibai <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Martin Gleize <[email protected]> Signed-off-by: Shangming Cai <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Yuan Tang <[email protected]> Signed-off-by: Chen Zhang <[email protected]> Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: isikhi <[email protected]> Signed-off-by: Jason Cheng <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: NickLucche <[email protected]> Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Konrad Zawora <[email protected]> Co-authored-by: Wallas Henrique <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> Co-authored-by: Gregory Shtrasberg <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Hongxia Yang <[email protected]> Co-authored-by: Russell Bryant <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: yancong <[email protected]> Co-authored-by: Simon Mo <[email protected]> Co-authored-by: Michal Adamczyk <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: gujing <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: imkero <[email protected]> Co-authored-by: Martin Gleize <[email protected]> Co-authored-by: mgleize user <[email protected]> Co-authored-by: shangmingc <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Yuan Tang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: wangxiyuan <[email protected]> Co-authored-by: Işık <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Cheng Kuan Yong Jason <[email protected]> Co-authored-by: Jinzhen Lin <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Jee Jee Li <[email protected]>

youkaichao added 6 commits January 17, 2025 01:08

fix

fa32104

Signed-off-by: youkaichao <[email protected]>

fix

41bf4f2

Signed-off-by: youkaichao <[email protected]>

fix

01b4364

Signed-off-by: youkaichao <[email protected]>

fix

42ee511

Signed-off-by: youkaichao <[email protected]>

fix

16779bd

Signed-off-by: youkaichao <[email protected]>

add tests

7e36d4a

Signed-off-by: youkaichao <[email protected]>

youkaichao requested review from zhuohan123, comaniac and njhill as code owners January 16, 2025 17:42

youkaichao commented Jan 16, 2025

View reviewed changes

youkaichao requested a review from tlrmchlsmth January 17, 2025 01:52

tlrmchlsmth approved these changes Jan 17, 2025

View reviewed changes

youkaichao enabled auto-merge (squash) January 17, 2025 15:48

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 17, 2025

youkaichao disabled auto-merge January 18, 2025 04:25

youkaichao merged commit da02cb4 into vllm-project:main Jan 18, 2025
67 of 70 checks passed

youkaichao deleted the fix_profiling branch January 18, 2025 04:25

joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025

[core] further polish memory profiling (vllm-project#12126)

bb23d3c

Signed-off-by: youkaichao <[email protected]>

joennlae pushed a commit to 44ai-labs/vllm that referenced this pull request Jan 19, 2025

[core] further polish memory profiling (vllm-project#12126)

a6ac593

Signed-off-by: youkaichao <[email protected]>

lckr pushed a commit to lckr/vllm that referenced this pull request Jan 19, 2025

[core] further polish memory profiling (vllm-project#12126)

914bc14

Signed-off-by: youkaichao <[email protected]>

kzawora-intel mentioned this pull request Jan 21, 2025

Rebase 2025.01.21 HabanaAI/vllm-fork#714

Merged

abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025

[core] further polish memory profiling (vllm-project#12126)

56eea1e

Signed-off-by: youkaichao <[email protected]> Signed-off-by: Bowen Wang <[email protected]>

abmfy pushed a commit to abmfy/vllm-flashinfer that referenced this pull request Jan 24, 2025

[core] further polish memory profiling (vllm-project#12126)

fd77c0a

Signed-off-by: youkaichao <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] further polish memory profiling #12126

[core] further polish memory profiling #12126

youkaichao commented Jan 16, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Jan 16, 2025

youkaichao Jan 16, 2025

youkaichao Jan 16, 2025

youkaichao Jan 16, 2025

youkaichao Jan 16, 2025

tlrmchlsmth Jan 16, 2025

youkaichao Jan 17, 2025

gshtras commented Jan 16, 2025

tlrmchlsmth left a comment

youkaichao commented Jan 18, 2025

		assert abs(non_torch_ratio - 1) <= 0.05
		assert abs(torch_peak_ratio - 1) <= 0.05

		torch.cuda.reset_peak_memory_stats()
		self.baseline_snapshot = MemorySnapshot()

[core] further polish memory profiling #12126

[core] further polish memory profiling #12126

Conversation

youkaichao commented Jan 16, 2025 • edited by github-actions bot Loading

github-actions bot commented Jan 16, 2025

youkaichao Jan 16, 2025

Choose a reason for hiding this comment

youkaichao Jan 16, 2025

Choose a reason for hiding this comment

youkaichao Jan 16, 2025

Choose a reason for hiding this comment

youkaichao Jan 16, 2025

Choose a reason for hiding this comment

tlrmchlsmth Jan 16, 2025

Choose a reason for hiding this comment

youkaichao Jan 17, 2025

Choose a reason for hiding this comment

gshtras commented Jan 16, 2025

tlrmchlsmth left a comment

Choose a reason for hiding this comment

youkaichao commented Jan 18, 2025

youkaichao commented Jan 16, 2025 •

edited by github-actions bot

Loading