[Dynamic Spec Decoding] Minor fix for disabling speculative decoding #5000
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
comaniac
approved these changes
May 23, 2024
| @@ -276,7 +276,8 @@ def execute_model( | |||
| # If no spec tokens, call the proposer and scorer workers normally. | |||
| # Used for prefill. | |||
There was a problem hiding this comment.
Refine the comment to include auto disable?
cadedaniel
approved these changes
May 23, 2024
| "ngram_prompt_lookup_max": 3, | ||
| "speculative_disable_by_batch_size": 4 | ||
| }]) | ||
| @pytest.mark.parametrize("batch_size", [1, 2, 5, 8]) |
There was a problem hiding this comment.
nit: suggest only [1, 5] to reduce test time
added 2 commits
May 23, 2024 19:16
|
Looks like the CI failure is unrelated and we should just merge this. cc @simon-mo |
dtrifiro
pushed a commit
to opendatahub-io/vllm
that referenced
this pull request
May 31, 2024
robertgshaw2-redhat
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Jun 8, 2024
joerunde
pushed a commit
to joerunde/vllm
that referenced
this pull request
Jun 17, 2024
robertgshaw2-redhat
pushed a commit
to neuralmagic/nm-vllm
that referenced
this pull request
Jul 14, 2024
Temirulan
pushed a commit
to Temirulan/vllm-whisper
that referenced
this pull request
Sep 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the current implementation, even if
running_queue_size >= speculative_disable_by_batch_size, it will still go through the speculative decoding logic, which includes get_spec_proposals with k=0, score_proposal, rejection sampling and create sampler output. The flow introduces extra overhead (especially rejection sampling), which makes disabling speculative decoding slower than 'real' without speculative decoding.To fix this, we can just reuse the
_run_no_specto avoid touching the sd flow at all. Also add a test to check the correctness.Concretely, for a batch size of 8 with 128 output tokens, TP=4, for LLama3-70B, the batch latency is
Here, without SD means not using sd flag at all. Disable SD means using SD flag, but set
speculative_disable_by_batch_sizesmaller than batch size to disable speculative decoding.After the fix, we are still slower than the original case, this is caused by broadcasting control flow, which will be fixed in future PRs.