Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check gpu availability at server args creation #2340

Merged
merged 1 commit into from
Dec 4, 2024

Conversation

MrAta
Copy link
Contributor

@MrAta MrAta commented Dec 3, 2024

Motivation

During server args construction we check the gpu memory size to configure knobs like chunked prefill and cuda graph sizes.
For use cases where the args are constructed programmatically on a non-GPU environment and sent to a GPU environment, this fails because we use nvidia-smi to check nv GPU memory size while it's not available in those environments.

Modifications

Add a check for gpu availability before checking the memory size.

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

Copy link
Collaborator

@ByronHsu ByronHsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM to unblock our internal use. cc @merrymercy for the final stamp

@zhyncs zhyncs merged commit ed45e50 into sgl-project:main Dec 4, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants