Update doc for server arguments #2742
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
simveit
force-pushed
the
feature/server-arguments-docs
branch
2 times, most recently
from
fbc1a63 to
abb44cf
Compare
simveit
force-pushed
the
feature/server-arguments-docs
branch
from
abb44cf to
0a288d7
Compare
zhaochenyang20
requested changes
Jan 6, 2025
There was a problem hiding this comment.
I love the detailed and educational docs of parameters. Having two suggestions:
- We are documenting the official usage, so we can move the educational part to other unoffical repos, like my ML sys tutorial. 😂
- Keep the things concise. If we want to explain the concept, I think just one sentence of educational explanation and give a link to details, which could be bettter.
| ``` | ||
|
|
||
| </details> | ||
| ## Model and tokenizer |
There was a problem hiding this comment.
Cool. But for the docs, always keep one first-order title # and several second-order title ##, do not use forth-order title ####.
There was a problem hiding this comment.
Adjusted to include Server Arguments title.
docs/backend/server_arguments.md
Outdated
|
|
||
| * `tp_size`: This parameter is important if we have multiple GPUs and our model doesn't fit on a single GPU. *Tensor parallelism* means we distribute our model weights over multiple GPUs. Note that his technique is mainly aimed at *memory efficency* and not at a *higher throughput* as there is inter GPU communication needed to obtain the final output of each layer. For better understanding of the concept you may look for example [here](https://pytorch.org/tutorials/intermediate/TP_tutorial.html#how-tensor-parallel-works). | ||
|
|
||
| * `stream_interval`: If we stream the output to the user this parameter determines at which interval we perform streaming. The interval length is measured in tokens. |
There was a problem hiding this comment.
I am not so sure. Could you double check this and make it more clear.
There was a problem hiding this comment.
I will look this more carefully up. For now I left it as to do and come back to it at the end.
docs/backend/server_arguments.md
Outdated
|
|
||
| * `random_seed`: Can be used to enforce deterministic behavior. | ||
|
|
||
| * `constrained_json_whitespace_pattern`: When using `Outlines` grammar backend we can use this to allow JSON with syntatic newlines, tabs or multiple spaces. |
There was a problem hiding this comment.
I think we can create a ## for constraint decoding parameters.
There was a problem hiding this comment.
In general I think that we could restructure the whole sections. I suggest to do that after I included all parameters.
simveit
force-pushed
the
feature/server-arguments-docs
branch
from
58efd67 to
b939c56
Compare
…nto feature/server-arguments-docs
zhaochenyang20
requested changes
Jan 14, 2025
docs/backend/server_arguments.md
Outdated
| * `dist_init_addr`: The TCP address used for initializing PyTorch’s distributed backend (e.g. `192.168.0.2:25000`). | ||
| * `nnodes`: Total number of nodes in the cluster. | ||
| * `node_rank`: Rank (ID) of this node among the `nnodes` in the distributed setup. | ||
|
|
||
|
|
||
| ## Model override args in JSON |
There was a problem hiding this comment.
better call this ## Constraint Decoding
…lang into feature/server-arguments-docs
zhaochenyang20
requested changes
Jan 16, 2025
| ``` | ||
|
|
||
| </details> | ||
| ## Model and tokenizer |
zhaochenyang20
requested changes
Jan 18, 2025
| @@ -66,7 +66,7 @@ In this document we aim to give an overview of the possible arguments when deplo | |||
| * `watchdog_timeout`: Adjusts the watchdog thread’s timeout before killing the server if batch generation takes too long. | |||
| * `download_dir`: Use to override the default Hugging Face cache directory for model weights. | |||
| * `base_gpu_id`: Use to adjust first GPU used to distribute the model across available GPUs. | |||
|
|
|||
| * `allow_auto_truncate`: Automatically truncate requests that exceed the maximum input length. | |||
|
@zhyncs Wait for a final go over. |
Edenzzzz
reviewed
Jan 23, 2025
docs/backend/server_arguments.md
Outdated
Comment on lines
3
to
23
| - To enable multi-GPU tensor parallelism, add `--tp 2`. If it reports the error "peer access is not supported between these two devices", add `--enable-p2p-check` to the server launch command. | ||
| ``` | ||
| python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 2 | ||
| ``` | ||
| - To enable multi-GPU data parallelism, add `--dp 2`. Data parallelism is better for throughput if there is enough memory. It can also be used together with tensor parallelism. The following command uses 4 GPUs in total. | ||
| ``` | ||
| python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --dp 2 --tp 2 | ||
| ``` | ||
| - If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. The default value is `0.9`. | ||
| ``` | ||
| python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --mem-fraction-static 0.7 | ||
| ``` | ||
| - See [hyperparameter tuning](../references/hyperparameter_tuning.md) on tuning hyperparameters for better performance. | ||
| - If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size. | ||
| ``` | ||
| python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096 | ||
| ``` | ||
| - To enable torch.compile acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. This does not work for FP8 currently. | ||
| - To enable torchao quantization, add `--torchao-config int4wo-128`. It supports other [quantization strategies (INT8/FP8)](https://github.com/sgl-project/sglang/blob/v0.3.6/python/sglang/srt/server_args.py#L671) as well. | ||
| - To enable fp8 weight quantization, add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments. | ||
| - To enable fp8 kv cache quantization, add `--kv-cache-dtype fp8_e5m2`. |
There was a problem hiding this comment.
In addition to explaining individual args, keep some of the popular launch commands/arg combinations here for plug and play?
There was a problem hiding this comment.
In addition to explaining individual args, keep some of the popular launch commands/arg combinations here for plug and play?
Great. Suggestions. We should give some examples to command parameters. @simveit
There was a problem hiding this comment.
Included previous doc with small adjustment to use Router in case of DP.
zhaochenyang20
approved these changes
Jan 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
As explained here the current documentation of the backend needs update which we intend to implement here.
Checklist