Skip to content
/ sglang Public
forked from sgl-project/sglang

SGLang is yet another fast serving framework for large language models and vision language models.

License

Notifications You must be signed in to change notification settings

yukavio/sglang

This branch is 555 commits behind sgl-project/sglang:main.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

d95a5f5 · Dec 19, 2024
Dec 17, 2024
Nov 30, 2024
Aug 1, 2024
Dec 17, 2024
Dec 6, 2024
Dec 17, 2024
Dec 6, 2024
Dec 19, 2024
Dec 19, 2024
Dec 15, 2024
Dec 12, 2024
Dec 16, 2024
Nov 6, 2024
Dec 1, 2024
Feb 7, 2024
Nov 6, 2024
Nov 22, 2024
Nov 22, 2024
Nov 29, 2024
Dec 17, 2024

Repository files navigation

logo

PyPI PyPI - Downloads license issue resolution open issues


| Blog | Documentation | Join Slack | Join Bi-Weekly Development Meeting | Slides |

News

  • [2024/12] 🔥 SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs (blog).
  • [2024/10] 🔥 The First SGLang Online Meetup (slides).
  • [2024/09] SGLang v0.3 Release: 7x Faster DeepSeek MLA, 1.5x Faster torch.compile, Multi-Image/Video LLaVA-OneVision (blog).
  • [2024/07] Faster Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) (blog).
More
  • [2024/02] SGLang enables 3x faster JSON decoding with compressed finite state machine (blog).
  • [2024/04] SGLang is used by the official LLaVA-NeXT (video) release (blog).
  • [2024/01] SGLang provides up to 5x faster inference with RadixAttention (blog).
  • [2024/01] SGLang powers the serving of the official LLaVA v1.6 release demo (usage).

About

SGLang is a fast serving framework for large language models and vision language models. It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language. The core features include:

  • Fast Backend Runtime: Provides efficient serving with RadixAttention for prefix caching, jump-forward constrained decoding, overhead-free CPU scheduler, continuous batching, token attention (paged attention), tensor parallelism, FlashInfer kernels, chunked prefill, and quantization (FP8/INT4/AWQ/GPTQ).
  • Flexible Frontend Language: Offers an intuitive interface for programming LLM applications, including chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.
  • Extensive Model Support: Supports a wide range of generative models (Llama, Gemma, Mistral, QWen, DeepSeek, LLaVA, etc.), embedding models (e5-mistral, gte, mcdse) and reward models (Skywork), with easy extensibility for integrating new models.
  • Active Community: SGLang is open-source and backed by an active community with industry adoption.

Getting Started

Benchmark And Performance

Learn more in our release blogs: v0.2 blog, v0.3 blog, v0.4 blog

Roadmap

Development Roadmap (2024 Q4)

Adoption and Sponsorship

The project is supported by (alphabetically): AMD, Baseten, Etched, Hyperbolic, Jam & Tea Studios, LinkedIn, Meituan, NVIDIA, RunPod, Stanford, UC Berkeley, xAI and 01.AI.

Acknowledgment and Citation

We learned from the design and reused code from the following projects: Guidance, vLLM, LightLLM, FlashInfer, Outlines, and LMQL. Please cite our paper, SGLang: Efficient Execution of Structured Language Model Programs, if you find the project useful.

About

SGLang is yet another fast serving framework for large language models and vision language models.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 95.7%
  • Rust 2.9%
  • Cuda 0.6%
  • Shell 0.4%
  • Dockerfile 0.1%
  • C++ 0.1%
  • Other 0.2%