Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Tensor Parallel in Python API #1661

Merged
merged 14 commits into from
Aug 8, 2024
Merged

Conversation

rasbt
Copy link
Contributor

@rasbt rasbt commented Aug 7, 2024

Adds tensor parallel support for the Python API:

from litgpt.api import LLM

llm = LLM.load(
    model="microsoft/phi-2",
    distribute=None
)

llm.distribute(generate_strategy="tensor_parallel", devices=2)

llm.generate(prompt="What do llamas eat?", top_k=1)

@rasbt rasbt requested review from awaelchli and lantiga as code owners August 7, 2024 20:36
@rasbt rasbt marked this pull request as draft August 7, 2024 20:36
@rasbt rasbt force-pushed the python-api-tensor-parallel branch from fce7644 to 0268e65 Compare August 7, 2024 21:48
@rasbt rasbt marked this pull request as ready for review August 8, 2024 13:53
@rasbt rasbt marked this pull request as draft August 8, 2024 14:04
@rasbt rasbt marked this pull request as ready for review August 8, 2024 19:54
@rasbt rasbt enabled auto-merge (squash) August 8, 2024 20:51
@rasbt rasbt merged commit 40c293d into main Aug 8, 2024
8 checks passed
@rasbt rasbt deleted the python-api-tensor-parallel branch August 8, 2024 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant