Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add environment variable to control the intra-thread number of thread #738

Closed
wsxiaoys opened this issue Nov 9, 2023 · 2 comments · Fixed by #742
Closed

Add environment variable to control the intra-thread number of thread #738

wsxiaoys opened this issue Nov 9, 2023 · 2 comments · Fixed by #742
Labels
enhancement New feature or request good first issue Good for newcomers
Milestone
Tabby 0.6.0

Comments

@wsxiaoys
Copy link
Member

wsxiaoys commented Nov 9, 2023

Create an environment variable LLAMA_CPP_N_THREADS and pass it to llama_set_n_threads.

Code location: https://github.com/TabbyML/tabby/blob/03ff80efdb792e7e504d28ccedf622d8a851423b/crates/llama-cpp-bindings/src/engine.cc#L291C66-L291C66

Reference: https://github.com/ggerganov/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/llama.h#L488C20-L488C39
@wsxiaoys wsxiaoys added enhancement New feature or request good first issue Good for newcomers labels Nov 9, 2023
@wsxiaoys wsxiaoys mentioned this issue Nov 9, 2023
Closed
@wsxiaoys wsxiaoys added this to the Tabby 0.6.0 milestone Nov 9, 2023
@erfanium
Copy link
Contributor

erfanium commented Nov 9, 2023 •
edited
Loading

@wsxiaoys Can i work on it? I know where to start

@wsxiaoys
Copy link
Member Author

wsxiaoys commented Nov 9, 2023

Sure! Feel free to send out a PR

@erfanium erfanium mentioned this issue Nov 9, 2023
Merged
@wsxiaoys wsxiaoys mentioned this issue Nov 25, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
Tabby 0.6.0
Development

Successfully merging a pull request may close this issue.

feat: add LLAMA_CPP_N_THREADS env
2 participants
@wsxiaoys @erfanium