Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: count tokens before calling '/v1/chat/completions' #4027

Open
GPTLocalhost opened this issue Nov 1, 2024 · 0 comments
Open

Feature: count tokens before calling '/v1/chat/completions' #4027

GPTLocalhost opened this issue Nov 1, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@GPTLocalhost
Copy link
Contributor

We recently integrated Microsoft Word with LocalAI through a local Word Add-in. You can see a demo of the integration here. We're planning to add a feature that allows users to see how many tokens are left before calling '/v1/chat/completions'. The question is:

Is it possible for LocalAI to count the tokens of the prompt 
before calling '/v1/chat/completions'? Any insights would be greatly appreciated.

For OpenAI models, this cookbook might be a direction. For other models with different tokenizers, it is not as clear how to implement such a function.

@GPTLocalhost GPTLocalhost added the enhancement New feature or request label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant