Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added groq support via local option w/ auth #1203

Merged
merged 4 commits into from
Apr 1, 2024
Merged

Conversation

cpacker
Copy link
Collaborator

@cpacker cpacker commented Apr 1, 2024

Please describe the purpose of this pull request.

Adds Groq API support.

Note: Groq API doesn't support either function calling OR a raw /completions (not /chat/completions) API route. Because of this, we need to package our MemGPT prompt into a dummy user message inside the /chat/completions payload - this is very similar to how the lmstudio-legacy option works.

TODO in the future once Groq's API either allows function calling (and it stable / works well) and/or adds support for raw /completions endpoints, we should deprecate this code into groq-legacy similar to lmstudio-legacy.

How to test

(pymemgpt-PCQPn4ce-py3.10) (base) loaner@MacBook-Pro-55 MemGPT-2 % memgpt configure
Loading config from /Users/loaner/.memgpt/config
? Select LLM inference provider: local
? Select LLM backend (select 'openai' if you have an OpenAI compatible proxy): groq
? Enter default endpoint: https://api.groq.com/openai
? Enter your Groq API key: ********************************************************
? Select default model: mixtral-8x7b-32768
? Select default model wrapper (recommended: chatml): chatml
? Select your model's context window (for Mistral 7B models, this is probably 8k / 8192): 32768
? Select embedding provider: openai
? Select default preset: memgpt_chat
? Select default persona: sam_pov
? Select default human: basic
? Select storage backend for archival data: chroma
? Select chroma backend: persistent
? Select storage backend for recall data: sqlite
📖 Saving config to /Users/loaner/.memgpt/config

(pymemgpt-PCQPn4ce-py3.10) (base) loaner@MacBook-Pro-55 MemGPT-2 % memgpt run
Loading config from /Users/loaner/.memgpt/config

? Would you like to select an existing agent? No

🧬 Creating new agent...
->  🤖 Using persona profile: 'sam_pov'
->  🧑 Using human profile: 'basic'
Loading config from /Users/loaner/.memgpt/config
Loading config from /Users/loaner/.memgpt/config
🎉 Created new agent 'DecisiveSeashell' (id=d4666cde-aa82-44b2-bffd-bb0587bee324)

Hit enter to begin (will request first MemGPT message)

💭 User logged in. Welcome, Chad. Initiating persona protocols.
🤖 Hello Chad, I'm Sam. It's nice to meet you.
> Enter your message:

@cpacker cpacker mentioned this pull request Apr 1, 2024
32 tasks
@cpacker cpacker requested a review from sarahwooders April 1, 2024 21:31
Copy link
Collaborator

@sarahwooders sarahwooders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@cpacker cpacker merged commit d6e44fd into main Apr 1, 2024
4 checks passed
@cpacker cpacker deleted the groq-support branch April 1, 2024 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants