Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add common + custom settings files for completion endpoints #631

Merged
merged 5 commits into from
Dec 18, 2023

Conversation

cpacker
Copy link
Collaborator

@cpacker cpacker commented Dec 16, 2023

Closes #623


TODOs

  • Add two example common settings files
  • Make completions settings pull from common
  • Allow override from user directory (eg ~/.memgpt/settings/completions.py)

Separate PR?
- [ ] Make openai/azure pull from common /chat/completions settings file
- [ ] Allow override from user directory (eg override temperature easily)

Please describe the purpose of this pull request

  • Right now if users want to modify the LLM inference settings they have to go into the package python files to open the constants
  • All the completion-based backends should share a single base settings file
  • This settings file should also be overwritten by settings that are placed somewhere in the ~/.memgpt home directory

How to test

Run inference via the backend and watch the backend endpoint to make sure settings are getting passed through properly (certain endpoints may require renaming of variables).

Have you tested this PR?

  • Test koboldcpp
  • Test llamacpp
  • Test webui
  • Test lmstudio
  • Test vllm

@cpacker cpacker changed the title Add common + custom settings files for completion endpoints [Draft] Add common + custom settings files for completion endpoints Dec 16, 2023
@cpacker cpacker self-assigned this Dec 16, 2023
@cpacker cpacker changed the title [Draft] Add common + custom settings files for completion endpoints feat: Add common + custom settings files for completion endpoints Dec 17, 2023
@cpacker cpacker requested a review from sarahwooders December 18, 2023 09:01
@cpacker cpacker marked this pull request as ready for review December 18, 2023 10:21
Copy link
Collaborator

@sarahwooders sarahwooders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@sarahwooders sarahwooders merged commit 8d44490 into main Dec 18, 2023
3 checks passed
@cpacker cpacker deleted the common-completions-settings branch December 22, 2023 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Common base settings file for LLM inference w/ MemGPT directory override
2 participants