Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] Model support dashboard (v2) #1126

Open
mudler opened this issue Oct 2, 2023 · 9 comments
Open

[EPIC] Model support dashboard (v2) #1126

mudler opened this issue Oct 2, 2023 · 9 comments
Assignees
Labels
enhancement New feature or request high prio new-backend roadmap up for grabs Tickets that no-one is currently working on

Comments

@mudler
Copy link
Owner

mudler commented Oct 2, 2023

This epic is a major tracker for all the backends additions that should be part of LocalAI v2 and ongoing efforts.

The objective is to release a v2 which deprecates old models which are now superseded, plus adding a new set. In order to achieve this my idea is to clean up the current state and start pinning dependencies for all the backends which requires specific environment settings (python-based ones).

Some key points:

  • The go-llama backend will be sunset by the c++ version which is directly tied and easier to pinpoint to llama.cpp versions
  • we will deprecate some (if not all) the duplicated backends arch (like falcon,gpt-2, gpt-neo-x, etc..) which are now supported by llama.cpp
  • the python backends will use conda envionments to pin specific versions
  • having a base point for backends for several languages (rust, c++, python, golang, ...)
  • re-order internal layout to accomodate more backends as needed

Some backends will be deprecated as superseded, as such some repositories will be archived (TBD yet).

Backends:

After conda

We need to still test on master:

  • Vall-E-X
  • diffusers
  • bark
  • autogptq
  • exllama
  • vllm

Some rough first-steps required :

Re-layout

After the re-layout we can add new backends listed above without any clashes and:

@mudler mudler added the enhancement New feature or request label Oct 2, 2023
@mudler mudler self-assigned this Oct 2, 2023
@mudler mudler changed the title [EPIC Backends to add [EPIC] Backends to add Oct 2, 2023
@Aisuko Aisuko self-assigned this Oct 2, 2023
@localai-bot
Copy link
Contributor

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Issue description:
The issue is related to the LocalAI project and its implementation of the Image endpoint. The user wants to create a cute baby sea otter image with the given prompt, size, and number of images. However, the code provided in the readInput function is not able to parse the input correctly and returns an error.

Proposed solution:
The issue seems to be related to the parsing of the input parameters. The user can try modifying the code in the readInput function to handle the specific format of the prompt, size, and number of images provided by the OpenAI API. Alternatively, they can try using a different API that provides more flexibility in the input format or use a different tool to generate the image.

If the issue persists, the user can also check if there are any updates available for the LocalAI project or report the issue on the GitHub repository to get help from the community.

Sources:

@mudler mudler changed the title [EPIC] Backends to add [EPIC] Backends to add (v2) Oct 2, 2023
@Aisuko
Copy link
Collaborator

Aisuko commented Oct 14, 2023

I am thinking here, Should we need to have a outline likes below:

  • C++
    • llama.cpp series LLM, like: Qwen, chatglm)
  • Python
    • [ ]Huggingface
    • autoGPT
  • Rust
    • rustformers
    • candle

If we keep this in mind, we can split the model to the specific backend. For example, I saw the Qwen LLM mentioned their C++ implementation of it "working in the same way as llama.cpp". So, maybe it can be loaded by using llama.cpp. It should be compatible with C++ our backend.

I also suggest to add some labels are related to the series of the backends. We can know a new model can be compatible by our backends.

@mudler mudler changed the title [EPIC] Backends to add (v2) [EPIC] Backends for v2 Oct 17, 2023
@mudler
Copy link
Owner Author

mudler commented Nov 4, 2023

conda branch was merged in #1144 . I'm looking now into make the llama.cpp backend on par with llama-go and also add llava support to it.

I'm going to refactor and re-layout things in the new backend directory too

@Aisuko
Copy link
Collaborator

Aisuko commented Nov 5, 2023

@mudler thank you for mentioning. Here are some questions(second, and third one) may need you help #1180 (comment). Here what I am thinking is that we can use a tiny model to test the Rust backend features. And make sure everything ok. Maybe we can merge it.

And if everything ok. We can add other LLMs. I have plan to support Llama2(60% finished but it still has an issue), whisper, and also support onnex format.

@Aisuko Aisuko pinned this issue Nov 5, 2023
@mudler
Copy link
Owner Author

mudler commented Nov 13, 2023

Breaking re-layout PR: #1279

@kno10
Copy link
Contributor

kno10 commented Nov 27, 2023

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

@muka
Copy link
Contributor

muka commented Dec 27, 2023

I may start looking into #1273 while this progresses. What do you think ?

@mudler
Copy link
Owner Author

mudler commented Dec 27, 2023

I may start looking into #1273 while this progresses. What do you think ?

please feel free to go ahead, there are many pieces involved in here, any help is more than appreciated 👍

@mudler mudler changed the title [EPIC] Backends for v2 [EPIC] Model support dashboard (v2) Jan 24, 2024
@mudler
Copy link
Owner Author

mudler commented Mar 4, 2024

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

in #1746 I'm taking care of automatically binding the HF cache variables to the models directories if not set already

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request high prio new-backend roadmap up for grabs Tickets that no-one is currently working on
Projects
None yet
Development

No branches or pull requests

6 participants
@muka @mudler @kno10 @Aisuko @localai-bot and others