More Model Providers support 更多模型服务商支持

[arvinxx]

LobeChat currently supports many model providers, and considering the development of the AI ecosystem, we have specifically opened this discussion to record everyone's needs for new model service providers.

LobeChat 目前已经支持了诸多模型供应商，考虑到 AI 生态的发展，我们专门开放此贴用于记录大家对于新模型服务商的需求。

---

Currently, the service providers currently supported by LobeChat are as follows:

目前 LobeChat 已支持的服务商如下：

• OpenAI
• Azure OpenAI
  • ✨ feat: support Azure OpenAI #177
• AWS Bedrock
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
  • Claude3: ✨ feat: add claude 3 to bedrock provider #1551
• Google Model （Gemini Pro 、Gemini Pro Vision）
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
  • Gemini 1.5 pro: 🐛 fix: fix Google Gemini pro 1.5 and system role issue #1801
• Zhipu AI (智谱AI)
  • ✨ feat: support Google / Zhipu / AWS Bedrock model providers #1173
• Moonshot AI (月之暗面) by @arvinxx
  • ✨ feat: support Moonshot AI #1232
• Ollama (Local LLM) Thanks to @sjy
  • ✨ feat(llm): support Ollama AI Provider (local llm) #1265
• Perplexity AI Thanks to @gijigae
  • ✨ feat: support the perplexity provider #1339
• Anthropic Thanks to @danielglh
  • ✨ feat: support anthropic as model provider #1409
• Mistral Thanks to @danielglh
  • ✨ feat: support mistral as model provider #1455
• Groq Thanks to @jcdiv47 and @pzcn
  • ✨ feat: add support for model provider Groq #1562
  • Add Groq localization #1570
  • ✨ feat: support model provider groq #1569
• OpenRouter: Thanks to @danielglh
  • ✨ feat: support openrouter as a new model provider #1572
• 01.AI（零一万物）: Thanks to @MapleEve
  • ✨ feat: support 01.AI as a new provider #1627
• TogetherAI: Thanks to @phsontung
  • ✨ feat: Support TogetherAI as new model provider #1709
• MiniMax Thanks to @lwshen
  • ✨ feat: support minimax as provider #2087
• DeepSeek: Thanks to @RubuJam
  • ✨ feat: Support DeepSeek as new model provider #2446
  • 🐛 fix: fix DeepSeek using wrong model ID #2484
• Qwen: Thanks to @justmywyw
  • ✨ feat: Support Qwen as new model provider #2715
• Stepfun(阶跃星辰)： by @arvinxx
  • 💄 style: add stepfun as a new provider #2803
• Baichuan (百川）： Thanks to @hezhijie0327
  • ✨ feat: Add Baichuan model provider #3097
• Taichu(太初): Thanks to @hezhijie0327
  • ✨ feat: Add Taichu model provider #3129
• 360 AI（360智脑） : Thanks to @hezhijie0327
  • ✨ feat: Add 360AI model provider #3130
• Novita AI: Thanks to @jasonhp
  • 💄 style: Add new model provider Novita AI #3177
• SiliconCloud: Thanks to @sxjeru
  • ✨ feat: Add SiliconCloud model provider #3092
• Fireworks AI Thanks to @hezhijie0327
  • ✨ feat: Add Fireworks AI Model Provider #3392
• Spark (讯飞星火): Thanks to @hezhijie0327
  • ✨ feat: Add Spark model provider #3098
• Github Models: Thanks to @CloudPassenger
  • ✨ feat: add Github Models provider #3869
• AI21 Labs: Thanks to @hezhijie0327
  • ✨ feat: add Ai21Labs model provider #3727
• Tencent Hunyuan(腾讯混元): Thanks to @hezhijie0327
  • ✨ feat: add Hunyuan(Tencent) model provider #4147
• Wenxin(文心一言): by @arvinxx
  • ✨ feat: add wenxin as a new provider #2802
  • ✨ feat: add wenxin model provider #4018
• Hugging Face Inference API: by @arvinxx
  • ✨ feat: add HuggingFace Model Provider #4225
• SenseNova: Thanks to @hezhijie0327
• ✨ feat: add SenseNova (商汤) model provider #4162
• Cloudflare AI: Thanks to @BrandonStudio , @sxjeru
  • ✨ feat: Support Cloudflare Workers AI #3402
  • ✨ feat: Support Cloudflare Workers AI #2966
• xAI : Thanks to @hezhijie0327
  • ✨ feat: add support xAI provider #4627

WIP：

• Cohere : Thanks to @ystoneman
  • Add initial implementation for Cohere provider support #2539
• Doubao: Thanks to @sxjeru
  • ✨ feat: Support Doubao Models #3055
• LM Studio: by @arvinxx
  • 🚧 wip: add lm studio #4483
• Google Vertex AI: by @arvinxx
  • ✨ feat: support vertex ai #4487

Plan：

• replicate

Important

If you have any requests for new model suppliers, feel free to comment below. When the number of upvotes exceeds 15, we will consider implementing it.

如果大家对于新模型供应商有诉求，欢迎在下方跟帖，当up数超过 15 时，我们会考虑将其实现

---

PRs Welcome 💞

• A demo PR of add OpenAI Compatible Provider: 📑 [DEMO] This is a demo PR of how to add an OpenAI Compatible Provider #2804

Tip

We also welcome PRs. We have received many contributions regarding model Providers, so we have maintained a library of icons for LLMs and Provider vendors, facilitating everyone's future development. https://github.com/lobehub/lobe-icons

同时欢迎 PR，我们已经收获到许多关于模型 Provider 的贡献, 因此我们维护了一个 LLMs 和 Provider 供应商的图标库，方便大家后续开发 https://github.com/lobehub/lobe-icons

[lesonky]

申请增加GCP的Vertex AI大模型服务，此类服务用服务账号身份认证。可以满足企业级数据合规要求。🙏

[arvinxx]

the upvote is more than 8, we will add Google Vertex AI Soon.

[AiharaMahiru]

There are still fewer API providers that offer the Claude family of models, I've used AWS before as a Claude API, but asking me to provide sensitive data such as credit card statements is unacceptable!

[Hewuliu]

请问是否可以兼容POE的api，这里面的模型种类也很丰富
https://creator.poe.com/docs/accessing-other-bots-on-poe#how-to-access-the-bot-query-api-directly

[BenjaminFourmaux]

Support Azure ML Studio Endpoint

Can you please support Azure ML Studio endpoints like https://*.inference.ml.azure.com

An exemple on how to use this endpoint (from Azure code snippet helper)

// Request data goes here
// The example below assumes JSON formatting which may be updated
// depending on the format your endpoint expects.
// More information can be found here:
// https://docs.microsoft.com/azure/machine-learning/how-to-deploy-advanced-entry-script
const requestBody = `{}`;

const requestHeaders = new Headers({"Content-Type" : "application/json"});

// Replace this with the primary/secondary key, AMLToken, or Microsoft Entra ID token for the endpoint
const apiKey = "";
if (!apiKey)
{
	throw new Error("A key should be provided to invoke the endpoint");
}
requestHeaders.append("Authorization", "Bearer " + apiKey)

// This header will force the request to go to a specific deployment.
// Remove this line to have the request observe the endpoint traffic rules
requestHeaders.append("azureml-model-deployment", "Phi-3-mini-4k-instruct");

const url = "https://model_workspace.westeurope.inference.ml.azure.com/score";

fetch(url, {
  method: "POST",
  body: requestBody,
  headers: requestHeaders
})
	.then((response) => {
	if (response.ok) {
		return response.json();
	} else {
		// Print the headers - they include the request ID and the timestamp, which are useful for debugging the failure
		console.debug(...response.headers);
		console.debug(response.body)
		throw new Error("Request failed with status code" + response.status);
	}
	})
	.then((json) => console.log(json))
	.catch((error) => {
		console.error(error)
	});

And an example to use my model (Phi-3-mini-4k-instruct) endpoint :

// Request body
{
  data: "<|system|>the system prompt<|end|><|user|>the user prompt<|end|><|assistant|>"
}

The input data structure use input format of the Phi3 model

Thanks !

[BrandonStudio]

Is ML API compatible with ai-inference or azureopenai?

[kac487]

SambaNova would make for a nice addition

[CXwudi]

Beside Azure ML Studio, there is also an Azure AI Studio at https://ai.azure.com/

Here I deployed the LLaMa 3.1 405B and this is the sample code from Azure code snippet:

# pip install azure-ai-inference
import os
from azure.ai.inference import ChatCompletionsClient
from azure.core.credentials import AzureKeyCredential

api_key = os.getenv("AZURE_INFERENCE_CREDENTIAL", '')
if not api_key:
  raise Exception("A key should be provided to invoke the endpoint")

client = ChatCompletionsClient(
    endpoint='https://Meta-Llama-3-1-405B-Instruct-bft.eastus.models.ai.azure.com',
    credential=AzureKeyCredential(api_key)
)

model_info = client.get_model_info()
print("Model name:", model_info.model_name)
print("Model type:", model_info.model_type)
print("Model provider name:", model_info.model_provider_name)

payload = {
  "messages": [
    {
      "role": "user",
      "content": "I am going to Paris, what should I see?"
    },
    {
      "role": "assistant",
      "content": "Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n\n1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n\nThese are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world."
    },
    {
      "role": "user",
      "content": "What is so great about #1?"
    }
  ],
  "max_tokens": 4096,
  "temperature": 0.8,
  "top_p": 0.1,
  "presence_penalty": 0
}
response = client.complete(payload)

print("Response:", response.choices[0].message.content)
print("Model:", response.model)
print("Usage:")
print("	Prompt tokens:", response.usage.prompt_tokens)
print("	Total tokens:", response.usage.total_tokens)
print("	Completion tokens:", response.usage.completion_tokens)

[BrandonStudio]

Is this compatible with azureOpenAI?

[razertory]

Cerebras AI

[lyc0826]

建议增加对“无问芯穹”的支持。
网址：infini-ai.com

[lyfhq]

LM studio +

[e1100x]

AIMLAPI.com

[BrandonStudio]

什么时候能更新一下模板pr呢？原来那个已经不适用了吧

[arvinxx]

我最近加一个 LM Studio，到时候以那个为新的好了

[CrazyGoudanli]

期望尽快能将豆包合并到主分支。

[CrazyGoudanli]

@arvinxx We hope to merge Doubao into the main branch as soon as possible.

[yordyi]

麻烦添加英伟达云大模型,新晋王者:https://build.nvidia.com/nvidia/llama-3_1-nemotron-70b-instruct?snippet_tab=Python

[deeeiznc]

vertex

[SAnBlog]

Cloudflare AI 何时并到主分支呢?

[arvinxx]

等 @BrandonStudio 什么时候改好了~ 😄

[BrandonStudio]

@arvinxx 我评论你还没回我呢

[arvinxx]

Cloudflare 已经合并上线

[duoartisan]

xAI's Grok-2 and Grok-mini. Thanks.

[jiongjiongJOJO]

xAI's Grok is compatible with OpenAI's SDK, and you only need to modify the base_url to https://api.x.ai/v1

[duoartisan]

the thing is, I'm already using the openai slot with official api and I'm just too lazy to set up the new-api/one-api thingy.

[githubdebugger]

While this xAI's endpoint are getting added for grok on lobe-chat, you can still use OpenAI's slot, if you have used up OpenAI's slot, you can use Groq's slot, which is again OpenAI compatible. Can even use Anthropic's slot, as xAI is compatible with Anthropic SDK too - https://x.ai/blog/api

[arvinxx]

we have support xAI in 1.31.0 with #4627 🚀

[534002646]

@arvinxx 希望可以增加个按钮，增加openai api兼容的模型