Is there any way to load GGUF models?

[InAnYan]

Title says everything :)

[yifanmai]

HI @InAnYan, we don't have official support for GGUF and we don't support running local in-process inference on GGUF models, but you can try one of these ways:

• vLLM has limited GGUF support. You can try running your GGUF model on vLLM and then use the HELM vLLM integration.
• LLaMA.cpp HTTP Server and Ollama server both have partial OpenAI API compatibility, so it may be possible to use it with HELM's OpenAIClient by setting base_url to your server's URL (example)