How to send CURL request for MLLM models?

[anidh]

Reminder

• I have read the README and searched the existing issues.

Reproduction

Hi There,
I am trying to run inference of MLLM models using the CUDA_VISIBLE_DEVICES=0 llamafactory-cli api training_recipes/infer.yaml command which starts a local open ai like api server.
I want to now send images with prompt to this server to do inference.
How can I send a CURL request for the same?

Expected behavior

I would like to be able to send a curl request with image and get status 200 response.

System Info

• transformers version: 4.41.0
• Platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.23.1
• Safetensors version: 0.4.3
• Accelerate version: 0.30.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Parallel

Others

No response

[anidh]

@BUAADreamer @hiyouga

[hiyouga]

The openai API does not support vision models yet, it will supported in the future version

[anidh]

@hiyouga Thanks for the prompt response. So right now there is now way to feed multiple images to the model? Or do it programmatically through python?

[anidh]

The only way I am guessing right now is to load the weights in HF library and do inference using that?
@hiyouga

[hiyouga]

the model we supported accepts only one image at inference, you can use llamafactory-cli webchat to launch a web interface

[anidh]

Thanks!

[hiyouga]

update: now api mode supports image inputs

[anidh]

Thanks for the update!! 😀