Add support for Phi3V

[ravi03071991]

PR to add support for Phi3V.

Fixes #1108

[ravi03071991]

NOTE: Testing is still pending.

[merrymercy]

I added you as a co-author in #2500, so your future commits will trigger CI automatically.

[merrymercy]

How did you test the correctness of this model locally? Did you compare the logits against HF implements, similar to this one? #2365 (comment)

[ravi03071991]

How did you test the correctness of this model locally? Did you compare the logits against HF implements, similar to this one? #2365 (comment)

Thanks for the pointer @merrymercy. I'll test it out.

Currently, it seems that Phi3VConfig and Phi3VModel are not part of the transformers library. Instead, they reside in the model files. As a result, I need to add them to the transformers library first to make this PR functional.

In other words, the following imports currently throw an error:

from transformers import Phi3VConfig, Phi3Vmodel

[merrymercy]

@ravi03071991 Can you fix the error in CI? https://github.com/sgl-project/sglang/actions/runs/12452217969/job/34760948018?pr=2383#step:4:984 Please make sure you can run it locally.

[ravi03071991]

@merrymercy yeah sure. I am still working on it.

[ravi03071991]

Updates:

1. Added an image processing step prior to image embedding.
2. Implemented image embedding.
3. Integrated Phi3VForCausalLM functionalities:
   - Loading weights
   - Embedding text and images
   - Combining text and image embeddings and passing them to the LLM.

TODO:

The logic for combining text and image embeddings remains unclear. I have imported the logic from Qwen2 VL, but it differs from the approach suggested in the Hugging Face Phi3V code base. Specifically, I am struggling to understand the rationale behind combining the embeddings using image offset and padding.

can someone help me here?

[zhyncs]

can someone help me here?

@yizhang2077 may help take a look. Thanks.

[yizhang2077]

can someone help me here?

In function pad_input_ids, it do padding for original input_ids with image tokens and add record image offsets here (where a image embedding start from). I think you can do some modification here, and in forward you can replace embedding by using image embedding and image offsets

[ravi03071991]

can someone help me here?

In function pad_input_ids, it do padding for original input_ids with image tokens and add record image offsets here (where a image embedding start from). I think you can do some modification here, and in forward you can replace embedding by using image embedding and image offsets

Thanks @yizhang2077. The pad_input_ids seems to be defined in qwen2_vl but it does not seem to be used?

[yizhang2077]

Thanks @yizhang2077. The pad_input_ids seems to be defined in qwen2_vl but it does not seem to be used?

It is used in pad_input_ids_func here

[ravi03071991]

It is used in pad_input_ids_func here

Oh, I see. I missed that. The padding function seems different for qwen2_vl and llava. Is it specific to the model, and should it be checked on HF?

[ravi03071991]

Looks like the model provider has some padding logic here.

[yizhang2077]

Oh, I see. I missed that. The padding function seems different for qwen2_vl and llava. Is it specific to the model, and should it be checked on HF?

I think it is specific to the model since how to do padding depends on model implementation

[ravi03071991]

@yizhang2077 / @zhyncs I think I kinda stuck here:

1. pad_input_ids requires num_img_tokens.
2. The current implementation computes num_img_tokens based on the image processor, which is called inside the forward method.

I’m a bit confused about how to proceed from here. Could you help me here?

[ravi03071991]

@yizhang2077 / @zhyncs I think I kinda stuck here:

1. pad_input_ids requires num_img_tokens.
2. The current implementation computes num_img_tokens based on the image processor, which is called inside the forward method.

I’m a bit confused about how to proceed from here. Could you help me here?

Okay. I figured out the best way to solve this is by moving image_processing step into pad_input_ids and store the necessary information in image_inputs so that they can be used later in forward.