Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix some features in llava-style training #3835

Merged
merged 11 commits into from
May 27, 2024
Merged

Conversation

BUAADreamer
Copy link
Collaborator

@BUAADreamer BUAADreamer commented May 21, 2024

What does this PR do?

  1. support pretraining style like LLaVA by only fine-tuning mm_projector (freeze LM)
  2. add mllm_pt_demo data with only image-captioning single-turn data, see BUAADreamer/mllm_pt_demo
  3. modify target_modules to a regex string to only tune the Lora params in LM and mm_proj in LLaVA-style MLLM.

Usage

  1. pre-train: use sft. set tune_mm_proj=true and finetuning_type="full" and dataset="mllm_pt_demo"
  2. Lora's usage is the same as previously.

Before submitting

@hiyouga hiyouga added the pending This problem is yet to be addressed label May 21, 2024
@BUAADreamer BUAADreamer mentioned this pull request May 26, 2024
2 tasks
@BUAADreamer BUAADreamer changed the title support pretraining of mllm like llava style fix some features in llava-style training May 27, 2024
@hiyouga
Copy link
Owner

hiyouga commented May 27, 2024

LGTM

@hiyouga hiyouga merged commit 838f2fb into hiyouga:main May 27, 2024
1 check passed
@hiyouga hiyouga added solved This problem has been already solved and removed pending This problem is yet to be addressed labels May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
solved This problem has been already solved
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants