Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question on MixtralMOE SFT Loss #3085

Closed
1 task done
mces89 opened this issue Apr 1, 2024 · 1 comment
Closed
1 task done

Question on MixtralMOE SFT Loss #3085

mces89 opened this issue Apr 1, 2024 · 1 comment
Labels
solved This problem has been already solved

Comments

@mces89
Copy link

mces89 commented Apr 1, 2024

Reminder

  • I have read the README and searched the existing issues.

Reproduction

For the Mixtral MOE finetuning, I wonder whether it use some router balancing loss or use next token prediction loss only? Thanks.

Expected behavior

No response

System Info

No response

Others

No response
@mces89 mces89 changed the title Question on Mistral SFT Loss Question on MixtralMOE SFT Loss Apr 1, 2024
hiyouga added a commit that referenced this issue Apr 2, 2024
@hiyouga
add moe aux loss control #3085
b267aeb
@hiyouga
Copy link
Owner

hiyouga commented Apr 2, 2024

We set the coefficient of auxiliary router loss to 0.2 by default, you can change it by specifying --moe_aux_loss_coef 0.1 in fine-tuning

@hiyouga hiyouga added the solved This problem has been already solved label Apr 2, 2024
@hiyouga hiyouga closed this as completed Apr 2, 2024
tybalex added a commit to sanjay920/LLaMA-Factory that referenced this issue Apr 10, 2024
@tybalex @hiyouga
@khazic @SirlyDreamer @sanjay920 @0xez @marko1616 @rkinas @Tsumugii24 @yunhaoli24 @sliderSun @codemayq @kno10
merge into rubra-9b (#2)
6ee7683 
* fix packages

* Update wechat.jpg

* Updated README with new information

* Updated README with new information

* Updated README with new information

* Follow HF_ENDPOINT environment variable

* fix hiyouga#2346

* fix hiyouga#2777 hiyouga#2895

* add orca_dpo_pairs dataset

* support fsdp + qlora

* update readme

* update tool extractor

* paper release

* add citation

* move file

* Update README.md, fix the release date of the paper

* Update README_zh.md, fix the release date of the paper

* Update wechat.jpg

* fix hiyouga#2941

* fix hiyouga#2928

* fix hiyouga#2936

* fix Llama lora merge crash

* fix Llama lora merge crash

* fix Llama lora merge crash

* pass ruff check

* tiny fix

* Update requirements.txt

* Update README_zh.md

* release v0.6.0

* add arg check

* Update README_zh.md

* Update README.md

* update readme

* tiny fix

* release v0.6.0 (real)

* Update wechat.jpg

* fix hiyouga#2961

* fix bug

* fix hiyouga#2981

* fix ds optimizer

* update trainers

* fix hiyouga#3010

* update readme

* fix hiyouga#2982

* add project

* update readme

* release v0.6.1

* Update wechat.jpg

* fix pile datset hf hub url

* upgrade gradio to 4.21.0

* support save args in webui hiyouga#2807 hiyouga#3046

some ideas are borrowed from @marko1616

* Fix Llama model save for full param train

* fix blank line contains whitespace

* tiny fix

* support ORPO

* support orpo in webui

* update readme

* use log1p in orpo loss

huggingface/trl#1491

* fix plots

* fix IPO and ORPO loss

* fix ORPO loss

* update webui

* support infer 4bit model on GPUs hiyouga#3023

* fix hiyouga#3077

* add qwen1.5 moe

* fix hiyouga#3083

* set dev version

* Update SECURITY.md

* fix hiyouga#3022

* add moe aux loss control hiyouga#3085

* simplify readme

* update readme

* update readme

* update examples

* update examples

* add zh readme

* update examples

* update readme

* update vllm example

* Update wechat.jpg

* fix hiyouga#3116

* fix resize vocab at inference hiyouga#3022

* fix requires for windows

* fix bug in latest gradio

* back to gradio 4.21 and fix chat

* tiny fix

* update examples

* update readme

* support Qwen1.5-32B

* support Qwen1.5-32B

* fix spell error

* support hiyouga#3152

* rename template to breeze

* rename template to breeze

* add empty line

* Update wechat.jpg

* tiny fix

* fix quant infer and qwen2moe

* Pass additional_target to unsloth

Fixes hiyouga#3200

* Update adapter.py

* Update adapter.py

* fix hiyouga#3225

---------

Co-authored-by: hiyouga <[email protected]>
Co-authored-by: 刘一博 <[email protected]>
Co-authored-by: khazic <[email protected]>
Co-authored-by: SirlyDreamer <[email protected]>
Co-authored-by: Sanjay Nadhavajhala <[email protected]>
Co-authored-by: sanjay920 <[email protected]>
Co-authored-by: 0xez <[email protected]>
Co-authored-by: marko1616 <[email protected]>
Co-authored-by: Remek Kinas <[email protected]>
Co-authored-by: Tsumugii24 <[email protected]>
Co-authored-by: li.yunhao <[email protected]>
Co-authored-by: sliderSun <[email protected]>
Co-authored-by: codingma <[email protected]>
Co-authored-by: Erich Schubert <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
solved This problem has been already solved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mces89 @hiyouga