v0.2.0: Web UI Refactor, LongLoRA

New features

• Support LongLoRA for the LLaMA models
• Support training the Qwen-14B and InternLM-20B models
• Support training state recovery for the all-in-one Web UI
• Support Ascend NPU by @statelesshz in #975
• Integrate MMLU, C-Eval and CMMLU benchmarks

Modifications

• Rename repository to LLaMA Factory (former LLaMA Efficient Tuning)
• Use the cutoff_len argument instead of max_source_length and max_target_length #944
• Add a train_on_prompt option #1184

Bug fix

• Fix numeric error caused by the layer norm dtype in 84b7486 [1]
• Fix bugs in PPO Trainer by @mmbwf in #900
• Fix #424 #762 #814 #887 #913 #1000 #1026 #1032 #1064 #1068 #1074 #1086 #1097 #1176 #1177 #1190 #1191

[1] huggingface/transformers#25598 (comment)

v0.1.8: FlashAttention-2 and Baichuan2

New features

• Support FlashAttention-2 for LLaMA models. (RTX4090, A100, A800 or H100 GPU is required)
• Support training the Baichuan2 models
• Use right-padding to avoid overflow in fp16 training (also mentioned here)
• Align the computation method of the reward score with DeepSpeed-Chat (better generation)
• Support --lora_target all argument which automatically finds the applicable modules for LoRA training

Bug fix

• Use efficient EOS tokens to align with the Baichuan training ( baichuan-inc/Baichuan2#23 )
• Remove PeftTrainer to save model checkpoints in DeepSpeed training
• Fix bugs in web UI by @beat4ocean in #596 by @codemayq in #644 #651 #678 #741 by @kinghuin in #786
• Add dataset explanation by @panpan0000 in #629
• Fix a bug in the DPO data collator
• Fix a bug of the ChatGLM2 tokenizer in right-padding
• #608 #617 #649 #757 #761 #763 #809 #818

v0.1.7: Script Preview and RoPE Scaling

New features

• Preview training script in Web UI by @codemayq in #479 #511
• Support resuming from checkpoints by @niuba in #434 (transformers>=4.31.0 required)
• Two RoPE scaling methods: linear and NTK-aware scaling for LLaMA models (transformers>=4.31.0 required)
• Support training the ChatGLM2-6B model
• Support PPO training in bfloat16 data type #551

Bug fix

• Unusual output of quantized models #278 #391
• Runtime error in distributed DPO training #480
• Unexpected truncation in generation #532
• Dataset streaming error in pre-training #548 #549
• Tensor shape mismatch in PPO training using ChatGLM2 #527 #528
• #475 #476 #478 #481 #494 #551

v0.1.6: DPO Training and Qwen-7B

• Adapt DPO training from the TRL library
• Support fine-tuning the Qwen-7B, Qwen-7B-Chat, XVERSE-13B, and ChatGLM2-6B models
• Implement the "safe" ChatML template for Qwen-7B-Chat
• Better Web UI
• Pretty readme by @codemayq #382
• New features: #395 #451
• Fix InternLM-7B inference #312
• Fix bugs: #351 #354 #361 #376 #408 #417 #420 #423 #426

v0.1.5: Patch release

• Fix LLaMA-2 template #307
• Fix bug in preprocessing 968ce0d
• Fix #294 #296

v0.1.4: Dataset Streaming

• Support dataset streaming
• Fix LLaMA-2 #268
• Fix DeepSpeed ZeRO-3 model save #274
• Fix #242 #284

v0.1.3: Patch release

release v0.1.3

v0.1.2: LLaMA-2 Models

• Support LLaMA-2 (good issue #202 )
• Advanced configurations in Web UI
• Fix API (downgrade pydantic<2.0.0)
• Fix baichuan lora hparam #194 #212
• Fix padding #196
• Fix ZeRO-3 #199
• Allow pass args to app #213
• Code simplification
• Add ShareGPT dataset

v0.1.1

• Web UI: source_prefix, max_length, dev set
• Bug fix: reward token #179
• Update template #171 #177
• Bug fix: replace the Literal type with Enum for pydantic [1] #176
• Add Web demo #180

[1] pydantic/pydantic#5821, fastapi/sqlmodel#67

v0.1.0: All-in-one Web UI

• Fix gradient accumulation in PPO Trainer hiyouga/ChatGLM-Efficient-Tuning#299
• All-in-one Web UI by @hiyouga , @KanadeSiina and @codemayq