[Feature]: add DoRA support

[cmhungsteve]

🚀 The feature, motivation and pitch

The recent ICML'24 Oral paper, DoRA, has shown consistent improvement over LoRA on various tasks (LLM, MLLM, etc.) and backbones (LLaMA, LLaVA, etc.)
[Code] https://github.com/NVlabs/DoRA

DoRA has also been integrated into various open-source library/framework:

• Hugging Face: peft, diffusers
• Apple: MLX
• Meta: torchtune
• NVIDIA: NeMo
• Unsloth AI: unsloth
• Answer.AI: QDoRA + FSDP
• LLaMA-Factory
• LyCORIS
• ..., etc.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

Thank you very much for bringing this feature. We will consider supporting it.

[TC10127]

I also need vllm to support dora !!!

[PacinoAn]

We need this feature too，this will help a lot!