Megatron

[shaileshj2803]

Is it possible to give more details about which version of megatron and how to reference megatron during training. Detailed step by step instructions will be very very helpful. Thanks for the awesome work.

[dropreg]

Hi:

We updated the new version, which is easier to read.

We also give a step for efficient-finetuning using Megatron-LoRA (when you only have two 24G 3090s):

1. First, you need to download following (As shown in alpaca/scripts/utils/README.md)：

   • The LLaMA 7B model: consolidated.00.pth
   • The dictionary: alpaca/scripts/assert/dict.txt
   • The alpaca training data alpaca_data.json.

2. Processing model:
   python alpaca_lora/scripts/utils/process_llama_megatron_ckpt.py --llama-model-dir -llama-model-file --parallel-size

3. Processing data:
   bash prepare_llama_training_data.sh

4. Training step, As shown in alpaca/scripts/megatron_lora/README.md:
   bash alpaca/scripts/megatron_lora/run_train_megatron_lora.sh

5. Inference Step:

   • bash alpaca/scripts/megatron_lora/inference/run_inf_megatron_lora.sh
   • You can merge multiple megatron checkpoints into one:

     python merge_llama_megatron_ckpt.py
     bash alpaca/scripts/lora/inference/run_inf_hub.sh
     

In addition, please pay attention to modifying the parameters in above scripts, e.g., file path.