Megatron train

[germanjke]

Hi!

I'm asking about train_megatron.py

are you using parallel mechanisms from fairscale and I don't see any sources of megatron library

it's your custom megatron with fairscale?

[germanjke]

fairscale.nn.model_parallel is forked from [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), Copyright 2020, NVIDIA CORPORATION, licensed under [Apache License](http://www.apache.org/licenses/LICENSE-2.0).

I have found this in fariscale repo, so you mean, you just import some Megatron-LM from fairscale.nn.model_parallel?

[germanjke]

from fairscale.nn.model_parallel.layers import ( ParallelEmbedding, RowParallelLinear, ColumnParallelLinear, )
so yes, we are using this from fairscale, but it's fairscale forked by Megatron-LM, so we are using Megatron-LM, I think it's your logic