How does ZeRO DDP work with Tensor Parallelism?

[heya5]

I think TP has no memory redundancy for model data, so I'm confused about how ZeRO DDP works with TP.
In your gpt example ,

# The following options only valid when DISTPAN="colossalai"
export TPDEGREE=2
export GPUNUM=4
export PLACEMENT='cpu'
export USE_SHARD_INIT=False

Does this mean that we use 4 GPUs, 2 of which form a group to do tensor parallelism, and then we do ZeRO DDP on those two groups?

[feifeibear]

Yes, you are right!