Which version of transformer engine should I use, when I try to open ub_tp_comm_overlap?

[sallyjunjun]

I am using NeMo with version v2.0.0rc0. When I set ub_tp_comm_overlap to true with tp and sp 2, I met the following error:

The version of transformer engine 1.6.0+c81733f. Should I update to newer te version?

When I update transformer engine to 1.13.0+e5edd6c. There occurs another error in NeMo:

CUDA 11.8 should be used in NeMo. But transformer engine in 1.13.0+e5edd6c version requires CUDA newer than 12.0.

I'm stuck in these version issues.
Could you please tell me which version of NeMo and TE and CUDA should I use to enable tp_comm_overlap feature?

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.