[v.2.3.0] Release Tracker

[atalman]

🐛 Describe the bug

We cut a release branch for the 2.3.0 release.

Our plan from this point from this point is roughly:

• Phase 1 (until 4/1/24): work on finalizing the release branch
• Phase 2 (after 4/1/24): perform extended integration/stability/performance testing based on Release Candidate builds.

This issue is for tracking cherry-picks to the release branch.

Cherry-Pick Criteria

Phase 1 (until 4/1/24):

Only low-risk changes may be cherry-picked from main:

1. Fixes to regressions against the most recent minor release (e.g. 2.2.x for this release; see module: regression issue list)
2. Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
3. Critical fixes to new features introduced in the most recent minor release (e.g. 2.2.x for this release)
4. Test/CI fixes
5. Documentation improvements
6. Compilation fixes or ifdefs required for different versions of the compilers or third-party libraries
7. Release branch specific changes (e.g. change version identifiers)

Any other change requires special dispensation from the release managers (currently @atalman, @osalpekar, @huydhn, @malfet). If this applies to your change please write "Special Dispensation" in the "Criteria Category:" template below and explain.

Phase 2 (after 4/1/24):

Note that changes here require us to rebuild a Release Candidate and restart extended testing (likely delaying the release). Therefore, the only accepted changes are Release-blocking critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

Changes will likely require a discussion with the larger release team over VC or Slack.

Cherry-Pick Process

1. Ensure your PR has landed in master. This does not apply for release-branch specific changes (see Phase 1 criteria).

2. Create (but do not land) a PR against the release branch.

   # Find the hash of the commit you want to cherry pick
   # (for example, abcdef12345)
   git log
   
   git fetch origin release/2.3
   git checkout release/2.3
   git cherry-pick -x abcdef12345
   
   # Submit a PR based against 'release/2.3' either:
   # via the GitHub UI
   git push my-fork
   
   # via the GitHub CLI
   gh pr create --base release/2.3

3. Make a request below with the following format:

Link to landed trunk PR (if applicable):
* 

Link to release branch PR:
* 

Criteria Category:
* 

1. Someone from the release team will reply with approved / denied or ask for more information.
2. If approved, someone from the release team will merge your PR once the tests pass. Do not land the release branch PR yourself.

NOTE: Our normal tools (ghstack / ghimport, etc.) do not work on the release branch.

Please note HUD Link with branch CI status and link to the HUD to be provided here.
HUD

Versions

2.3.0

[snadampal]

Link to main PR (if applicable):

• aarch64: cherrypick onednn pr#1768 to improve torch.compile perf builder#1716
• aarch64: apply the cherrypicked onednn PR-1768 builder#1717

Link to 2.3 release branch PR:

• aarch64: cherrypick onednn pr#1768 to improve torch.compile performance builder#1728
• aarch64: apply the cherrypicked onednn PR-1768 builder#1730

Criteria Category:
The PRs improve the torch.compile performance on aarch64 linux systems substantially.

---

@malfet: Please do not cherry-pick those changes, we should revert it on builder on trunk and coordinate with oneDNN team to get those changes into release branch and/or wait until the next release

[atalman]

Link to landed trunk PR (if applicable):

• NA

Link to release branch PR:

• [RELEASE ONLY CHANGES] Apply release only changes Release 2.3 #121726
• [Release Only] Build triton using pinned version rather branch #121765
• [RELEASE ONLY CHANGES] Apply release only changes Release 2.3 #121813
• [RELEASE ONLY CHANGES] Increase timeout for linux binary jobs, fix workflow lint #121851
• [Release only] Release 2.3 start using triton package from pypi #123580

Criteria Category:

• Release only required changes

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• CI: Specify libc and libstdcxx versions in conda environments #121556

Link to release branch PR:

• CI: Specify libc and libstdcxx versions in conda environments #121929

Criteria Category:

• Critical fix: Without this we get mismatches between the GLIBC and GLIBCXX ABI used by conda packages vs pytorch.

---

@atalman merged

[NmomoN]

Link to landed trunk PR (if applicable):

• Add 'profiler/python' to the package.' #121892

Link to release branch PR:

• Add 'profiler/python' to the package.' #122065

Criteria Category:

• Fixes to new features being introduced in 2.3.0 release.

@zdevito

---

@atalman Please provide some details on what issue you are trying to address and what is the associated feature

[shunting314]

Link to landed trunk PR (if applicable):
N/A

Link to release branch PR:

• necessary change to make torch2.3 work with triton2.2 #122139

Criteria Category:

• need this to make pytorch release work with the corresponding triton release

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• N/A

Link to release branch PR:

• Revert "[Release Only] Build triton using pinned version rather branch" builder#1751
• Triton wheel build using 2.3.x branch #122403

Criteria Category:

• need this to make pytorch release work with the corresponding triton release

---

@atalman merged

[wz337]

Link to landed trunk PR (if applicable):

• Back out "[DeviceMesh] Add support for nD slicing (#119752)" #121763

Link to release branch PR:

• [CherryPick] Back out "[DeviceMesh] Add support for nD slicing (#119752)" (#121763) #122495

Criteria Category:

• Cherry-picking a revert. Bug fix.

---

@atalman merged

[SS-JIA]

Link to landed trunk PR (if applicable):

• [ET-VK] Enable Dynamic shape support via tensor virtual and physical resizing #121598

Link to release branch PR:

• [ET-VK] Enable Dynamic shape support via tensor virtual and physical resizing #122634

Criteria Category:

• This does not quite fit into any of the cherry pick criterias, however, it implements features required for ExecuTorch alpha release. It is also low risk as it only impacts the PyTorch-Vulkan backend. cc: @huydhn

[guangy10]

Link to landed trunk PR (if applicable):

• remove torchao dependency #122524

Link to release branch PR:

• remove torchao dependency #122635

Criteria Category:

A critical fix for ExecuTorch. Need to be merged so that ExecuTorch v0.2 can be compatible with PyTorch v2.3. cc: @huydhn

---

@huydh merged

[JacobSzwejbka]

Link to landed trunk PR (if applicable):

#122246

Link to release branch PR:

#122655

Criteria Category:

A critical fix for ExecuTorch: A new pass that ExecuTorch uses to allow mutable state with custom ops. Need to be merged so that ExecuTorch v0.2 can be compatible with PyTorch v2.3 cc: @huydhn

---

@huydhn merged

[JacobSzwejbka]

Link to landed trunk PR (if applicable):

#121990

Link to release branch PR:

#122654

Criteria Category:

A critical fix for ExecuTorch: Patching auto functionalized which ExecuTorch relies on for exporting llama. Need to be merged so that ExecuTorch v0.2 can be compatible with PyTorch v2.3 cc: @huydhn

---

@huydhn merged

[JacobSzwejbka]

Link to landed trunk PR (if applicable):

#122683

Link to release branch PR:

#122721

Criteria Category:

A critical fix for ExecuTorch: Makes dead code elimination pass not trash the output of capture_pre_autograd_graph if it contains kv cache mutation. Need to be merged so that ExecuTorch v0.2 can be compatible with PyTorch v2.3 cc: @huydhn

---

@huydhn merged

[jataylo]

Link to landed trunk PR (if applicable):

• Not applicable, release specific change.

Link to release branch PR:

• [ROCm] Update triton rocm branch to release/2.3.x #122493

Criteria Category:

• Release specific change for ROCm to use release/2.3.x triton branch cc: @atalman @jithunnair-amd

---

@atalman merged

[guangy10]

Link to landed trunk PR (if applicable):

• Proper view support for jagged layout NestedTensor #113279

Link to release branch PR:

• Proper view support for jagged layout NestedTensor #122854

Criteria Category:

A critical fix for ExecuTorch. W/o this PR, PyTorch v2.3 will not be compatible with ExecuTorch, even the simples add module will fail. What this PR does is adding new ops to native_functions.yaml and they are covered by tests. It’s relatively low risk as these ops are not used by many component except for ExecuTorch.

---

@atalman wrote: @guangy10 This is all looks like feature work. We normally accept only critical fixes at this point:

• Fixes to regressions against the most recent minor release (e.g. 2.2.x for this release; see module: regression issue list)
• Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks
• Critical fixes to new features introduced in the most recent minor release (e.g. 2.2.x for this release)

cc: @malfet @albanD

@atalman I don't think it's a feature work. I would category this to the "critical fix: compatibility" because it does break the compatibility with ExecuTorch w/o this fix.

---

@huydhn merged

[SS-JIA]

Link to landed trunk PR (if applicable):

• [ATen-Vulkan][EZ] Small fixes: fix gpu size calculation and Half scalartype ctype mapping #122096

Link to release branch PR:

• [ATen-Vulkan][EZ] Small fixes: fix gpu size calculation and Half scalartype ctype mapping #122861

Criteria Category:

• This does not quite fit into any of the cherry pick criterias, however, it implements fixes to PyTorch-Vulkan required for ExecuTorch alpha release. It is also low risk as it only impacts the PyTorch-Vulkan backend. cc: @huydhn

---

@atalman wrote: @SS-JIA For cherry-pick to be considered for a release it needs to have unit test with the cherry-pick

---

@SS-JIA wrote: @atalman the `linux-vulkan-focal-py3.11-clang10 / test job tests PyTorch Vulkan and is passing on the PR. Is that sufficient?

[htyu]

Link to landed trunk PR:

• [inductor][Autotune] Add matrix_instr_nonkdim to triton_meta #122852

Link to release branch PR:

• [inductor][Autotune] Add matrix_instr_nonkdim to triton_meta (#122852) #122906

Criteria Category:
A critical fix for matmul performance on RocM.

---

@atalman wrote: Please provide some unit tests for this cherry-pick

[jithunnair-amd]

Link to landed trunk PR (if applicable):

• Fix performance regression and memory storage handling of Flash Attention on ROCM #122857
• Remove cuda dependencies when building AOTriton #122982

Link to release branch PR:

• Fix performance regression and memory storage handling of Flash Attention on ROCM (#122857) #122967
• Remove cuda dependencies when building AOTriton (#122982) #123179

Criteria Category:

1. A critical fix for Flash Attention performance on ROCm
2. A fix for intermittent build issues observed on CI when building AOTriton

cc @xinyazhang @groenenboomj

---

@atalman merged

[chunyuan-w]

Link to landed trunk PR (if applicable):

• Inductor cpp wrapper: fix dtype of ShapeAsConstantBuffer #122297

Link to release branch PR:

• [CherryPick] Inductor cpp wrapper: fix dtype of ShapeAsConstantBuffer (#122297) #123064

Criteria Category:

• Fixes to regressions of Inductor CPP wrapper dynamic shape inference against the 2.2 release: [inductor][cpu] fastNLP_Bert, hf_BigBird, hf_Reformer, soft_actor_critic fp32 Dynamic shape CPP wrapper accuracy crashed #122292

---

@atalman merged

[wz337]

Link to landed trunk PR (if applicable):

• [DeviceMesh] Cache and reuse sliced result #122975

Link to release branch PR:

• [Cherrypick][DeviceMesh] Cache and reuse sliced result (#122975) #123073

Criteria Category:

• Critical fix for DeviceMesh to avoid recreating nccl communicators multiple times which could lead to deadlocks.

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Upgrade submodule oneDNN to v3.3.6 #122164
• Revert "aarch64: patch mkl-dnn for xbyak crashes due to /sys not accessible (#1648)" builder#1765

Link to release branch PR:

• Upgrade submodule oneDNN to v3.3.6 for release/2.3 (#122164) #122930
• Revert: aarch64: patch mkl-dnn for xbyak crashes due to /sys not accessible #1648 builder#1767

Criteria Category:

• Critical Fixes

---

@atalman merged

[wanchaol]

Link to landed trunk PR (if applicable):

• nn.Module: use swap_tensors for Tensor subclasses #122755
• Fix swap_tensors path in _apply for modules that inherit from RNNBase (RNN, GRU, LSTM) #122800

Link to release branch PR:

• nn.Module: use swap_tensors for Tensor subclasses (#122755) #123106
• Fix swap_tensors path in _apply for modules that inherit from RNNBase… #123116

Criteria Category:

• Critical fixes for: silent correctness on tensor subclasses

edit: @mikaylagawarecki added cherry pick of #122800 per discussion on #123106 since we are cherry picking #122755 as a critical fix for wrapper tensor subclasses

---

@huydhn merged

[huydhn]

Link to landed trunk PR (if applicable):

• Pin protobuf to 3.20.2 on macOS #121918

Link to release branch PR:

• Pin protobuf to 3.20.2 on macOS #123197

Criteria Category:

• CI fix

---

@atalman merged

[ajindal1]

Link to landed trunk PR (if applicable):

• fix breaking changes for ONNX Runtime Training #122000

Link to release branch PR:

• fix breaking changes for ONNX Runtime Training #123271

Criteria Category:

• Backward compatibility with ONNX Runtime Training.

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Fix torch.clamp in MPS to handle NaN correctly #121381
• [MPS] Fwd-fix for clamp regression #122148

Link to release branch PR:

• Fix torch.clamp in MPS to handle NaN correctly #122785
• [MPS] Fwd-fix for clamp regression #123383

Criteria Category:

• Critical fixes Inconsistent Behavior of torch.clamp with NaN Values on MPS Device #120899

---

@huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• [MPS] Fix naive matmul for BFloat16 #121731

Link to release branch PR:

• [MPS] Fix naive matmul for BFloat16 #123289

Criteria Category:

• Critical fixes RuntimeError: Undefined type BFloat16 from matmul above certain sizes (maybe MPS only) #121583

---

@atalman merged

[atalman]

Link to landed trunk PR (if applicable):

• Make PyTorch compilable against upcoming Numpy-2.0 #121880
• Upgrade submodule pybind to 2.12.0 #122899
• [CI] Test that NumPy-2.X builds are backward compatible with 1.X #122157
• Use numpy 2.0.0rc1 in CI #123286
• Set py3.x build-environment name consistently #122247

Link to release branch PR:

• Make PyTorch compilable against upcoming Numpy-2.0 #123380
• Upgrade submodule pybind to 2.12.0 #123355
• [CI] Test that NumPy-2.X builds are backward compatible with 1.X #123354
• Use numpy 2.0.0rc1 in CI #123356
• [Release/2.3] Set py3.x build-environment name consistently #123446

Criteria Category:

• Numpy 2.0 pytorch/pytorch

---

@atalman @malfet merged

[atalman]

Link to landed trunk PR (if applicable):

• Add numpy 2.0.0rc1 support to conda and wheels builder#1768
• Add numpy 2.0.0rc1 support to conda and wheels builder#1769
• pytorch/builder@dcc58c3
• Update nightly wheel build numpy to 2.0 builder#1747
• [BE] Remove macos x86 build statements, remove python <3.8 builder#1745

Link to release branch PR:

• Numpy 2.0.0 support PRs cherry-pick builder#1770

Criteria Category:

• Numpy 2.0 support for pytorch/builder

---

@atalman merged

[BowenBao]

Link to landed trunk PR (if applicable):

• [ONNX] beartype to emit warning instead of error by default #123205

Link to release branch PR:

• [ONNX] beartype to emit warning instead of error by default #123363

Criteria Category:

• Critical fixes: we found beartype 0.18 released on 4/2/2024 would uncover a type annotation issue in current ONNX related code and raise error for every export call. This fix changes the behavior from raising an error to warning.

---

@atalman @huydhn merged

[atalman]

Link to landed trunk PR (if applicable):

• update submodule onnx==1.16.0 #123125
• pytorch/builder@bfc38ef

Link to release branch PR:

• update submodule onnx==1.16.0 #123387
• Bump MACOSX_DEPLOYMENT_TARGET to 10.15  builder#1773

Criteria Category:

• Critical fix Integrate with ONNX 1.16.0 release branch #121258

---

@atalman @malfet merged

[atalman]

Link to landed trunk PR (if applicable):

• [Wheel] Change libtorch_cpu OpenMP search path #123417

Link to release branch PR:

• [Wheel] Change libtorch_cpu OpenMP search path (#123417) #123442

Criteria Category:

• Critical fix torch.compile is not usable on MacOS out of box #122705

---

@atalman merged

[malfet]

Link to landed trunk PR (if applicable):

• Fix for MPS regression in #122016 and #123178 #123234

Link to release branch PR:

• Fix for MPS regression in #122016 and #123178 #123385

Criteria Category:

• Silent correctness fix/regression [MPS] Regression from macOS 14.3 to 14.4 in PyTorch 2.2.0/2.2.1 #122016

---

@atalman merged

[cyyever]

Link to landed trunk PR (if applicable):

• Fix order conditioning of norm kernel #122874

Link to release branch PR:

• N/A

Criteria Category:

• Critical fix: A bug fix for norm kernel when called with value = 1.

---

@atalman wrote: Please note, at this point we already generated final RC. Is there an issue associated with this PR ?
@cyyever wrote: No such an issue.

[WeizhuoZhang-intel]

Link to landed trunk PR (if applicable):

• Add skip models by device in Dynamo Test #122591
• Fix skip logic bug in dynamo benchmark runner #123544

Link to release branch PR:

• Add skip models by device in Dynamo Test #123827

Criteria Category:

• Critical fix: Add skip model by device in Dynamo test for torchbench suite. Some of those models have OOM issue and other models did not support CPU currently, which will help to improve the CPU side passrate when using runner.py.

---

@atalman wrote: Hi @WeizhuoZhang-intel we already built final rc. this only affects torchbench suite. Is running this on nightly is possibility ? This cherry-pick does not fit the criteria for Critical fixes for: silent correctness, backwards compatibility, crashes, deadlocks, (large) memory leaks

[huydhn]

Link to landed trunk PR (if applicable):

• Add submodule input for build_wheels_*.yaml test-infra#5095

Link to release branch PR:

• cherry-pick: Add submodule input for build_wheels_*.yaml (#5095) test-infra#5096

Criteria Category:

• CI fix to unblock ExecuTorch build

---

@huydhn merged