Use osx-arm64 packages on Apple silicon #379

victorlin · 2024-07-12T19:07:18Z

Description of proposed changes

Previously the osx-64 packages were used because osx-arm64 was not supported. Rosetta 2 is no longer required for the conda runner on Apple silicon.

Checklist

Merge Add build for osx-arm64 conda-base#80
nextstrain setup --force conda on an M1 Mac runs successfully, pulling from osx-arm64 channels
Test locally: run a draft ebola workflow using this runtime
Checks pass
- Failures of CI / test-source are unrelated.

Previously the osx-64 packages were used because osx-arm64 was not supported. Rosetta 2 is no longer required for the conda runner on Apple silicon.

nextstrain/cli/runner/conda.py

tsibley · 2025-02-06T20:23:52Z

nextstrain/cli/runner/conda.py

@@ -610,9 +605,10 @@ def package_distribution(channel: str, package: str, version: str = None, label:

    if (system, machine) == ("Linux", "x86_64"):
        subdir = "linux-64"
-    elif (system, machine) in {("Darwin", "x86_64"), ("Darwin", "arm64")}:
-        # Use the x86 arch even on arm (https://docs.nextstrain.org/en/latest/reference/faq.html#why-intel-miniconda-installer-on-apple-silicon)


Can this doc page go away too? (Later, non-blocking)

tsibley

LGTM. I haven't tested the changes, but trust you have.

tsibley · 2025-02-06T20:43:54Z

Oh, wait. How does this interact with nextstrain update conda when the existing environment is osx-64 and the package to-be-installed is osx-arm64?

victorlin · 2025-02-06T20:55:18Z

Oh good thinking, I'll check after lunch.

victorlin · 2025-02-06T21:50:24Z

First, I installed an earlier version so that nextstrain update will pull 20250205T180501Z which has an osx-arm64 variant.

NEXTSTRAIN_CONDA_BASE_PACKAGE="nextstrain-base ==20250203T212457Z" nextstrain setup --force conda

Then I ran nextstrain update conda with this PR's changes. It ran fine and pulled the osx-64 variant. This makes sense, because while package_distribution() will detect arch=arm64 and return a value based on that, the code path taken only takes the version tag, which is platform-agnostic, and passes it to micromamba update.

All this to say, this is not a breaking change as long as we continue to provide both osx-arm64 and osx-64 variants in conda-base. Having done this testing, I've added a changelog entry:

The Conda runtime now supports native installation for Apple silicon (e.g. M1 chip) devices, which improves performance when using the runtime. To take full advantage of this feature, force a fresh installation with nextstrain setup --force conda.

Full output of `nextstrain update conda`

Checking for newer versions of Nextstrain CLI…

nextstrain-cli is up to date!

Updating conda runtime…
Updating Conda package nextstrain-base from 20250203T212457Z to 20250205T180501Z…

Updating Conda packages in /Users/victor/.nextstrain/runtimes/conda/env…
  - nextstrain-base ==20250205T180501Z
bioconda/noarch (check zst)                         Checked  0.1s
bioconda/osx-64 (check zst)                         Checked  0.1s
nextstrain/noarch (check zst)                       Checked  0.2s
nextstrain/osx-64 (check zst)                       Checked  0.3s
bioconda/osx-64                                      4.1MB @  14.6MB/s  0.3s
nextstrain/osx-64                                   39.4kB @ 135.1kB/s  0.3s
nextstrain/noarch                                  116.0 B @ 258.0 B/s  0.6s
bioconda/noarch                                      4.7MB @   5.9MB/s  0.5s
conda-forge/noarch                                  19.0MB @  19.0MB/s  1.0s
conda-forge/osx-64                                  36.4MB @  29.0MB/s  1.3s
Transaction

  Prefix: /Users/victor/.nextstrain/runtimes/conda/env

  Updating specs:

   - nextstrain-base==20250205T180501Z


  Package                     Version  Build              Channel          Size
─────────────────────────────────────────────────────────────────────────────────
  Upgrade:
─────────────────────────────────────────────────────────────────────────────────

  - awscli                    2.23.11  py311h6eed73b_0    conda-forge      13MB
  + awscli                    2.23.13  py311h6eed73b_0    conda-forge      13MB
  - paramiko                    3.5.0  pyhd8ed1ab_1       conda-forge     161kB
  + paramiko                    3.5.1  pyhd8ed1ab_0       conda-forge     161kB
  - yte                         1.5.6  pyha770c72_0       conda-forge      14kB
  + yte                         1.5.7  pyha770c72_0       conda-forge      14kB
  - nextstrain-base  20250203T212457Z  h0dc7051_0_locked  nextstrain       13kB
  + nextstrain-base  20250205T180501Z  h0dc7051_0_locked  nextstrain       12kB

  Summary:

  Upgrade: 4 packages

  Total download: 13MB

─────────────────────────────────────────────────────────────────────────────────



Transaction starting
paramiko                                           161.0kB @   2.5MB/s  0.1s
yte                                                 14.4kB @ 165.2kB/s  0.1s
awscli                                              13.3MB @  71.6MB/s  0.2s
nextstrain-base                                     12.4kB @  20.9kB/s  0.6s
Changing awscli-2.23.11-py311h6eed73b_0 ==> awscli-2.23.13-py311h6eed73b_0
Changing paramiko-3.5.0-pyhd8ed1ab_1 ==> paramiko-3.5.1-pyhd8ed1ab_0
Changing yte-1.5.6-pyha770c72_0 ==> yte-1.5.7-pyha770c72_0
Changing nextstrain-base-20250203T212457Z-h0dc7051_0_locked ==> nextstrain-base-20250205T180501Z-h0dc7051_0_locked

Transaction finished

To activate this environment, use:

    micromamba activate /Users/victor/.nextstrain/runtimes/conda/env

Or to execute a single command in this environment, use:

    micromamba run -p /Users/victor/.nextstrain/runtimes/conda/env mycommand

Cleaning up…
Collect information..
Cleaning index cache..
Cleaning lock files..
  Package file                                                 Size
─────────────────────────────────────────────────────────────────────
  /Users/victor/.nextstrain/runtimes/conda/micromamba/pkgs
─────────────────────────────────────────────────────────────────────

  awscli-2.23.13-py311h6eed73b_0.conda                         13MB
  nextstrain-base-20250205T180501Z-h0dc7051_0_locked.conda     12kB
  paramiko-3.5.1-pyhd8ed1ab_0.conda                           161kB
  yte-1.5.7-pyha770c72_0.conda                                 14kB

  /Users/victor/.nextstrain/runtimes/conda/.mamba/pkgs
─────────────────────────────────────────────────────────────────────


  
─────────────────────────────────────────────────────────────────────

  Total size:                                                  13MB
Cleaning tarballs..
  Package folder                                         Size
───────────────────────────────────────────────────────────────
  /Users/victor/.nextstrain/runtimes/conda/micromamba/pkgs
───────────────────────────────────────────────────────────────

  awscli-2.23.13-py311h6eed73b_0                        127MB
  nextstrain-base-20250205T180501Z-h0dc7051_0_locked     66kB
  paramiko-3.5.1-pyhd8ed1ab_0                           844kB
  yte-1.5.7-pyha770c72_0                                 42kB

  /Users/victor/.nextstrain/runtimes/conda/.mamba/pkgs
───────────────────────────────────────────────────────────────


  
───────────────────────────────────────────────────────────────

  Total size:                                           128MB
Cleaning packages..
warning  libmamba This does not check for packages installed using
    symlinks back to the package cache.

Runtime updated!

CHANGES.md

tsibley · 2025-02-06T22:08:19Z

All this to say, this is not a breaking change as long as we continue to provide both osx-arm64 and osx-64 variants in conda-base.

An edge case here is that nextstrain check-setup conda will no longer test Rosetta 2 even for Conda runtime setups that are still using osx-64 on arm64 hardware. But I think that's ok.

tsibley · 2025-02-06T22:10:24Z

We're gonna have to remember for a long while into the future that not all Conda runtimes on macOS will be arm64, if they've been long-lived and simply updated continually. I think it'd be simpler to reason about if we explicitly switched setups over from osx-64 → osx-arm64. But that's a bit more work to get right too.

victorlin · 2025-02-07T00:21:12Z

Good points. How about a deprecation message like this? 76543e8

Checking for newer versions of Nextstrain CLI…

nextstrain-cli is up to date!

Updating conda runtime…

Your Conda runtime was created before Nextstrain CLI version X.X.X
and is using is using Rosetta 2 for emulation. In a future major
version release, Nextstrain CLI will stop supporting Conda runtimes
created before version X.X.X. To re-create your runtime, force a
fresh installation with `nextstrain setup conda --force`.

…

The same code that detects "variant" of a Conda runtime setup (by inspecting installed packages, maybe there is a better way) can also be used to keep checking Rosetta 2 for osx-64 setups on arm64 hardware.

Co-authored-by: Thomas Sibley <[email protected]>

tsibley · 2025-02-07T20:07:16Z

How about a deprecation message like this?

For a deprecation message, that's good. But I'm wondering why we would "stop supporting Conda runtimes created before version X.X.X" (would it be merely to remove the subdir detection and conditional Rosetta 2 check?) and why we'd ask the user to recreate the runtime instead of doing it automatically on update?

My thinking was we could add code to update() to automatically re-setup the runtime fresh if updating nextstrain-base to a version >= "20250203T212457Z" when the currently-installed version's subdir is osx-64. (The latter can be read out of installed package metadata, and there's a helper function for that already.)

victorlin · 2025-02-08T06:54:37Z

That makes sense, I'll look into updating this PR with that behavior.

victorlin self-assigned this Jul 12, 2024

victorlin mentioned this pull request Jul 12, 2024

Add build for osx-arm64 nextstrain/conda-base#80

Merged

2 tasks

tsibley mentioned this pull request Jul 30, 2024

CI on macOS failing due to IQTREE #382

Closed

victorlin mentioned this pull request Jul 31, 2024

Run pathogen CI on multiple subdirs nextstrain/conda-base#83

Open

victorlin force-pushed the victorlin/conda-osx-arm64 branch from 53a96c5 to b48c516 Compare February 5, 2025 21:01

victorlin marked this pull request as ready for review February 5, 2025 21:18

victorlin requested a review from tsibley February 5, 2025 21:18

Use osx-arm64 packages on Apple silicon

e274a7c

Previously the osx-64 packages were used because osx-arm64 was not supported. Rosetta 2 is no longer required for the conda runner on Apple silicon.

victorlin force-pushed the victorlin/conda-osx-arm64 branch from b48c516 to e274a7c Compare February 5, 2025 23:13

tsibley reviewed Feb 6, 2025

View reviewed changes

tsibley approved these changes Feb 6, 2025

View reviewed changes

tsibley reviewed Feb 6, 2025

View reviewed changes

CHANGES.md Outdated Show resolved Hide resolved

Add changelog entry for conda osx-arm64 support

3704648

Co-authored-by: Thomas Sibley <[email protected]>

victorlin force-pushed the victorlin/conda-osx-arm64 branch from 9330a6e to 3704648 Compare February 7, 2025 00:24

genehack approved these changes Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use osx-arm64 packages on Apple silicon #379

Use osx-arm64 packages on Apple silicon #379

victorlin commented Jul 12, 2024 •

edited

Loading

tsibley Feb 6, 2025

tsibley left a comment

tsibley commented Feb 6, 2025

victorlin commented Feb 6, 2025

victorlin commented Feb 6, 2025 •

edited

Loading

tsibley commented Feb 6, 2025

tsibley commented Feb 6, 2025

victorlin commented Feb 7, 2025 •

edited

Loading

tsibley commented Feb 7, 2025

victorlin commented Feb 8, 2025

Use osx-arm64 packages on Apple silicon #379

Are you sure you want to change the base?

Use osx-arm64 packages on Apple silicon #379

Conversation

victorlin commented Jul 12, 2024 • edited Loading

Description of proposed changes

Checklist

tsibley Feb 6, 2025

Choose a reason for hiding this comment

tsibley left a comment

Choose a reason for hiding this comment

tsibley commented Feb 6, 2025

victorlin commented Feb 6, 2025

victorlin commented Feb 6, 2025 • edited Loading

tsibley commented Feb 6, 2025

tsibley commented Feb 6, 2025

victorlin commented Feb 7, 2025 • edited Loading

tsibley commented Feb 7, 2025

victorlin commented Feb 8, 2025

victorlin commented Jul 12, 2024 •

edited

Loading

victorlin commented Feb 6, 2025 •

edited

Loading

victorlin commented Feb 7, 2025 •

edited

Loading