Fix Darwin ARM64 build (macOS/M1 support) #16621

simonfall · 2021-06-22T21:53:59Z

Why are these changes needed?

This PR fixes the wheel build for platform "darwin_arm64" to support Macs with Apple's M1 chipset.

Fix Bazel
- Toolchain detection bug: the "cc-compiler-darwin_arm64" toolchain cannot be detected when install CommandLineTools and Xcode.app is not installed. bazelbuild/bazel#13514
- Will be fixed in Bazel 4.2.0 (June 2021): Release 4.2 - June 2021 bazelbuild/bazel#13558
- For now use Bazel built from source (HEAD at 0ad9fac9b3d24e78f91c831c393520787022e2a1, branch 4.1.0-unix-toolchain-fix-for-darwin-arm64)
Upgrade @com_github_nelhage_rules_boost version
- Fallthrough patch is obsolete
Patch @com_github_cares_cares build in @com_github_grpc_grpc dependency
- See Add darwin_arm64 and darwin_arm64e to c-ares’ config settings grpc/grpc#24872
- Directly upgrading @com_github_grpc_grpc or @com_github_cares_cares breaks the build

Related issue number

Fixes #14697
Note that until Bazel 4.2.0 is released, Bazel must either be built from source or the toolchain must be manually configured.

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

simon-mo · 2021-06-24T00:15:44Z

@thomasdesr @scv119 can you help check this PR?

thomasdesr · 2021-07-03T01:37:25Z

@simonfall Thanks for putting this together.

I've made several attempts to get bazel built from source and been failing so far. Could you share the process you followed to build bazel from source?

scv119 · 2021-07-03T02:33:37Z

Sorry I don't have a M1 machine for verifying this change :'(

simonfall · 2021-07-04T14:50:04Z

Edit: Use Bazel 4.2.0rc1 through Bazelisk.

Original comment:

@thomasdesr Sure, here are the steps I followed:

As the time of writing, Bazel does not work with a full installation of Xcode. If you have Xcode installed, temporarily rename Xcode.app so that Bazel doesn't detect it and set the xcrun path to developer tools: sudo xcode-select -s /Library/Developer/CommandLineTools. There should be no error messages related to Xcode.app in bazel build commands now.
Install bazel 4.1.0 through homebrew: brew install bazel. bazel --version should return bazel 4.1.0-homebrew.
Follow the steps to build Bazel from source.
The first build should fail because no matching toolchain is found.
Edit the local_config_cc configuration to include a darwin_arm64 toolchain name. You could do this manually or just copy this gist: cp toolchain-fix-darwin_arm64.bazel $(bazel info output_base)/external/local_config_cc/BUILD.
Now you should be able to build Bazel from source.

Let me know if this works for you :)

PS: AFAIK, the Bazel team actively worked on Apple Silicon support in the last weeks. You could try to simply reinstall an up-to-date release candidate and see if it works to build Ray.

thomasdesr · 2021-07-07T01:32:14Z

Hey @simonfall !

Thanks for the instructions. Things aren't quite working for me yet but we've definitely made some serious progress!

Everything is now happily compling but I'm failing when trying to link the raylet. Output:

$ JAVA_HOME=/opt/jdk/zulu8.50.0.1017-ca-jdk8.0.275-macos_aarch64 /opt/bazel/bazel-master build //:raylet
Starting local Bazel server and connecting to it...
DEBUG: $HOME/repos/ray/bazel/ray_deps_setup.bzl:63:14: No implicit mirrors used because urls were explicitly provided
DEBUG: /private/var/tmp/_bazel_thomas/ef57690b644d99bb515228a51909ae9a/external/rules_proto_grpc/internal/common.bzl:262:14: Bazel is not a release version, rules_proto_grpc requires at least 1.0.0
INFO: Analyzed target //:raylet (76 packages loaded, 12631 targets configured).
INFO: Found 1 target...
INFO: From Linking external/com_github_facebook_zstd/libthreading.a:
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_github_facebook_zstd/libthreading.a the table of contents is empty (no object file members in the library define global symbols)
ERROR: $HOME/repos/ray/BUILD.bazel:418:10: Linking raylet failed: (Exit 1): cc_wrapper.sh failed: error executing command
  (cd /private/var/tmp/_bazel_thomas/ef57690b644d99bb515228a51909ae9a/sandbox/darwin-sandbox/2046/execroot/com_github_ray_project_ray && \
  exec env - \
    LANG=en_US.UTF-8 \
    PATH=$PATH \
    PWD=/proc/self/cwd \
  external/local_config_cc/cc_wrapper.sh @bazel-out/darwin_arm64-opt/bin/raylet-2.params)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:raylet failed to build
INFO: Elapsed time: 376.627s, Critical Path: 115.46s
INFO: 3590 processes: 1545 internal, 2045 darwin-sandbox.
FAILED: Build did NOT complete successfully

I think the critical piece of that is this:

ld: library not found for -lrt

Did you hit this error?

simonfall · 2021-07-09T14:20:40Z

@thomasdesr librt doesn't exist on macOS. This error basically means that some dependency references Linux code where it should select macOS code (or your Bazel version selects the wrong toolchain). The fixes in this PR should define working dependencies. Have you tried bazel clean --expunge and build with bazel --repository_cache= build //:raylet?

PS: Also make sure you've built Bazel from the right branch (branch 4.1.0-unix-toolchain-fix-for-darwin-arm64, HEAD at 0ad9fac9b3d24e78f91c831c393520787022e2a1)

thomasdesr · 2021-07-14T05:27:21Z

Hey! I got a hold of a new M1 to try again from scratch on a clean everything. Unfortunately I reproduced the same behavior (missing -lrt) :(

amholler · 2021-07-15T17:02:37Z

I am trying to get ray to build/work on my macOS/M1, and this patch is very helpful! Thank you, @simonfall.

With this patch and this version of bazel, https://github.com/erwincoumans/bazel/releases/tag/bazel-3.7.1-mac-arm64, build:
JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home /usr/local/bin/bazel build //:raylet --sandbox_debug
fails with "Undefined symbols for architecture arm64". All undefined symbols are in libcryto.a (from boringssl), e.g.:
"_ChaCha20_ctr32", referenced from:
_CRYPTO_chacha_20 in libcrypto.a(chacha.pic.o)
"_aes_hw_cbc_encrypt", referenced from:
_AES_cbc_encrypt in libcrypto.a(bcm.pic.o)
_boringssl_fips_self_test in libcrypto.a(bcm.pic.o)
_aes_init_key in libcrypto.a(bcm.pic.o), e.g.:
...

Do you happen to have any insights on this? Is is because I am not using the manual build of bazel 4.2 build mentioned here or ???

amholler · 2021-07-16T02:44:32Z

One other followup on my last comment: The undefined symbols for architecture arm64 are coming
from boringssl, and the version of boringssl included in ray seems to be from October 2019 AFAICT,
this is being included to support grpc, which is from the same time-frame. It seems desirable to
update both packages to 2021 versions, which are being actively worked on for arm64.

simonfall · 2021-07-16T12:07:01Z

@amholler AFAICT, missing symbols come from linking Linux libs where macOS system libs would have provided the right symbols. Also I think that upgrading Ray's dependencies to the latest versions will require more effort than simply bumping up version numbers.

@thomasdesr @amholler I'll do a clean build in the next days and give you an update. It is crucial that you use a Bazel version which includes the mentioned fix (branch 4.1.0-unix-toolchain-fix-for-darwin-arm64, HEAD at 0ad9fac9b3d24e78f91c831c393520787022e2a1). Other versions might succeed in building the targets, but the targets will not run correctly.

amholler · 2021-07-16T16:00:14Z

Thanks @simonfall BTW, would it be possible for you to provide a wheel for your M1 build of Ray?
Just wondering if it could be an interesting exercise for me and/or @thomasdesr to run from your build.

simonfall · 2021-07-19T13:37:25Z

Thanks @simonfall BTW, would it be possible for you to provide a wheel for your M1 build of Ray?
Just wondering if it could be an interesting exercise for me and/or @thomasdesr to run from your build.

@amholler Find a wheel here. Let me know if you need anything else.

Edit: Updated release.

amholler · 2021-07-20T15:26:24Z

Thanks @simonfall BTW, would it be possible for you to provide a wheel for your M1 build of Ray?
Just wondering if it could be an interesting exercise for me and/or @thomasdesr to run from your build.

@amholler Find a wheel here. Let me know if you need anything else.

Hi, @simonfall, Thanks for posting your wheels!

They install fine on my mac m1, but when I try to run ray in my conda virtual env, I get this failure:
(rayenv) MacBook-Air:ludwig anne$ ray up test-aws.yaml
Traceback (most recent call last):
  File "/Users/anne/miniforge3/envs/rayenv/bin/ray", line 5, in <module>
    from ray.scripts.scripts import main
  File "/Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/__init__.py", line 68, in <module>
    _configure_system()
  File "/Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/__init__.py", line 65, in _configure_system
    CDLL(so_path, ctypes.RTLD_GLOBAL)
  File "/Users/anne/miniforge3/envs/rayenv/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/_raylet.so, 10): Library not loaded: librt.so
  Referenced from: /Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/_raylet.so
  Reason: image not found

This file matches the ray-1.4.0.data/purelib/ray/_raylet.so from the wheel; checking the dynamic load:
(rayenv) MacBook-Air:rayPackages anne$ otool -L /Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/_raylet.so
/Users/anne/miniforge3/envs/rayenv/lib/python3.8/site-packages/ray/_raylet.so:
        bazel-out/darwin_arm64-opt/bin/python/ray/_raylet.so (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
        librt.so (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1775.118.101)
        /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1775.118.101)

So I assume there's something uncool about my load path?  When I look at my rayenv libc++.1.dylib, e.g., it looks cool:
(rayenv) MacBook-Air:rayPackages anne$ file /Users/anne/miniforge3/envs/rayenv/lib/libc++.1.dylib
/Users/anne/miniforge3/envs/rayenv/lib/libc++.1.dylib: Mach-O 64-bit dynamically linked shared library arm64

So maybe if I can fix my load path, then I can show your wheels working on my system?
Sorry for my naive questions.  Thoughts?  Can you show your load path?

simonfall · 2021-07-24T22:38:50Z

@thomasdesr I checked the PR with a clean environment and it seems like renaming Xcode.app is not sufficient, but moving it to the trash works (xcode-select -p should return /Library/Developer/CommandLineTools).
You can now also skip the manual Bazel build, as version 4.2.0rc1 can be used through Bazelisk:

$ brew uninstall bazel
$ brew install bazelisk
$ export USE_BAZEL_VERSION=4.2.0rc1
$ bazel --version
bazel 4.2.0rc1

With Bazel 4.2.0rc1 and Xcode.app in the trash, the build works as expected and tests pass.

simonfall · 2021-07-24T22:57:10Z

Edit: environment wasn't clean but contained dummy librt.so, see below.

@amholler I rebuilt the required wheels in a clean environment, see this release. The following steps should work to use Ray with conda:

$ wget https://github.com/simonfall/ray/releases/download/2.0.0.dev0/ray-2.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl \
https://github.com/simonfall/ray/releases/download/2.0.0.dev0/py_spy-0.3.7-py2.py3-none-macosx_11_0_arm64.whl \
https://github.com/simonfall/ray/releases/download/2.0.0.dev0/dm_tree-0.1.6-cp39-cp39-macosx_11_0_arm64.whl
$ conda create -y -n test_ray python=3.9.5 numpy scipy pandas matplotlib grpcio
$ conda activate test_ray
$ pip install py_spy-0.3.7-py2.py3-none-macosx_11_0_arm64.whl
$ pip install dm_tree-0.1.6-cp39-cp39-macosx_11_0_arm64.whl
$ pip install "ray-2.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl[default]"

amholler · 2021-07-25T18:37:24Z

Thanks, @simonfall ! Install of the wheels still doesn't work for me, though the otool output looks cleaner.
Not sure what I am doing wrong.

(base) MacBook-Air:rayPackages.1 anne$ conda create -y -n test_ray python=3.9.5 numpy scipy pandas matplotlib grpcio
<snip>

(base) MacBook-Air:rayPackages.1 anne$ conda activate test_ray

(test_ray) MacBook-Air:rayPackages.1 anne$ pip install py_spy-0.3.7-py2.py3-none-macosx_11_0_arm64.whl
Processing ./py_spy-0.3.7-py2.py3-none-macosx_11_0_arm64.whl
Installing collected packages: py-spy
Successfully installed py-spy-0.3.7

(test_ray) MacBook-Air:rayPackages.1 anne$ pip install dm_tree-0.1.6-cp39-cp39-macosx_11_0_arm64.whl
Processing ./dm_tree-0.1.6-cp39-cp39-macosx_11_0_arm64.whl
Installing collected packages: dm-tree
Successfully installed dm-tree-0.1.6

(test_ray) MacBook-Air:rayPackages.1 anne$ pip install "ray-2.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl[default]"
<snip>
Successfully installed aiohttp-3.7.4.post0 aiohttp-cors-0.7.0 aioredis-1.3.1 async-timeout-3.0.1 attrs-21.2.0 blessings-1.7 cachetools-4.2.2 chardet-4.0.0 charset-normalizer-2.0.3 click-8.0.1 colorful-0.5.4 filelock-3.0.12 google-api-core-1.31.0 google-auth-1.33.1 googleapis-common-protos-1.53.0 gpustat-0.6.0 hiredis-2.0.0 idna-3.2 jsonschema-3.2.0 msgpack-1.0.2 multidict-5.1.0 nvidia-ml-py3-7.352.0 opencensus-0.7.13 opencensus-context-0.1.2 packaging-21.0 prometheus-client-0.11.0 protobuf-3.17.3 psutil-5.8.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyrsistent-0.18.0 pyyaml-5.4.1 ray-2.0.0.dev0 redis-3.5.3 requests-2.26.0 rsa-4.7.2 typing-extensions-3.10.0.0 urllib3-1.26.6 yarl-1.6.3

(test_ray) MacBook-Air:rayPackages.1 anne$ which ray
/Users/anne/miniforge3/envs/test_ray/bin/ray

(test_ray) MacBook-Air:rayPackages.1 anne$ ray --version
Traceback (most recent call last):
  File "/Users/anne/miniforge3/envs/test_ray/bin/ray", line 5, in <module>
    from ray.scripts.scripts import main
  File "/Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/__init__.py", line 69, in <module>
    _configure_system()
  File "/Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/__init__.py", line 66, in _configure_system
    CDLL(so_path, ctypes.RTLD_GLOBAL)
  File "/Users/anne/miniforge3/envs/test_ray/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/_raylet.so, 10): Library not loaded: librt.so
  Referenced from: /Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/_raylet.so
  Reason: image not found

(test_ray) MacBook-Air:rayPackages.1 anne$ otool -L /Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/_raylet.so
/Users/anne/miniforge3/envs/test_ray/lib/python3.9/site-packages/ray/_raylet.so:
        bazel-out/darwin_arm64-opt/bin/python/ray/_raylet.so (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
        librt.so (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1775.118.101)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1775.118.101)

simonfall · 2021-07-26T08:21:21Z

@amholler ~~I can only reproduce this behavior on a non-ARM Mac. Are you sure that you're not running Rosetta? What is the output of the arch command?~~

This is not true, see below

nuscire · 2021-07-26T09:21:35Z

@simonfall Thank you for all your efforts! Unfortunately, I got exactly the same error as @amholler described from installing the linked wheels, and not under Rosetta (arch returns arm64). otool did report the shared lib _raylet.so had a dependence of librt.so, which is not available on macOS. It's puzzling how librt.so got involved into _raylet.so, because trying to build from source using bazel (either your patched version or official 4.2.0rc1) would always fail at linking raylet (ld: library not found for -lrt).

john@macbook ray_fix % xcode-select -p
/Library/Developer/CommandLineTools
john@macbook ray_fix % git branch
* fix-darwin_arm64-build
  master
john@macbook ray_fix % export USE_BAZEL_VERSION=4.2.0rc1
john@macbook ray_fix % which bazel
/opt/homebrew/bin/bazel
john@macbook ray_fix % bazel --version
bazel 4.2.0rc1
john@macbook ray_fix % bazel clean --expunge
Starting local Bazel server and connecting to it...
INFO: Starting clean.
john@macbook ray_fix % bazel build //:raylet
Starting local Bazel server and connecting to it...
DEBUG: /Users/john/Desktop/ray_build/ray_fix/bazel/ray_deps_setup.bzl:63:14: No implicit mirrors used because urls were explicitly provided
INFO: Analyzed target //:raylet (76 packages loaded, 12631 targets configured).
INFO: Found 1 target...
INFO: From Linking external/com_github_facebook_zstd/libthreading.a:
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: archive library: bazel-out/darwin_arm64-opt/bin/external/com_github_facebook_zstd/libthreading.a the table of contents is empty (no object file members in the library define global symbols)
ERROR: /Users/john/Desktop/ray_build/ray_fix/BUILD.bazel:418:10: Linking raylet failed: (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_john/463dd050f293cd62f9d9729b645dba45/sandbox/darwin-sandbox/2046/execroot/com_github_ray_project_ray && \
  exec env - \
    LANG=en_US.UTF-8 \
    PATH='/Users/john/Library/Caches/bazelisk/downloads/bazelbuild/bazel-4.2.0rc1-darwin-arm64/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/share/dotnet:~/.dotnet/tools' \
    PWD=/proc/self/cwd \
  external/local_config_cc/cc_wrapper.sh @bazel-out/darwin_arm64-opt/bin/raylet-2.params)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:raylet failed to build
INFO: Elapsed time: 323.861s, Critical Path: 81.66s
INFO: 3590 processes: 1545 internal, 2045 darwin-sandbox.
FAILED: Build did NOT complete successfully

amholler · 2021-07-26T14:20:35Z

Hi, @simonfall, Thanks. Like for @nuscire the output of arch is arm64:

(test_ray) MacBook-Air:/ anne$ arch
arm64

Checking my conda list, all packages look arm64 except a few noarch

(test_ray) MacBook-Air:/ anne$ conda list --explicit
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-arm64
@EXPLICIT
https://conda.anaconda.org/conda-forge/osx-arm64/c-ares-1.17.1-h27ca646_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/ca-certificates-2021.5.30-h4653dfc_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/jbig-2.1-h3422bc3_2003.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/jpeg-9d-h27ca646_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libcxx-12.0.1-h168391b_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libdeflate-1.7-h27ca646_5.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libwebp-base-1.2.0-h27ca646_2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/llvm-openmp-12.0.1-hf3c4609_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/ncurses-6.2-h9aa5885_4.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/tzdata-2021a-he74cb21_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/xz-5.2.5-h642e427_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/zlib-1.2.11-h31e879b_1009.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/lerc-2.2.1-h9f76cd9_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libgfortran5-11.0.1.dev0-hf114ba7_22.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libpng-1.6.37-hf7e6567_2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/lz4-c-1.9.3-h9f76cd9_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/openssl-1.1.1k-h27ca646_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/readline-8.1-hedafd6a_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/tk-8.6.10-hf7e6567_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/freetype-2.10.4-h17b34a0_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libgfortran-5.0.0.dev0-11_0_1_hf114ba7_22.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/sqlite-3.36.0-h72a2b83_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/zstd-1.5.0-h861e0a7_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libopenblas-0.3.15-openmp_hf330de4_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libtiff-4.3.0-hc6122e1_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/python-3.9.5-h54d631c_0_cpython.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/lcms2-2.12-had6a04f_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libblas-3.9.0-9_openblas.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/olefile-0.46-pyh9f0ad1d_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/openjpeg-2.4.0-h062765e_1.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pyparsing-2.4.7-pyh9f0ad1d_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/python_abi-3.9-2_cp39.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pytz-2021.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wheel-0.36.2-pyhd3deb0d_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/certifi-2021.5.30-py39h2804cbe_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/cycler-0.10.0-py_2.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/kiwisolver-1.3.1-py39h4d2d688_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/libcblas-3.9.0-9_openblas.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/liblapack-3.9.0-9_openblas.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/pillow-8.3.1-py39ha74c66e_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/python-dateutil-2.8.2-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/tornado-6.1-py39h5161555_1.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/numpy-1.21.1-py39h1f3b974_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/setuptools-49.6.0-py39h2804cbe_3.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/grpcio-1.38.1-py39h9e1b6db_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/matplotlib-base-3.4.2-py39h5aa4fe7_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/pandas-1.3.1-py39h7f752ed_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/pip-21.2.1-pyhd8ed1ab_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/scipy-1.7.0-py39h5060c3b_0.tar.bz2
https://conda.anaconda.org/conda-forge/osx-arm64/matplotlib-3.4.2-py39hdf13c20_0.tar.bz2

amholler · 2021-07-26T18:07:21Z

One more thing (cc @simonfall @nuscire ), I can see that _raylet.so contains librt.so using otool -l.
And if I patch librt.so out by changing it to something harmless via:
install_name_tool -change librt.so /usr/lib/libSystem.B.dylib _raylet.so
then I can run ray without getting the librt.so library not loaded problem.
(test_ray) MacBook-Air:/ anne$ ray --version
ray, version 2.0.0.dev0

simonfall · 2021-07-26T20:03:24Z

@thomasdesr @nuscire @amholler Finally found the issue. You were right, and of course the _raylet.so links to librt.so. During debugging the build, I created an empty dummy librt.so and forgot about it. I am sorry for wasting your time! I will trace the target which links librt.so in the next weeks, but any input is highly appreciated!

nuscire · 2021-07-27T04:02:54Z

@simonfall @amholler Cheers! we can finally get a working Ray on ARM Macs. By making a dummy librt.so (and placing it under /usr/local/lib), building from source (fix-darwin_arm64-build branch from @simonfall) went without any quirk. Also, with a dummy librt.so in place, the linked wheels above will work out of the box. So far so good. Tracing Bazel build chains is beyond my current knowledge, sorry bro.

iamhatesz · 2021-07-28T21:54:23Z

@simonfall @nuscire hey, I've tried your solution with a dummy librt.so, but I got:

ld: file too small (length=0) file '/usr/local/lib/librt.so' for architecture arm64

nuscire · 2021-07-29T01:09:16Z

@iamhatesz Try to make the dummy librt.so with

echo "" | clang -dynamiclib -o librt.so -x c -

librt.so needs to be a dynamic lib per se.

ZetiMente · 2021-08-02T16:06:11Z

echo "" | clang -dynamiclib -o librt.so -x c -

This worked for me! Thanks so much! Looking forward to the release.

cb0207 · 2021-08-14T19:24:47Z

@simonfall follow the same step. It works for python3.9.
Is it possible to make it for python3.8 (https://github.com/simonfall/ray/tree/ray-1.4.0-cp38-darwin_arm64) ?
Only found this release about Apple M1 python3.8.

However, similar issue about librt.so. It shown missing librt.so.

(py38) ➜  ray ray --version
Traceback (most recent call last):
  File "/Users/testuser/opt/miniforge3/envs/py38/bin/ray", line 5, in <module>
    from ray.scripts.scripts import main
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/__init__.py", line 68, in <module>
    _configure_system()
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/__init__.py", line 65 in _configure_system
    CDLL(so_path, ctypes.RTLD_GLOBAL)
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/_raylet.so, 10): Library not loaded: librt.so
  Referenced from: /Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/_raylet.so
  Reason: image not found

Tried to create dummy one and below info came with.

(py38) ➜  ray ray --version
Traceback (most recent call last):
  File "/Users/testuser/opt/miniforge3/envs/py38/bin/ray", line 5, in <module>
    from ray.scripts.scripts import main
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/__init__.py", line 68, in <module>
    _configure_system()
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/__init__.py", line 65, in _configure_system
    CDLL(so_path, ctypes.RTLD_GLOBAL)
  File "/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/_raylet.so, 10): Symbol not found: _getservbyport_r
  Referenced from: /Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/_raylet.so
  Expected in: /usr/local/lib/librt.so
 in /Users/testuser/opt/miniforge3/envs/py38/lib/python3.8/site-packages/ray/_raylet.so

ipsec · 2021-08-22T03:45:22Z

Bazel 4.2.0 https://github.com/bazelbuild/bazel/releases/tag/4.2.0 released.

dhrumilp31 · 2021-08-27T16:32:34Z

I have the same issue as @cb0207, is there any way to fix import error on build 1.4.0 (Apple M1) for py3.8 ?

gjoliver · 2021-09-04T16:09:39Z

Hi guys, what's the status on this PR? I got issued an M1 laptop, and would really like this to get fixed soon :)

I tried to follow the steps here (using 4.2.1 bazel, so I didn't have to build that)
some things seem to have been updated, like I don't think we need the rules_boost-windows-linkopts.patch anymore.
and the grpc library may be updated too? the patch doesn't work anymore.

But I was able to build a lot of stuff if I comment out those 2 patches, until it failed at some assembly code from boost library.

ERROR: /private/var/tmp/_bazel_jungong/d6d41ed255e0f9f8888bca8c36b983b0/external/boost/BUILD.bazel:183:14: Compiling libs/context/src/asm/jump_arm64_aapcs_macho_gas.S failed: (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_jungong/d6d41ed255e0f9f8888bca8c36b983b0/sandbox/darwin-sandbox/743/execroot/com_github_ray_project_ray && \
  exec env - \
    LANG=en_US.UTF-8 \
    PATH=/opt/local/bin:/opt/local/sbin:/opt/homebrew/bin:/Users/jungong/bin:/opt/local/bin:/opt/local/sbin:/Users/jungong/opt/anaconda3/envs/raym1/bin:/Users/jungong/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin \
    PWD=/proc/self/cwd \
  external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/darwin_arm64-opt/bin/external/boost/_objs/context/jump_arm64_aapcs_macho_gas.pic.d -fPIC -iquote external/boost -iquote bazel-out/darwin_arm64-opt/bin/external/boost -isystem external/boost -isystem bazel-out/darwin_arm64-opt/bin/external/boost -Wno-unused -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S -o bazel-out/darwin_arm64-opt/bin/external/boost/_objs/context/jump_arm64_aapcs_macho_gas.pic.o)
Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:58:21: error: unexpected token in argument list
    ; prepare stack for GP + FPU
                    ^
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:59:17: error: unknown token in expression
    sub sp, sp, #0xb0
                ^
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:61:7: error: invalid instruction mnemonic 'save'
    ; save d8 - d15
      ^~~~
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:62:17: error: brackets expression not supported on this target
    stp d8, d9, [sp, #0x00]
                ^
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:63:19: error: brackets expression not supported on this target
    stp d10, d11, [sp, #0x10]
                  ^
external/boost/libs/context/src/asm/jump_arm64_aapcs_macho_gas.S:64:19: error: brackets expression not supported on this target
    stp d12, d13, [sp, #0x20]

......

Any idea what may be the problem here? I thought we are pulling a different boost package via https://github.com/nelhage/rules_boost/archive/652b21e35e4eeed5579e696da0facbe8dba52b1f.tar.gz ??

Thanks

gjoliver · 2021-09-07T09:45:58Z

happy to report that if I pull in the same boost tarball and patch, just drop the grpc patch, I can actually successfully build ray on my M1.

simonfall · 2021-09-10T06:19:02Z

happy to report that if I pull in the same boost tarball and patch, just drop the grpc patch, I can actually successfully build ray on my M1.

@gjoliver Thank you for figuring this out! Unfortunately, I wasn't able to reproduce the build from the steps mentioned in your comment. Would you mind clarifying the required steps or posting a patch?

gjoliver · 2021-09-10T06:44:27Z

happy to report that if I pull in the same boost tarball and patch, just drop the grpc patch, I can actually successfully build ray on my M1.

@gjoliver Thank you for figuring this out! Unfortunately, I wasn't able to reproduce the build from the steps mentioned in your comment. Would you mind clarifying the required steps or posting a patch?

Sure.

I hacked that bazel-install.sh script to install bazel 4.2.1
This patch is the only code changes I had to do. https://github.com/gjoliver/ray/commit/2e1a784a3bb2f1e4118a1bf8c241e64cb7a2f196.patch
We still need the empty librt.so hack.

And together, I was able to build everything.

simonfall · 2021-09-11T11:33:12Z

From my perspective, there are three options to proceed and provide builds for ARM Macs:

Do the librt.so hack and write a small util which unlinks it from raylet.so after the build is complete (temporary workaround)
Investigate which dependency doesn't correctly detect the darwin_arm64 target and update/patch it (correct way, I would need some help on this)
Do nothing and wait until the dependencies are updated and the problem vanishes

@simon-mo @thomasdesr What do you prefer?

gjoliver · 2021-09-14T06:56:53Z

Hey Guys,

So I figured out the offending dependency. It's civetweb, pulled in via prometheus_cpp library.
This patch allows us to build Ray without any manual work, including that librt.so hack:
https://github.com/gjoliver/ray/commit/7ff4a8213c2c25141cd19cfbb7782632e9305f0e.patch

should we start submitting things, so developers on M1s are unblocked?

wuisawesome · 2021-09-21T01:39:07Z

@gjoliver do you know if there's a way to test this in our CI?

Either way, it would be great if you could submit that patch as PR and tag me in it.

wuisawesome · 2021-09-21T21:54:09Z

@gjoliver an I chatted offline and came up with a plan. We'll add support in 3 steps:

Upgrade bazel to 4.2.1
Upgrade boost
Update the rest of the patch

Hopefully if anything breaks this. will make it easy to narrow down.

gjoliver · 2021-10-13T03:42:56Z

I have merged the other ARM build PR.
thanks for all the ideas and guidances offered here.
https://docs.google.com/document/d/1XgpuUX0vtk82NIVQyEk8d3tnbpSFHO8ZGmF-i2J8_9A/edit?usp=sharing

erwincoumans · 2021-11-02T01:21:11Z

Thanks for all the great work on this. Can someone please share a precompiled ray wheel for M1 arm64 python 3.9?

pcmoritz · 2021-11-02T03:42:56Z

@erwincoumans We will provide M1 wheels for the Ray 1.8.0 release which will be released this week :)

xwjiang2010 · 2021-11-03T04:16:26Z

Hi all, Ray 1.8.0 was officially released today. Please find M1 wheels here.

MarvinSt · 2021-11-16T21:48:38Z

Is there also a possibility to run Ray in Docker on the M1? I tried cross compiling for linux x64 (using buildx build --platform linux/x64), but that does not work correctly on the M1.

Installing and running Ray directly without Docker is working fine.

Thanks!

gjoliver · 2021-11-16T21:58:08Z

Why don't you just use a linux wheel?

MarvinSt · 2021-11-16T22:02:22Z

the Linux wheel is not supported on arm and building for linux/x64 and emulating does not work properly

gjoliver · 2021-11-16T22:17:12Z

oh I see. yeah, we don't have official linux arm build yet, but I heard building ARM wheels for linux is generally working out of the box.
#13780

simonfall mentioned this pull request Jun 22, 2021

Mac M1 Support #14697

Closed

simonfall added 2 commits June 23, 2021 00:20

Upgrade @com_github_nelhage_rules_boost version

f64ab85

Fix @com_github_cares_cares build in @com_github_grpc_grpc dependency

8fbed13

architkulkarni assigned wuisawesome and simon-mo Jun 24, 2021

scv119 mentioned this pull request Jul 9, 2021

[prototype] boost fiber instead of asio for async #16699

Closed

6 tasks

simon-mo mentioned this pull request Jul 20, 2021

does not support mac with M1 cpu #17205

Closed

This was referenced Sep 24, 2021

[Bug] [Ray Client] [M1] RuntimeError: Starting up Server Failed! with empty logs on M1 chip #18876

Closed

Upgrade bazel version to 4.2.1 #18996

Merged

gjoliver mentioned this pull request Oct 1, 2021

Update build rules and patches for darwin_arm64 platform. #19037

Merged

4 tasks

gjoliver closed this Oct 13, 2021

philwinder mentioned this pull request Nov 4, 2021

Update KServe -> Ray dependency to most recent version kserve/kserve#1897

Closed

Fix Darwin ARM64 build (macOS/M1 support) #16621

Fix Darwin ARM64 build (macOS/M1 support) #16621

Conversation

simonfall commented Jun 22, 2021 • edited Loading

Why are these changes needed?

Related issue number

Checks

simon-mo commented Jun 24, 2021

thomasdesr commented Jul 3, 2021 • edited Loading

scv119 commented Jul 3, 2021

simonfall commented Jul 4, 2021 • edited Loading

thomasdesr commented Jul 7, 2021 • edited Loading

simonfall commented Jul 9, 2021 • edited Loading

thomasdesr commented Jul 14, 2021

amholler commented Jul 15, 2021

amholler commented Jul 16, 2021 • edited Loading

simonfall commented Jul 16, 2021 • edited Loading

amholler commented Jul 16, 2021

simonfall commented Jul 19, 2021 • edited Loading

amholler commented Jul 20, 2021 • edited Loading

simonfall commented Jul 24, 2021 • edited Loading

simonfall commented Jul 24, 2021 • edited Loading

amholler commented Jul 25, 2021 • edited Loading

simonfall commented Jul 26, 2021 • edited Loading

nuscire commented Jul 26, 2021

amholler commented Jul 26, 2021

amholler commented Jul 26, 2021 • edited Loading

simonfall commented Jul 26, 2021 • edited Loading

nuscire commented Jul 27, 2021

iamhatesz commented Jul 28, 2021

nuscire commented Jul 29, 2021

ZetiMente commented Aug 2, 2021

cb0207 commented Aug 14, 2021

ipsec commented Aug 22, 2021

dhrumilp31 commented Aug 27, 2021

gjoliver commented Sep 4, 2021 • edited Loading

gjoliver commented Sep 7, 2021

simonfall commented Sep 10, 2021

gjoliver commented Sep 10, 2021

simonfall commented Sep 11, 2021 • edited Loading

gjoliver commented Sep 14, 2021

wuisawesome commented Sep 21, 2021

wuisawesome commented Sep 21, 2021

gjoliver commented Oct 13, 2021

erwincoumans commented Nov 2, 2021

pcmoritz commented Nov 2, 2021

xwjiang2010 commented Nov 3, 2021

MarvinSt commented Nov 16, 2021

gjoliver commented Nov 16, 2021

MarvinSt commented Nov 16, 2021

gjoliver commented Nov 16, 2021

simonfall commented Jun 22, 2021 •

edited

Loading

thomasdesr commented Jul 3, 2021 •

edited

Loading

simonfall commented Jul 4, 2021 •

edited

Loading

thomasdesr commented Jul 7, 2021 •

edited

Loading

simonfall commented Jul 9, 2021 •

edited

Loading

amholler commented Jul 16, 2021 •

edited

Loading

simonfall commented Jul 16, 2021 •

edited

Loading

simonfall commented Jul 19, 2021 •

edited

Loading

amholler commented Jul 20, 2021 •

edited

Loading

simonfall commented Jul 24, 2021 •

edited

Loading

simonfall commented Jul 24, 2021 •

edited

Loading

amholler commented Jul 25, 2021 •

edited

Loading

simonfall commented Jul 26, 2021 •

edited

Loading

amholler commented Jul 26, 2021 •

edited

Loading

simonfall commented Jul 26, 2021 •

edited

Loading

gjoliver commented Sep 4, 2021 •

edited

Loading

simonfall commented Sep 11, 2021 •

edited

Loading