Update cross-compile assistance for Apple silicon #312

Malax · 2022-02-08T21:47:17Z

The current instructions to set-up cross compilation only work on Intel based Macs since homebrew-musl-cross is not available for Apple silicon Macs yet:

Apple silicon support? FiloSottile/homebrew-musl-cross#23.

This PR updates the cross-compile assistance to use homebrew-macos-cross-toolchains instead which also supports Apple silicon. However, support for homebrew-musl-cross hasn't been removed, existing installations will continue to work without the need to install new toolchains after this change is released.

edmorley

Thank you for adding M1 support! I wish I had an M1 to try this out on... soon hopefully :-)

Trying this locally on an Intel Macbook using Rust 1.59.0-beta.6 (and after purging target/), I got:

example-02-ruby-sample $ rm -rf ../../target/ ../../Cargo.lock && cargo libcnb package
INFO - Reading buildpack metadata...
INFO - Found buildpack libcnb-examples/ruby with version 0.1.0.
INFO - Determining automatic cross-compile settings...
INFO - Building buildpack binary (x86_64-unknown-linux-musl)...
   Compiling libc v0.2.117
   Compiling memchr v2.4.1
   Compiling proc-macro2 v1.0.36
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.86
   Compiling cfg-if v1.0.0
   Compiling cc v1.0.72
   Compiling typenum v1.15.0
   Compiling version_check v0.9.4
   Compiling serde_derive v1.0.136
   Compiling untrusted v0.7.1
   Compiling once_cell v1.9.0
   Compiling spin v0.5.2
   Compiling bit-vec v0.6.3
   Compiling serde v1.0.136
   Compiling autocfg v1.1.0
   Compiling tinyvec_macros v0.1.0
   Compiling regex-syntax v0.6.25
   Compiling crc32fast v1.3.2
   Compiling matches v0.1.9
   Compiling log v0.4.14
   Compiling unicode-bidi v0.3.7
   Compiling adler v1.0.2
   Compiling percent-encoding v2.1.0
   Compiling cpufeatures v0.2.1
   Compiling chunked_transfer v1.4.0
   Compiling remove_dir_all v0.5.3
   Compiling fastrand v1.7.0
   Compiling base64 v0.13.0
   Compiling tinyvec v1.5.1
   Compiling bit-set v0.5.2
   Compiling generic-array v0.14.5
   Compiling form_urlencoded v1.0.1
   Compiling miniz_oxide v0.4.4
   Compiling aho-corasick v0.7.18
   Compiling unicode-normalization v0.1.19
   Compiling ring v0.16.20
   Compiling quote v1.0.15
   Compiling filetime v0.2.15
   Compiling xattr v0.2.2
   Compiling tempfile v3.3.0
   Compiling tar v0.4.38
   Compiling flate2 v1.0.22
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/emorley/src/libcnb.rs/target/debug/build/ring-0959747d21901273/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-musl")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-unknown-linux-musl = None
  CC_x86_64_unknown_linux_musl = Some("/usr/local/bin/x86_64-unknown-linux-musl-gcc")
  CFLAGS_x86_64-unknown-linux-musl = None
  CFLAGS_x86_64_unknown_linux_musl = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "/usr/local/bin/x86_64-unknown-linux-musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/emorley/src/libcnb.rs/target/x86_64-unknown-linux-musl/debug/build/ring-c387f252de53e7e2/out/aesni-x86_64-elf.o" "/Users/emorley/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  dyld[80503]: Library not loaded: /usr/local/opt/zstd/lib/libzstd.1.dylib
    Referenced from: /usr/local/Cellar/x86_64-unknown-linux-musl/10.3.0/toolchain/libexec/gcc/x86_64-unknown-linux-musl/10.3.0/cc1
    Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/deps/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file)
  x86_64-unknown-linux-musl-gcc: internal compiler error: Abort trap: 6 signal terminated program cc1
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <https://gcc.gnu.org/bugs/> for instructions.
  thread 'main' panicked at 'execution failed', /Users/emorley/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
^C  Building [==================>        ] 69/95: syn
example-02-ruby-sample $ rm -rf ../../target/ ../../Cargo.lock && cargo libcnb package
INFO - Reading buildpack metadata...
INFO - Found buildpack libcnb-examples/ruby with version 0.1.0.
INFO - Determining automatic cross-compile settings...
INFO - Building buildpack binary (x86_64-unknown-linux-musl)...
   Compiling libc v0.2.117
   Compiling memchr v2.4.1
   Compiling proc-macro2 v1.0.36
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.86
   Compiling cfg-if v1.0.0
   Compiling cc v1.0.72
   Compiling version_check v0.9.4
   Compiling typenum v1.15.0
   Compiling once_cell v1.9.0
   Compiling untrusted v0.7.1
   Compiling serde_derive v1.0.136
   Compiling spin v0.5.2
   Compiling bit-vec v0.6.3
   Compiling regex-syntax v0.6.25
   Compiling autocfg v1.1.0
   Compiling tinyvec_macros v0.1.0
   Compiling serde v1.0.136
   Compiling crc32fast v1.3.2
   Compiling log v0.4.14
   Compiling matches v0.1.9
   Compiling adler v1.0.2
   Compiling unicode-bidi v0.3.7
   Compiling percent-encoding v2.1.0
   Compiling fastrand v1.7.0
   Compiling remove_dir_all v0.5.3
   Compiling chunked_transfer v1.4.0
   Compiling cpufeatures v0.2.1
   Compiling base64 v0.13.0
   Compiling tinyvec v1.5.1
   Compiling bit-set v0.5.2
   Compiling generic-array v0.14.5
   Compiling form_urlencoded v1.0.1
   Compiling miniz_oxide v0.4.4
   Compiling aho-corasick v0.7.18
   Compiling quote v1.0.15
   Compiling ring v0.16.20
   Compiling unicode-normalization v0.1.19
   Compiling filetime v0.2.15
   Compiling xattr v0.2.2
   Compiling tempfile v3.3.0
   Compiling tar v0.4.38
   Compiling flate2 v1.0.22
error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/Users/emorley/src/libcnb.rs/target/debug/build/ring-0959747d21901273/build-script-build` (exit status: 101)
  --- stdout
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-musl")
  HOST = Some("x86_64-apple-darwin")
  CC_x86_64-unknown-linux-musl = None
  CC_x86_64_unknown_linux_musl = Some("/usr/local/bin/x86_64-unknown-linux-musl-gcc")
  CFLAGS_x86_64-unknown-linux-musl = None
  CFLAGS_x86_64_unknown_linux_musl = None
  TARGET_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")

  --- stderr
  running "/usr/local/bin/x86_64-unknown-linux-musl-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "include" "-Wall" "-Wextra" "-pedantic" "-pedantic-errors" "-Wall" "-Wextra" "-Wcast-align" "-Wcast-qual" "-Wconversion" "-Wenum-compare" "-Wfloat-equal" "-Wformat=2" "-Winline" "-Winvalid-pch" "-Wmissing-field-initializers" "-Wmissing-include-dirs" "-Wredundant-decls" "-Wshadow" "-Wsign-compare" "-Wsign-conversion" "-Wundef" "-Wuninitialized" "-Wwrite-strings" "-fno-strict-aliasing" "-fvisibility=hidden" "-fstack-protector" "-g3" "-U_FORTIFY_SOURCE" "-DNDEBUG" "-c" "-o/Users/emorley/src/libcnb.rs/target/x86_64-unknown-linux-musl/debug/build/ring-c387f252de53e7e2/out/aesni-x86_64-elf.o" "/Users/emorley/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesni-x86_64-elf.S"
  dyld[80671]: Library not loaded: /usr/local/opt/zstd/lib/libzstd.1.dylib
    Referenced from: /usr/local/Cellar/x86_64-unknown-linux-musl/10.3.0/toolchain/libexec/gcc/x86_64-unknown-linux-musl/10.3.0/cc1
    Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/deps/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file)
  x86_64-unknown-linux-musl-gcc: internal compiler error: Abort trap: 6 signal terminated program cc1
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <https://gcc.gnu.org/bugs/> for instructions.
  thread 'main' panicked at 'execution failed', /Users/emorley/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:656:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
ERROR - Packaging buildpack failed due to a build related error!
ERROR - Unexpected Cargo exit status: 101
ERROR - Examine Cargo output for details and potential compilation errors.

libcnb-cargo/src/cross_compile.rs

Co-authored-by: Ed Morley <[email protected]>

Malax · 2022-02-10T13:14:59Z

@edmorley I cannot reproduce your compile error. I tried building a buildpack that depends on ring on an Intel mac, but it just went through with both toolchains. I used the latest Rust: rustc 1.58.1 (db9d1b20b 2022-01-20).

Looking at the compile error, it looks like it cannot find zstd on your machine:

 dyld[80671]: Library not loaded: /usr/local/opt/zstd/lib/libzstd.1.dylib
    Referenced from: /usr/local/Cellar/x86_64-unknown-linux-musl/10.3.0/toolchain/libexec/gcc/x86_64-unknown-linux-musl/10.3.0/cc1
    Reason: tried: '/usr/local/opt/zstd/lib/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/deps/libzstd.1.dylib' (no such file), '/Users/emorley/src/libcnb.rs/target/debug/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/.rustup/toolchains/beta-x86_64-apple-darwin/lib/libzstd.1.dylib' (no such file), '/Users/emorley/lib/libzstd.1.dylib' (no such file), '/usr/local/lib/libzstd.1.dylib' (no such file), '/usr/lib/libzstd.1.dylib' (no such file)

I checked, I have it installed:

14:06:03 [malax/apple-silicon-cross-compile] libcnb.rs $ brew search zstd
==> Formulae
zstd ✔

It was installed as a dependency to gcc. Any ideas why it would've worked with https://github.com/FiloSottile/homebrew-musl-cross previously?

edmorley

I don't have zstd installed via brew, but then I don't have gcc via brew either (which gcc reports /usr/bin/gcc), however the build works fine prior to this PR (I checked again just now that the Ruby example compiles fine).

With this PR, I can't compile (I also tried again using Rust stable, rather than beta).

I tried setting the additional env vars mentioned in the https://github.com/messense/homebrew-macos-cross-toolchains README:

export CXX_x86_64_unknown_linux_musl=x86_64-unknown-linux-musl-g++
export AR_x86_64_unknown_linux_musl=x86_64-unknown-linux-musl-ar

...however that didn't help.

When I installed zstd manually the build succeeded.

It seems perhaps the cc1 binary that the x86_64-unknown-linux-musl package bundles might be dynamically linked against zstd, but not provide/require it?

If so, that's a bug that we should (a) report upstream, (b) document a workaround for in our instructions.

The fix upstream for x86_64-unknown-linux-musl is presumably either to:

Have the brew package declare a dependency on the zstd package
Statically compile cc1 against zstd
Bundle libzstd in some other way in the x86_64-unknown-linux-musl package

edmorley · 2022-02-12T16:08:21Z

If so, that's a bug that we should (a) report upstream, (b) document a workaround for in our instructions.

Filed:
messense/homebrew-macos-cross-toolchains#8

edmorley · 2022-02-13T11:42:28Z

Filed:
messense/homebrew-macos-cross-toolchains#8

That issue was fixed in:
messense/homebrew-macos-cross-toolchains@43c1873
messense/homebrew-macos-cross-toolchains@13b3250

I also found another missing dependency (messense/homebrew-macos-cross-toolchains#9) which was fixed in:
messense/homebrew-macos-cross-toolchains@ca2d720

With those changes (and purging the currently installed package, since the revision in the formula wasn't bumped as part of those fixes, so brew doesn't know to refresh it for existing installs otherwise), the setup instructions text added in this PR now works out of the box :-)

Malax added the libcnb-cargo label Feb 8, 2022

Malax force-pushed the malax/apple-silicon-cross-compile branch from 07fa2ac to 3733afa Compare February 8, 2022 21:51

Malax added 2 commits February 8, 2022 23:00

Update cross-compile assistance for Apple silicon

175dbe6

Update CHANGELOG

b10f809

Malax force-pushed the malax/apple-silicon-cross-compile branch from 3733afa to b10f809 Compare February 8, 2022 22:00

Malax marked this pull request as ready for review February 8, 2022 22:02

Malax requested review from dzuelke, hone, joshwlewis and edmorley February 8, 2022 22:02

edmorley requested changes Feb 9, 2022

View reviewed changes

libcnb-cargo/src/cross_compile.rs Outdated Show resolved Hide resolved

libcnb-cargo/src/cross_compile.rs Show resolved Hide resolved

Malax and others added 2 commits February 10, 2022 13:59

Fix cross-compile binary name in user output

2bba1e7

Co-authored-by: Ed Morley <[email protected]>

Clarify cross-compile help text for macOS

da818f3

Malax requested a review from edmorley February 11, 2022 14:41

edmorley requested changes Feb 11, 2022

View reviewed changes

edmorley approved these changes Feb 13, 2022

View reviewed changes

Malax merged commit 7ce7161 into main Feb 13, 2022

Malax deleted the malax/apple-silicon-cross-compile branch February 13, 2022 14:41

edmorley assigned Malax Feb 21, 2022

runesoerensen mentioned this pull request Feb 2, 2024

Refactor cross compile assistance #769

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update cross-compile assistance for Apple silicon #312

Update cross-compile assistance for Apple silicon #312

Malax commented Feb 8, 2022 •

edited

Loading

edmorley left a comment

Malax commented Feb 10, 2022

edmorley left a comment •

edited

Loading

edmorley commented Feb 12, 2022

edmorley commented Feb 13, 2022

Update cross-compile assistance for Apple silicon #312

Update cross-compile assistance for Apple silicon #312

Conversation

Malax commented Feb 8, 2022 • edited Loading

edmorley left a comment

Choose a reason for hiding this comment

Malax commented Feb 10, 2022

edmorley left a comment • edited Loading

Choose a reason for hiding this comment

edmorley commented Feb 12, 2022

edmorley commented Feb 13, 2022

Malax commented Feb 8, 2022 •

edited

Loading

edmorley left a comment •

edited

Loading