fd segfaults on macos Catalina

[raw-bin]

Hi.

Here are some details:

$ rustc --version
rustc 1.38.0 (625451e37 2019-09-23)

$ uname -a
Darwin nucleus 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

$ port version
Version: 2.6.99

To demonstrate the problem, I reinstalled fd as below:

$ cargo install fd-find --force
    Updating crates.io index
  Installing fd-find v7.4.0
   Compiling libc v0.2.62
   Compiling memchr v2.2.1
   Compiling bitflags v1.2.0
   Compiling fs_extra v1.1.0
   Compiling lazy_static v1.4.0
   Compiling cc v1.0.45
   Compiling unicode-width v0.1.6
   Compiling log v0.4.8
   Compiling regex-syntax v0.6.12
   Compiling cfg-if v0.1.10
   Compiling strsim v0.8.0
   Compiling ansi_term v0.11.0
   Compiling nix v0.14.1
   Compiling vec_map v0.8.1
   Compiling version_check v0.9.1
   Compiling same-file v1.0.5
   Compiling fnv v1.0.6
   Compiling void v1.0.2
   Compiling quick-error v1.2.2
   Compiling ansi_term v0.12.1
   Compiling thread_local v0.3.6
   Compiling crossbeam-utils v0.6.6
   Compiling walkdir v2.2.9
   Compiling humantime v1.3.0
   Compiling lscolors v0.6.0
   Compiling crossbeam-channel v0.3.9
   Compiling term_size v0.3.1
   Compiling atty v0.2.13
   Compiling num_cpus v1.10.1
   Compiling textwrap v0.11.0
   Compiling aho-corasick v0.7.6
   Compiling bstr v0.2.8
   Compiling clap v2.33.0
   Compiling regex v1.3.1
   Compiling ctrlc v3.1.3
   Compiling globset v0.4.4
   Compiling ignore v0.4.10
   Compiling jemalloc-sys v0.3.2
   Compiling jemallocator v0.3.2
   Compiling fd-find v7.4.0
    Finished release [optimized] target(s) in 55.56s
   Replacing /Users/robin/.cargo/bin/fd
    Replaced package `fd-find v7.4.0` with `fd-find v7.4.0` (executable `fd`)

The resulting binary segfaults as follows:

$ cd /tmp
$ fd oysters-clams-and-cockles
Segmentation fault: 11

On prying with lldb, I see this:

$ lldb `which fd`
(lldb) target create "/Users/robin/.cargo/bin/fd"
Current executable set to '/Users/robin/.cargo/bin/fd' (x86_64).
(lldb) r
Process 93119 launched: '/Users/robin/.cargo/bin/fd' (x86_64)
Process 93119 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010019989d fd`_rjem_je_tcache_bin_flush_small + 685
fd`_rjem_je_tcache_bin_flush_small:
->  0x10019989d <+685>: movq   (%rax), %rax
    0x1001998a0 <+688>: shlq   $0x10, %rax
    0x1001998a4 <+692>: sarq   $0x10, %rax
    0x1001998a8 <+696>: andq   $-0x2, %rax
Target 0: (fd) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x000000010019989d fd`_rjem_je_tcache_bin_flush_small + 685
    frame #1: 0x000000010019b722 fd`tcache_flush_cache + 34
    frame #2: 0x000000010019bb41 fd`tcache_destroy + 33
    frame #3: 0x000000010019cd91 fd`_rjem_je_tsd_cleanup + 65
    frame #4: 0x000000010019d0e6 fd`_rjem_je_malloc_tsd_boot1 + 54
    frame #5: 0x000000010016f1b6 fd`jemalloc_constructor + 662
    frame #6: 0x00000001004eb635 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 539
    frame #7: 0x00000001004eba5a dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
    frame #8: 0x00000001004e6933 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 493
    frame #9: 0x00000001004e569c dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
    frame #10: 0x00000001004e573c dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
    frame #11: 0x00000001004d44b2 dyld`dyld::initializeMainExecutable() + 199
    frame #12: 0x00000001004d9aab dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 6760
    frame #13: 0x00000001004d3227 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 453
    frame #14: 0x00000001004d3025 dyld`_dyld_start + 37
(lldb)

Note that the problem goes away if I build fd in debug mode:

$ cargo install fd-find --debug --force
    Updating crates.io index
  Installing fd-find v7.4.0
   Compiling libc v0.2.62
   Compiling memchr v2.2.1
   Compiling bitflags v1.2.0
   Compiling cc v1.0.45
   Compiling fs_extra v1.1.0
   Compiling log v0.4.8
   Compiling lazy_static v1.4.0
   Compiling unicode-width v0.1.6
   Compiling cfg-if v0.1.10
   Compiling regex-syntax v0.6.12
   Compiling vec_map v0.8.1
   Compiling nix v0.14.1
   Compiling strsim v0.8.0
   Compiling ansi_term v0.11.0
   Compiling same-file v1.0.5
   Compiling fnv v1.0.6
   Compiling void v1.0.2
   Compiling version_check v0.9.1
   Compiling ansi_term v0.12.1
   Compiling quick-error v1.2.2
   Compiling thread_local v0.3.6
   Compiling crossbeam-utils v0.6.6
   Compiling walkdir v2.2.9
   Compiling humantime v1.3.0
   Compiling lscolors v0.6.0
   Compiling crossbeam-channel v0.3.9
   Compiling aho-corasick v0.7.6
   Compiling bstr v0.2.8
   Compiling term_size v0.3.1
   Compiling atty v0.2.13
   Compiling num_cpus v1.10.1
   Compiling textwrap v0.11.0
   Compiling jemalloc-sys v0.3.2
   Compiling clap v2.33.0
   Compiling regex v1.3.1
   Compiling ctrlc v3.1.3
   Compiling globset v0.4.4
   Compiling ignore v0.4.10
   Compiling fd-find v7.4.0
   Compiling jemallocator v0.3.2
    Finished dev [unoptimized + debuginfo] target(s) in 20.91s
   Replacing /Users/robin/.cargo/bin/fd
    Replaced package `fd-find v7.4.0` with `fd-find v7.4.0` (executable `fd`)
$ cd /tmp
$ fd oysters-clams-and-cockles
$ echo $?
0

Happy to provide any other info as needed.

Thanks!

[sharkdp]

Thank you very much for reporting this.

This seems to be related to gnzlbg/jemallocator#136 and jemalloc/jemalloc#1633

Rolling back to fd v7.3.0 could work, because it did not include jemalloc. Alternatively, you could also take the current master and remove the lines that were added in this PR.

MacOS Catalina really seems to be cause (or reveal?) a lot of problems, first in bat, now here. Maybe related: https://tyler.io/broken/

[raw-bin]

You're welcome!

It does appear that this is related to jemallocator. Thanks for the pointers.
I'm able to work around the problem by simply using the debug builds for the moment.

My personal experience with Catalina has been quite positive TBH. I do hear that it's not been smooth sailing for a lot of folks.

I assume that this issue is to remain open until the upstream issue is resolved ?

Thanks.

PS: Thanks also for a great tool! :)

[fxcoudert]

I can confirm we are seeing this in Homebrew builds. This is fixed by applying the following patch:

diff -pur a/Cargo.toml b/Cargo.toml
--- a/Cargo.toml	2019-09-15 19:29:15.000000000 +0200
+++ b/Cargo.toml	2019-10-19 10:14:25.000000000 +0200
@@ -52,9 +52,6 @@ features = ["suggestions", "color", "wra
 [target.'cfg(all(unix, not(target_os = "redox")))'.dependencies]
 libc = "0.2"
 
-[target.'cfg(all(not(windows), not(target_env = "musl")))'.dependencies]
-jemallocator = "0.3.0"
-
 [dev-dependencies]
 diff = "0.1"
 tempdir = "0.3"
diff -pur a/src/main.rs b/src/main.rs
--- a/src/main.rs	2019-09-15 19:29:15.000000000 +0200
+++ b/src/main.rs	2019-10-19 10:14:39.000000000 +0200
@@ -35,11 +35,6 @@ use crate::internal::{
     pattern_has_uppercase_char, transform_args_with_exec, FileTypes,
 };
 
-// We use jemalloc for performance reasons, see https://github.com/sharkdp/fd/pull/481
-#[cfg(all(not(windows), not(target_env = "musl")))]
-#[global_allocator]
-static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;
-
 fn main() {
     let checked_args = transform_args_with_exec(env::args_os());
     let matches = app::build_app().get_matches_from(checked_args);

[sharkdp]

Waiting until jemalloc/jemalloc#1633 gets resolved...

[pravic]

Well, this still is relevant. Why wouldn't put jemalloc behind a feature? Or exclude it from macOS?

[sharkdp]

Why wouldn't put jemalloc behind a feature? Or exclude it from macOS?

fd is significantly faster using jemalloc. Sure, we could put it behind a feature. But it would be enabled by default. And I wouldn't recommend that any package authors disable the feature.

[pravic]

Sure, we could put it behind a feature. But it would be enabled by default

It's okay. It will be better to be able to install without a feature (cargo install --no-default-deatures) than install in debug mode (the only option for macOS users).

[fxcoudert]

Homebrew is already shipping fd without jemalloc, and we have 7,792 installs per month. There really is no other choice, given that with jemalloc in the default it segfaults.

However, if the issue is not fixed, we will have to consider dropping fd from the software we distribute: we do not want to patch and deviate from the upstream version from the long term.

[sharkdp]

The only thing we can do here (as a workaround) is to disable jemalloc on MacOS completely (I don't think Rust feature gates support filtering by OS version: https://doc.rust-lang.org/reference/conditional-compilation.html#target_os). This would be a (performance) regression for users of older MacOS versions as this only seems to be an issue on Catalina 😞.

What we would really need is someone with a MacOS Catalina setup that could help debug the original issue with jemalloc in jemalloc/jemalloc#1633

[sharkdp]

Fix released in fd 8.1.0. It would be great if someone could confirm this, as I did not manage to reproduce the segfault on the TravisCI build machines: #596

[pravic]

Confirm, 8.1.0 works well on Catalina.