Output of packages using jemalloc-sys rust crate is dependend on host kernel pagesize

[yu-re-ka]

Some of the build parameters are auto-detected by jemalloc-sys:

https://github.com/gnzlbg/jemallocator/tree/master/jemalloc-sys#environment-variables

JEMALLOC_SYS_WITH_LG_PAGE=: [...] By default the configure script determines the host's page size and sets the allocator page size equal to the system page size, so this option need not be specified unless the system page size may change between configuration and execution, e.g. when cross compiling

In our jemalloc package, we already consider support for >4k page size on aarch64, but jemalloc-sys builds its own jemalloc and thus these fixes are not applied there.

    # AArch64 has configurable page size up to 64k. The default configuration
    # for jemalloc only supports 4k page sizes.
    ++ lib.optional stdenv.isAarch64 "--with-lg-page=16"

[yuka@m1:~]$ zgrep 16K /proc/config.gz 
CONFIG_ARM64_16K_PAGES=y

[yuka@m1:~]$ nix run nixpkgs#fd
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
memory allocation of 5 bytes failed
Aborted (core dumped)

[yuka@m1:~]$ nix build nixpkgs#fd --rebuild
error: derivation '/nix/store/anmwaczqh86waixcbfcklypj495nk8xg-fd-8.5.3.drv' may not be deterministic: output '/nix/store/74y46lga9v6rpc1b9qgqq1chh14mq4d7-fd-8.5.3' differs

[yuka@m1:~]$

IMO the best solution would be to make jemalloc-sys use our distro's jemalloc package everywhere, using the JEMALLOC_OVERRIDE env variable.

For instance, this would work to fix fd:

diff --git a/pkgs/tools/misc/fd/default.nix b/pkgs/tools/misc/fd/default.nix
index 6dd0e7a6e7c..ec45551a517 100644
--- a/pkgs/tools/misc/fd/default.nix
+++ b/pkgs/tools/misc/fd/default.nix
@@ -1,4 +1,4 @@
-{ lib, fetchFromGitHub, rustPlatform, installShellFiles }:
+{ lib, fetchFromGitHub, rustPlatform, installShellFiles, jemalloc }:
 
 rustPlatform.buildRustPackage rec {
   pname = "fd";
@@ -15,6 +15,13 @@ rustPlatform.buildRustPackage rec {
 
   nativeBuildInputs = [ installShellFiles ];
 
+  JEMALLOC_OVERRIDE = "${jemalloc.overrideAttrs (oldAttrs: {
+    configureFlags = oldAttrs.configureFlags ++ [
+      "--with-jemalloc-prefix=_rjem_"
+      "--with-private-namespace=_rjem_"
+    ];
+  })}/lib/libjemalloc.so";
+
   preFixup = ''
     installManPage doc/fd.1
 

[montchr]

For context, this affects all users running NixOS on Apple hardware by way of https://github.com/tpwrules/nixos-apple-silicon's Asahi Linux kernel and other packages.

From Asahi Linux's initial alpha release notes:

The Asahi Linux kernel is compiled to use 16K pages. This both performs better, and is required with our kernel branch right now in order to work properly with the M1’s IOMMUs. Unfortunately, some Linux software has problems running with 16K pages.

As suggested in sharkdp/fd#1085, passing the JEMALLOC_SYS_WITH_LG_PAGE=16 environment variable definition to the build process works to resolve the issue.

So, at least in the case of fd, that patch could be simplified to the following (tested locally):

diff --git a/pkgs/tools/misc/fd/default.nix b/pkgs/tools/misc/fd/default.nix
index b1845451491..ff603112a6a 100644
--- a/pkgs/tools/misc/fd/default.nix
+++ b/pkgs/tools/misc/fd/default.nix
@@ -15,6 +15,8 @@ rustPlatform.buildRustPackage rec {
 
   nativeBuildInputs = [ installShellFiles ];
 
+  JEMALLOC_SYS_WITH_LG_PAGE = 16;
+
   # skip flaky test
   checkFlags = [
     "--skip=test_owner_current_group"

But I'm guessing that this case-by-case workaround isn't the most efficient way to provide a fix – #243324 seems simpler.