Static binaries for Linux #528
Comments
|
Doesn’t spago depend on terminfo at the moment too? I think you’d need to drop that dependency for this to work, right? |
|
@hdgarrood I don't think so: IIRC we inherit that dependency from Dhall, which seems to be fine building static binaries with the same Nix machinery |
|
Interesting, thanks. |
|
@f-f I made a small proof-of-concept that builds let
# This is a derivation for dhall-1.27.0. nixpkgs master currently has
# dhall-1.28.0, but spago doesn't work with that yet.
# See below where this is used.
dhall_1_27_0_from_cabal2nix =
{ mkDerivation, aeson, aeson-pretty, ansi-terminal, base
, bytestring, case-insensitive, cborg, cborg-json, containers
, contravariant, cryptonite, data-fix, deepseq, Diff, directory
, doctest, dotgen, either, exceptions, filepath, foldl, gauge
, generic-random, hashable, haskeline, http-client, http-client-tls
, http-types, lens-family-core, megaparsec, memory, mockery, mtl
, network-uri, optparse-applicative, parsers, prettyprinter
, prettyprinter-ansi-terminal, profunctors, QuickCheck
, quickcheck-instances, repline, scientific, semigroups, serialise
, special-values, spoon, stdenv, tasty, tasty-expected-failure
, tasty-hunit, tasty-quickcheck, template-haskell, text
, th-lift-instances, transformers, transformers-compat, turtle
, unordered-containers, uri-encode, vector
}:
mkDerivation {
pname = "dhall";
version = "1.27.0";
sha256 = "e189fecd9ea22153252609a4d7c5cc4d61f2c36326b53758b61e5a851e701712";
isLibrary = true;
isExecutable = true;
libraryHaskellDepends = [
aeson aeson-pretty ansi-terminal base bytestring case-insensitive
cborg cborg-json containers contravariant cryptonite data-fix
deepseq Diff directory dotgen either exceptions filepath hashable
haskeline http-client http-client-tls http-types lens-family-core
megaparsec memory mtl network-uri optparse-applicative parsers
prettyprinter prettyprinter-ansi-terminal profunctors repline
scientific serialise template-haskell text th-lift-instances
transformers transformers-compat unordered-containers uri-encode
vector
];
executableHaskellDepends = [ base ];
testHaskellDepends = [
base bytestring cborg containers data-fix deepseq directory doctest
either filepath foldl generic-random lens-family-core megaparsec
mockery prettyprinter QuickCheck quickcheck-instances scientific
semigroups serialise special-values spoon tasty
tasty-expected-failure tasty-hunit tasty-quickcheck text
transformers turtle unordered-containers vector
];
benchmarkHaskellDepends = [
base bytestring containers directory gauge serialise text
];
description = "A configuration language guaranteed to terminate";
license = stdenv.lib.licenses.bsd3;
};
# This is an overlay that fixes spago so it builds in nixpkgs master.
spagoOverlay = self: super: {
haskell = super.haskell // {
packages = super.haskell.packages // {
ghc865 = super.haskell.packages.ghc865.override {
overrides = hself: hsuper: {
# Spago in nixpkgs is currently broken. Here is a PR fixing it up to work again:
# https://github.com/NixOS/nixpkgs/pull/75931#issuecomment-567526192
# However, instead of using `dhall-1.28.0`, it is easier just to go
# back to dhall-1.27.0.
spago = hsuper.spago.override {
dhall = hself.dhall_1_27_0;
# This is due to my mistake when originally packaging spago...
# https://github.com/NixOS/nixpkgs/pull/75931#issuecomment-567526192
directory = hself.directory;
};
# I'm not sure why callHackage doesn't work here.
# Maybe: https://github.com/nh2/static-haskell-nix/issues/9
# dhall_1_27_0 = hsuper.callHackage "dhall" "1.27.0" {};
dhall_1_27_0 = self.haskell.lib.dontCheck (self.haskell.lib.doJailbreak (hself.callPackage dhall_1_27_0_from_cabal2nix {}));
};
};
};
};
};
nixpkgs-src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/96d73edaf33f4c9ee452f65504bfdfab9a653eaa.tar.gz";
sha256 = "1qgxfaa23f982b6a9alghl3f8hgm61b4097bmkvn8084n05iv9sm";
};
nixpkgs = import nixpkgs-src {
overlays = [ spagoOverlay ];
};
static-haskell-nix-src = builtins.fetchTarball {
url = "https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz";
sha256 = "0ri4r77md77kmbqmx0j3h0paj538zpq4hki7vy8ycpwssm2xd442";
};
static-haskell-nix-survey-src = static-haskell-nix-src + "/survey/default.nix";
static-haskell-nix = import static-haskell-nix-survey-src {
normalPkgs = nixpkgs;
};
in
static-haskell-nix.haskellPackages.spagoThis is using the great If you have $ nix-build ./static-spago-test.nix
...
/nix/store/x1y6a6qh9rfkk2ngxi3602lsq87f8zrg-spago-0.12.1.0
$ file /nix/store/x1y6a6qh9rfkk2ngxi3602lsq87f8zrg-spago-0.12.1.0/bin/spago
/nix/store/x1y6a6qh9rfkk2ngxi3602lsq87f8zrg-spago-0.12.1.0/bin/spago: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
$ ldd /nix/store/x1y6a6qh9rfkk2ngxi3602lsq87f8zrg-spago-0.12.1.0/bin/spago
not a dynamic executable
$ /nix/store/x1y6a6qh9rfkk2ngxi3602lsq87f8zrg-spago-0.12.1.0/bin/spago version
0.12.1.0I didn't test everything, but it appears that The initial build took about 6 hours on my machine (although This is a good proof-of-concept, but it won't work for things like CI. nixpkgs isn't very good for Haskell packages that aren't in Stackage, since there is no way to easily have the correct version of dependencies you need. So if you bump some versions of spago's dependencies and release a new version of spago, we might have to update the spago derivation in nixpkgs to get it building again. The way to work around this is to automatically generate a nix Haskell package set based on the My recommendation on how to proceed with this would be as follows:
|
|
One last thing, if you change the last line of that nix script above to be $ nix-build static-spago-test.nix # with last line changed
...
/nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5
$ file /nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5/bin/purs
/nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5/bin/purs: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
$ ldd /nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5/bin/purs
not a dynamic executable
$ /nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5/bin/purs --version
0.13.5 [development build; commit: UNKNOWN]
$ /nix/store/h4588m743p091br79z0a0grsvrk7hipf-purescript-0.13.5/bin/purs
...Maybe @hdgarrood or the other PureScript maintainers would be interested in this. It doesn't look like there is currently a statically linked release of PureScript. |
|
@cdepillabout thank you so much for looking into this and for the great writeup ❤️ To answer your points:
|
|
@f-f Sounds good! However, it looks like there is already a statically linked version of spago uploaded: https://github.com/spacchetti/spago/releases/download/0.13.0/linux-static.tar.gz Did you already add this? Is this sufficient for now? Or is there something else you wanted me to add? |
|
@cdepillabout oh right, CI is building and uploading a broken static binary. Please remove that one and upload the good one there 😄 |
|
@f-f I've uploaded a static binary for spago for linux: https://github.com/spacchetti/spago/releases/download/0.13.0/linux-static.tar.gz It can be reproduced with this nix script: let
nixpkgs-src = builtins.fetchTarball {
# This is my branch that has spago-0.13.0 building:
# https://github.com/NixOS/nixpkgs/pull/76084
url = "https://github.com/NixOS/nixpkgs/archive/00303de7e2ef5967e2c8d51069c8c65dcb9ba09b.tar.gz";
sha256 = "1nylgj97fnb7si8v31vc22j0jd1n5zg80b7cdl3si3jq75mp1fkw";
};
nixpkgs = import nixpkgs-src { };
static-haskell-nix-src = builtins.fetchTarball {
url = "https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz";
sha256 = "0ri4r77md77kmbqmx0j3h0paj538zpq4hki7vy8ycpwssm2xd442";
};
static-haskell-nix-survey-src = static-haskell-nix-src + "/survey/default.nix";
static-haskell-nix = import static-haskell-nix-survey-src {
normalPkgs = nixpkgs;
};
createTarball = drv: nixpkgs.runCommand "${drv.name}-tarball" {} ''
mkdir -p "$out/"
# Create a temporary directory that we can eventually tar up.
tempdir="${drv.name}"
mkdir "$tempdir"
# Copy the spago and spago-curator binaries to the temporary directory.
cp ${drv}/bin/* "$tempdir/"
# Make sure temporary directory and binaries are writable by the user so
# they can easily rm it.
chmod u+w -R "$tempdir"
# Create the tarball.
tar -cf $out/linux-static.tar.gz "$tempdir"
'';
in
createTarball static-haskell-nix.haskellPackages.spago
# static-haskell-nix.haskellPackages.purescriptIf you save this script as $ nix-build ./static-spago-test.nix
...
/nix/store/1siy0765ax1v2xf57rn5i6bhpxab93gp-spago-0.13.0-tarball
$ ls /nix/store/1siy0765ax1v2xf57rn5i6bhpxab93gp-spago-0.13.0-tarball
linux-static.tar.gzIf people end up using this, maybe I can throw together another proof of concept using something like |
|
A few updates:
EDIT: I think we talked about using GitHub Actions to build this, and I just stumbled on a cachix action that we might want to use to speed builds up |
|
Thanks for the heads up. I just sent a PR updating the I've also modified the above nix script so that it builds a statically-linked let
nixpkgs-src = builtins.fetchTarball {
# This is my branch that has spago-0.13.1 building:
# https://github.com/NixOS/nixpkgs/pull/77484
url = "https://github.com/NixOS/nixpkgs/archive/6e2f340096703047b1eb56460f39b9b6642bbae4.tar.gz";
sha256 = "sha256:1di3kh42yp919j9y630zss810hi5i42z49k162nm192cwl296gsw";
};
nixpkgs = import nixpkgs-src { };
static-haskell-nix-src = builtins.fetchTarball {
url = "https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz";
sha256 = "0ri4r77md77kmbqmx0j3h0paj538zpq4hki7vy8ycpwssm2xd442";
};
static-haskell-nix-survey-src = static-haskell-nix-src + "/survey/default.nix";
static-haskell-nix = import static-haskell-nix-survey-src {
normalPkgs = nixpkgs;
};
createTarball = drv: nixpkgs.runCommand "${drv.name}-tarball" {} ''
mkdir -p "$out/"
# Create a temporary directory that we can eventually tar up.
tempdir="${drv.name}"
mkdir "$tempdir"
# Copy the spago and spago-curator binaries to the temporary directory.
cp ${drv}/bin/* "$tempdir/"
# Make sure temporary directory and binaries are writable by the user so
# they can easily rm it.
chmod u+w -R "$tempdir"
# Create the tarball.
tar -cf $out/linux-static.tar.gz "$tempdir"
'';
in
createTarball static-haskell-nix.haskellPackages.spago
# static-haskell-nix.haskellPackages.spago
# static-haskell-nix.haskellPackages.purescriptHowever, it seems like it will take a very long time on my machine as well, so I'll ping you again on this thread when it finishes compiling.
Yeah, I think that is a good idea. Using the The problem is that we ideally want to generate the nix build plan from the current I'm also not sure how easy it is to get After the above build finishes, I'll add a TODO for myself to look into compiling completely static executables with |
|
The nix script from the above post (#528 (comment)) did not work. It looks like there was some problem with using static-haskell-nix with a recent version of nixpkgs, but I didn't really investigate to figure out why. Instead, I decided to use the older (working) script from #528 (comment) and just update it to compile with spago-0.13.1 instead of spago-0.13.0. This was easy to do because spago's dependencies appear to have not really changed between 0.13.0 and 0.13.1. If they had changed drastically, then this would have been significantly more work: let
nixpkgs-src = builtins.fetchTarball {
# This is my branch that has spago-0.13.0 building.
# (See below how I get spago-0.13.1 instead):
# https://github.com/NixOS/nixpkgs/pull/76084
url = "https://github.com/NixOS/nixpkgs/archive/00303de7e2ef5967e2c8d51069c8c65dcb9ba09b.tar.gz";
sha256 = "1nylgj97fnb7si8v31vc22j0jd1n5zg80b7cdl3si3jq75mp1fkw";
};
# This is an overlay that overrides the version of spago.
#
# Building spago-0.13.1 from https://github.com/NixOS/nixpkgs/pull/77484
# didn't work because of what seemed to be a problem with
# static-haskell-nix.
#
# https://github.com/NixOS/nixpkgs/issues/76879#issuecomment-572872130
#
# So what the following does is overrides the source of the spago
# derivation to use 0.13.1 instead of 0.13.0. This is easy to do
# because 0.13.1 doesn't add any Haskell packages dependencies,
# change required versions, etc.
spago-0_13_1-overlay = self: super: {
haskell = super.haskell // {
packages = super.haskell.packages // {
ghc865 = super.haskell.packages.ghc865.override (oldAttrs: {
overrides = self.lib.composeExtensions (oldAttrs.overrides or (_: _: {})) (hself: hsuper: {
spago = self.haskell.lib.overrideCabal hsuper.spago {
version = "0.13.1";
# This is the source for version 0.13.1.
src = self.fetchgit {
url = "https://github.com/spacchetti/spago.git";
sha256 = "0l6sy1hz5dbnrjkvb2f44afhd48nwqx5kx1h29ns93xbbd57hci8";
rev = "b87858609c671d8f3dc78f858ce1d8c492bd1062";
fetchSubmodules = true;
};
};
});
});
};
};
};
nixpkgs = import nixpkgs-src { overlays = [ spago-0_13_1-overlay ]; };
static-haskell-nix-src = builtins.fetchTarball {
url = "https://github.com/nh2/static-haskell-nix/archive/d24dea3d46a727e1cd93e67458ce2768109efe0a.tar.gz";
sha256 = "0ri4r77md77kmbqmx0j3h0paj538zpq4hki7vy8ycpwssm2xd442";
};
static-haskell-nix-survey-src = static-haskell-nix-src + "/survey/default.nix";
static-haskell-nix = import static-haskell-nix-survey-src {
normalPkgs = nixpkgs;
};
createTarball = drv: nixpkgs.runCommand "${drv.name}-tarball" {} ''
mkdir -p "$out/"
# Create a temporary directory that we can eventually tar up.
tempdir="${drv.name}"
mkdir "$tempdir"
# Copy the spago and spago-curator binaries to the temporary directory.
cp ${drv}/bin/* "$tempdir/"
# Make sure temporary directory and binaries are writable by the user so
# they can easily rm it.
chmod u+w -R "$tempdir"
# Create the tarball.
tar -cf $out/linux-static.tar.gz "$tempdir"
'';
in
createTarball static-haskell-nix.haskellPackages.spago
# static-haskell-nix.haskellPackages.purescriptI've uploaded the tarball with the static executable to the 0.13.1 release: https://github.com/spacchetti/spago/releases/download/0.13.1/linux-static.tar.gz In the next couple weeks, I'll try throwing together a prototype of using |
|
I was wondering if we could cook some script ourselves that:
This would allow us to:
|
|
@f-f I threw together a very small example of building It is on this branch: https://github.com/cdepillabout/spago/tree/add-haskell.nix If you checkout this branch, you should be able to build the $ git clone https://github.com/cdepillabout/spago
$ git checkout add-haskell.nix
$ nix-build ./nix -A hs-pkgs.spago.components.exes.spago
$ ./result/bin/spago version
0.13.1Unfortunately, this doesn't work yet with full static linking. Here's an example command line that can be used to build a statically linked spago exe, but if you try this, you'll see it fails (with errors that appear to be related to static linking): Let me try to play around with this a little bit, and if I can't figure out how to make it work, I'll ask nh2 and the haskell.nix developers if they have any suggestions.
So there actually is a program like this: https://github.com/input-output-hk/stack2nix There is even support directly in https://github.com/nh2/static-haskell-nix/tree/master/static-stack2nix-builder-example However, it appears that If it is possible to haskell.nix working, I'd recommend we go with that. |
|
I wanted to give an update on what I've done with this so far. I've created some nix files that are able to use haskell.nix to successfully build all the dependencies of Here are the two files from the branch I am working on:
let
# haskell.nix master as of 2019-12-11
haskell-nix = import (builtins.fetchTarball {
url = https://github.com/input-output-hk/haskell.nix/archive/dbb83a4c7071f8e5ed9575267fbe4539c15e35df.tar.gz;
sha256 = "sha256:1mz7kr45061c5hfvmmk4r2vbipzsg19p3whcy6y0z41ngrx0idkl";
});
# nixpkgs master as of 2020-01-26
nixpkgs-src = builtins.fetchTarball {
url = https://github.com/NixOS/nixpkgs/archive/ba8fbd5352b8aca9410b10c8aa78e84740529e87.tar.gz;
sha256 = "sha256:0sanh2h4h60ir6mg12m6ckqamzgnipfdkszg1kl4qv39hdmy9xzm";
};
nixpkgs = import nixpkgs-src {
config = haskell-nix.config;
overlays = haskell-nix.overlays ++ [ (import ./spago-overlay.nix) ];
};
in
nixpkgs.pkgsStatic.hs-pkgs.spago.components.exes.spago
self: super:
let
inherit (self.haskell-nix) stackProject;
inherit (self.haskell-nix.haskellLib) cleanGit;
docsSearchAppJsFile = self.fetchurl {
url = "https://github.com/spacchetti/purescript-docs-search/releases/download/v0.0.5/docs-search-app.js";
sha256 = "11721x455qzh40vzfmralaynn9v8b5wix86r107hhs08vhryjib2";
};
purescriptDocsSearchFile = self.fetchurl {
url = "https://github.com/spacchetti/purescript-docs-search/releases/download/v0.0.5/purescript-docs-search";
sha256 = "16p1fmdvpwz1yswav8qjsd26c9airb22xncqw1rjnbd8lcpqx0p5";
};
spago-cleaned-source =
let orig-src = cleanGit { src = ../.; };
in
self.runCommand "fixup-spago-source" {} ''
mkdir -p "$out"
cp -r "${orig-src}"/. "$out"
chmod -R +w "$out"
# The source for spago is pulled directly from GitHub. It uses a
# package.yaml file with hpack, not a .cabal file. In the package.yaml file,
# it uses defaults from the master branch of the hspec repo. It will try to
# fetch these at build-time (but it will fail if running in the sandbox).
#
# The following line modifies the package.yaml to not pull in
# defaults from the hspec repo.
substituteInPlace "$out/package.yaml" --replace 'defaults: hspec/hspec@master' ""
# Spago includes the following two files directly into the binary
# with Template Haskell. They are fetched at build-time from the
# `purescript-docs-search` repo above. If they cannot be fetched at
# build-time, they are pulled in from the `templates/` directory in
# the spago source.
#
# However, they are not actually available in the spago source, so they
# need to fetched with nix and put in the correct place.
# https://github.com/spacchetti/spago/issues/510
cp ${docsSearchAppJsFile} "$out/templates/docs-search-app.js"
cp ${purescriptDocsSearchFile} "$out/templates/purescript-docs-search"
'';
hs-pkgs = stackProject {
src = spago-cleaned-source;
modules = [
({pkgs,config,...}: {
# Error with the haddocks in the fail library.
packages.fail.doHaddock = false;
# Error with the haddocks in the github library.
packages.github.doHaddock = false;
# Older versions of clock can't be cross compiled:
# https://github.com/corsis/clock/issues/48
# TODO: Probably should just bump this in stack.yaml.
packages.clock.package.identifier.version = pkgs.lib.mkForce "0.8";
packages.clock.sha256 = pkgs.lib.mkForce "sha256:0539w9bjw6xbfv9v6aq9hijszxqdnqhilwpbwpql1400ji95r8q8";
# TODO: https://github.com/jacobstanley/unix-compat/pull/43
packages.unix-compat.package.identifier.version = pkgs.lib.mkForce "0.5.2";
packages.unix-compat.sha256 = pkgs.lib.mkForce "sha256:1a8brv9fax76b1fymslzyghwa6ma8yijiyyhn12msl3i5x24x6k5";
# Disable haddocks to save time.
doHaddock = false;
# I'm trying (and failing) to get spago to build correctly.
packages.spago.components.library.configureFlags = [
"--ghc-option=-j1"
"--verbose"
"--ghc-option=-v"
# "--ld-option=-Wl,--start-group --ld-option=-Wl,-lstdc++"
# "--ghc-option=-lstdc++"
# "--ghc-option=-lgcc_s"
# "--ghc-option=-optl-lstdc++"
];
enableShared = false;
})
];
};
in
{
inherit hs-pkgs spago-cleaned-source;
}These files are available on this branch: https://github.com/cdepillabout/spago/tree/add-haskell.nix Most recent commit: https://github.com/cdepillabout/spago/tree/ad181828e19cdda5690b7dc7b70f102e90e1f378 When building $ nix-build ./nix
...
these derivations will be built:
/nix/store/6l26d943hxq9an2jv2ngzn810a691q22-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl.drv
/nix/store/apnchqqhj2rpmfi9ya5ds0wccgbgl5b7-spago-0.13.1-exe-spago-config.drv
/nix/store/12ky7giidfv2jbs5rki3vnw371nyxm56-spago-0.13.1-exe-spago-x86_64-unknown-linux-musl-ghc-8.4.4.drv
/nix/store/n8y33m0bl4xixnzzihlfqs1dypymygn5-spago-0.13.1-exe-spago-x86_64-unknown-linux-musl.drv
building '/nix/store/6l26d943hxq9an2jv2ngzn810a691q22-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl.drv'...
unpacking sources
unpacking source archive /nix/store/yimq3m4gfz9sm6d0s00fqq748r0kh4js-fixup-spago-source
source root is fixup-spago-source
patching sources
generated spago.cabal
updateAutotoolsGnuConfigScriptsPhase
configuring
Configure flags:
--prefix=/nix/store/d355liw7xa5jg88yv20zfsppy9l6dx5w-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl lib:spago --package-db=clear --package-db=/nix/store/kxs4wvm1kib3y0zhb3alqdxhi9b8v5dp-spago-0.13.1-lib-spago-config/package.conf.d --flags=-static --exact-configuration --dependency=Glob=Glob-0.10.0-AGTIc43lefE3A3ant9v9rM --dependency=aeson=aeson-1.4.4.0-AO7MsH5x9MwFgTYCwz64pJ --dependency=aeson-pretty=aeson-pretty-0.8.7-CeAZz7qMwvH5TsxMbWC6qN --dependency=ansi-terminal=ansi-terminal-0.8.2-C2wvbH9cfUiNntnO82a0v --dependency=async-pool=async-pool-0.9.0.2-2e4jHWW8iMoDgXWNXU6Fe5 --dependency=bower-json=bower-json-1.0.0.1-FnrrT0LLaRd8DHFLl7HBeU --dependency=dhall=dhall-1.27.0-8cAkPvXHDbt1kKv7jafUUD --dependency=either=either-5-1nv4KpSKpI0I12bwumjr5P --dependency=exceptions=exceptions-0.10.2-1a1thv9cmEVDN2LmIne0d5 --dependency=file-embed=file-embed-0.0.10.1-JBj0CTa3b3pL2aTV19QFZK --dependency=foldl=foldl-1.4.5-742pyejvavK9Rrz2pNybym --dependency=fsnotify=fsnotify-0.3.0.1-Aw7b9RuGFsvK32rCq9iIoI --dependency=github=github-0.23-8zWbcOYeNvq9vGdYSQaxV1 --dependency=http-client=http-client-0.5.14-KhZVMYATvosctax2xEF28 --dependency=http-conduit=http-conduit-2.3.2-ClXNcza069cARwJ0IxupRS --dependency=lens-family-core=lens-family-core-1.2.3-6tviOhl6sxTFAIlrdcxXEK --dependency=megaparsec=megaparsec-7.0.3-AY94YCsBWGmE6zCaPLtzUr --dependency=network-uri=network-uri-2.6.1.0-AstEwZoXrlUJQq4VkxaVo9 --dependency=open-browser=open-browser-0.2.1.0-Lj74S9sKrTXAgLlyjQqhhk --dependency=prettyprinter=prettyprinter-1.2.1-IIS5PzLn6frEDfvRrnOWaa --dependency=retry=retry-0.7.7.0-JNkYNcyJqjOGddLzxKEqHI --dependency=rio=rio-0.1.12.0-DfECtGrJlzZyakZprFBTn --dependency=rio-orphans=rio-orphans-0.1.1.0-1hwFtpyV2Xt1K5fN4GdvRG --dependency=safe=safe-0.3.17-HEmZRpXGD3s2ywM3rHJ1ur --dependency=semver-range=semver-range-0.2.8-7qMA0TdMtoj1U6gpYnk3j5 --dependency=stm=stm-2.4.5.1-1n1oRyz2WxkGi5pMiXlR3y --dependency=tar=tar-0.5.1.0-Gm2BD2cAsZs6CVhrbM8qEk --dependency=temporary=temporary-1.3-8eiCgFC798X6vEe6in5JNI --dependency=turtle=turtle-1.5.14-5ZUaegLFU89gaV2MDH2HN --dependency=unliftio=unliftio-0.2.12-68mZyv2LrEk7VjhEZiUuTd --dependency=unordered-containers=unordered-containers-0.2.10.0-8NfqYq7yMXo7Rw920KREQX --dependency=vector=vector-0.12.0.1-GGZqQZyzchy8YFPCF67wxL --dependency=versions=versions-3.5.0-B9DYw1kr3P85sUMHITeFqL --dependency=zlib=zlib-0.6.2-5t89gesQUWsKgPz4IjNOwP --dependency=rts=rts --dependency=ghc-prim=ghc-prim-0.5.2.0 --dependency=integer-gmp=integer-gmp-1.0.2.0 --dependency=base=base-4.11.1.0 --dependency=deepseq=deepseq-1.4.3.0 --dependency=array=array-0.5.2.0 --dependency=ghc-boot-th=ghc-boot-th-8.4.4 --dependency=pretty=pretty-1.1.3.6 --dependency=template-haskell=template-haskell-2.13.0.0 --dependency=ghc-boot=ghc-boot-8.4.4 --dependency=ghc=ghc-8.4.4 --dependency=Cabal=Cabal-2.2.0.1 --dependency=array=array-0.5.2.0 --dependency=binary=binary-0.8.5.1 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.5.11.0 --dependency=directory=directory-1.3.1.5 --dependency=filepath=filepath-1.4.2 --dependency=ghc-boot=ghc-boot-8.4.4 --dependency=ghc-compact=ghc-compact-0.1.0.0 --dependency=ghc-prim=ghc-prim-0.5.2.0 --dependency=hpc=hpc-0.6.0.3 --dependency=mtl=mtl-2.2.2 --dependency=parsec=parsec-3.1.13.0 --dependency=process=process-1.6.3.0 --dependency=text=text-1.2.3.1 --dependency=time=time-1.8.0.2 --dependency=transformers=transformers-0.5.5.0 --dependency=unix=unix-2.7.2.2 --with-ghc=x86_64-unknown-linux-musl-ghc --with-ghc-pkg=x86_64-unknown-linux-musl-ghc-pkg --with-hsc2hs=x86_64-unknown-linux-musl-hsc2hs --with-gcc=x86_64-unknown-linux-musl-cc --with-ld=x86_64-unknown-linux-musl-ld.gold --ghc-option=-optl-fuse-ld=gold --ld-option=-fuse-ld=gold --with-ar=x86_64-unknown-linux-musl-ar --with-strip=x86_64-unknown-linux-musl-strip --disable-executable-stripping --disable-library-stripping --disable-library-profiling --disable-executable-profiling --enable-static --disable-shared --enable-split-sections --hsc2hs-option=--cross-compile --ghc-option=-j1
Configuring library for spago-0.13.1..
Warning: The flag --disable-executable-profiling is deprecated. Please use
--disable-profiling instead.
building
Preprocessing library for spago-0.13.1..
Building library for spago-0.13.1..
[ 1 of 22] Compiling Paths_spago ( dist/build/autogen/Paths_spago.hs, dist/build/Paths_spago.o )
[ 2 of 22] Compiling Spago.Prelude ( src/Spago/Prelude.hs, dist/build/Spago/Prelude.o )
[ 3 of 22] Compiling Spago.Messages ( src/Spago/Messages.hs, dist/build/Spago/Messages.o )
[ 4 of 22] Compiling Spago.Git ( src/Spago/Git.hs, dist/build/Spago/Git.o )
[ 5 of 22] Compiling Spago.DryRun ( src/Spago/DryRun.hs, dist/build/Spago/DryRun.o )
[ 6 of 22] Compiling Spago.Dhall ( src/Spago/Dhall.hs, dist/build/Spago/Dhall.o )
[ 7 of 22] Compiling Spago.Build.Parser ( src/Spago/Build/Parser.hs, dist/build/Spago/Build/Parser.o )
[ 8 of 22] Compiling Spago.PscPackage ( src/Spago/PscPackage.hs, dist/build/Spago/PscPackage.o )
[ 9 of 22] Compiling Spago.Purs ( src/Spago/Purs.hs, dist/build/Spago/Purs.o )
[10 of 22] Compiling Spago.TH ( src/Spago/TH.hs, dist/build/Spago/TH.o )
[11 of 22] Compiling Spago.Templates ( src/Spago/Templates.hs, dist/build/Spago/Templates.o )
<command line>: can't load .so/.DLL for: /nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/lib/gcc/x86_64-unknown-linux-musl/9.2.0/../../../../x86_64-unknown-linux-musl/lib/../lib64/libstdc++.so (Error loading shared library libgcc_s.so.1: No such file or directory (needed by /nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/lib/gcc/x86_64-unknown-linux-musl/9.2.0/../../../../x86_64-unknown-linux-musl/lib/../lib64/libstdc++.so))
builder for '/nix/store/6l26d943hxq9an2jv2ngzn810a691q22-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl.drv' failed with exit code 1The error in question is the following: If you run with verbose logging, you can see that this is happening during the bytecode generation for the template haskell stuff in the module It appears to be able to load and link a bunch of Haskell packages, but it fails on the package $ nix-build ./nix # with verbose logging enabled:
these derivations will be built:
/nix/store/6l26d943hxq9an2jv2ngzn810a691q22-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl.drv
...
[ 9 of 22] Compiling Spago.Purs ( src/Spago/Purs.hs, dist/build/Spago/Purs.o )
[10 of 22] Compiling Spago.TH ( src/Spago/TH.hs, dist/build/Spago/TH.o )
[11 of 22] Compiling Spago.Templates ( src/Spago/Templates.hs, dist/build/Spago/Templates.o )
*** Parser [Spago.Templates]:
!!! Parser [Spago.Templates]: finished in 0.49 milliseconds, allocated 0.943 megabytes
*** Renamer/typechecker [Spago.Templates]:
*** Simplify [expr]:
!!! Simplify [expr]: finished in 0.03 milliseconds, allocated 0.008 megabytes
*** CorePrep [expr]:
!!! CorePrep [expr]: finished in 0.03 milliseconds, allocated 0.007 megabytes
*** ByteCodeGen [Ghci1]:
!!! ByteCodeGen [Ghci1]: finished in 0.07 milliseconds, allocated 0.028 megabytes
Loading package ghc-prim-0.5.2.0 ... linking ... done.
Loading package integer-gmp-1.0.2.0 ... linking ... done.
Loading package base-4.11.1.0 ... linking ... done.
Loading package array-0.5.2.0 ... linking ... done.
Loading package deepseq-1.4.3.0 ... linking ... done.
Loading package bytestring-0.10.8.2 ... linking ... done.
Loading package containers-0.5.11.0 ... linking ... done.
Loading package binary-0.8.5.1 ... linking ... done.
...
Loading package socks-0.5.6 ... linking ... done.
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name libpthread.so
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name liblibpthread.so
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name pthread.lib
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name libpthread.lib
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name libpthread.dll.a
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name pthread.dll.a
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/sv6pkifh6z10ym5f6r4n5m0ns81hnkxn-cryptonite-0.25-lib-cryptonite-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/cryptonite-0.25-IaCpxy7BvkdAANiQkOvlT --print-file-name libpthread.a
Loading package cryptonite-0.25 ... linking ... done.
Loading package hourglass-0.2.12 ... linking ... done.
Loading package asn1-types-0.3.2 ... linking ... done.
...
Loading package uri-bytestring-0.3.2.1 ... linking ... done.
Loading package http-api-data-0.3.8.1 ... linking ... done.
*** gcc:
/nix/store/si25mjcpqds0fm9ijz2298lbjyy0k7ik-x86_64-unknown-linux-musl-stage-final-gcc-debug-wrapper-9.2.0/bin/x86_64-unknown-linux-musl-cc -fno-stack-protector -DTABLES_NEXT_TO_CODE '-fuse-ld=gold' -B/nix/store/ggzf0fr60dk7g2qz0qxqvl0ngj96xwxk-double-conversion-2.0.2.0-lib-double-conversion-x86_64-unknown-linux-musl/lib/x86_64-linux-ghc-8.4.4/double-conversion-2.0.2.0-GcovPDvJowg3g0qUx9qq3o --print-file-name 'libstdc++.so'
Loading package double-conversion-2.0.2.0 ... *** Deleting temp files:
Deleting: /build/ghc100_0/ghc_1.hscpp
*** Deleting temp dirs:
Deleting: /build/ghc100_0
<command line>: can't load .so/.DLL for: /nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/lib/gcc/x86_64-unknown-linux-musl/9.2.0/../../../../x86_64-unknown-linux-musl/lib/../lib64/libstdc++.so (Error loading shared library libgcc_s.so.1: No such file or directory (needed by /nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/lib/gcc/x86_64-unknown-linux-musl/9.2.0/../../../../x86_64-unknown-linux-musl/lib/../lib64/libstdc++.so))
builder for '/nix/store/fynlzqjqmxb7kd5gxyc0bygg3v3l2mh7-spago-0.13.1-lib-spago-x86_64-unknown-linux-musl.drv' failed with exit code 1You can see that GHC has died shortly after running the command: However, this does work correctly on my machine when run from the command line, so this seems to indicate that GHC has died doing something shortly after running this command. The error message above says the shared object However, $ ls /nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/x86_64-unknown-linux-musl/lib64/libgcc_s.so.1
/nix/store/h3k8c62sxdn5zprfv927gf5vy0w4kcns-x86_64-unknown-linux-musl-stage-final-gcc-debug-9.2.0/x86_64-unknown-linux-musl/lib64/libgcc_s.so.1 |
|
I've uploaded a statically linked https://github.com/purescript/spago/releases/download/0.14.0/linux-static.tar.gz This was built with the following nix file: let
nixpkgs-src = builtins.fetchTarball {
# This is my branch that has spago-0.14.0 building.
# https://github.com/NixOS/nixpkgs/pull/79685
url = "https://github.com/NixOS/nixpkgs/archive/966910c58f30e6c2645210f2cf72e27fb3408d86.tar.gz";
sha256 = "11gmq125r7w12iwgg6js5gl33cw2fnbf0vx99n2m33gnswpvzrs7";
};
nixpkgs = import nixpkgs-src {};
static-haskell-nix-src = builtins.fetchTarball {
url = "https://github.com/nh2/static-haskell-nix/archive/38ef5a4e00b5f6fb421014829320d85899483874.tar.gz";
sha256 = "0mv93wpl507fz6nyp11bv14i4njc3aajnlq0y7pg942d5haws7d8";
};
static-haskell-nix-survey-src = static-haskell-nix-src + "/survey/default.nix";
static-haskell-nix = import static-haskell-nix-survey-src {
normalPkgs = nixpkgs;
};
createTarball = drv: nixpkgs.runCommand "${drv.name}-tarball" {} ''
mkdir -p "$out/"
# Create a temporary directory that we can eventually tar up.
tempdir="${drv.name}"
mkdir "$tempdir"
# Copy the spago and spago-curator binaries to the temporary directory.
cp ${drv}/bin/* "$tempdir/"
# Make sure temporary directory and binaries are writable by the user so
# they can easily rm it.
chmod u+w -R "$tempdir"
# Create the tarball.
tar -cf $out/linux-static.tar.gz "$tempdir"
'';
in
createTarball static-haskell-nix.haskellPackages.spago
# static-haskell-nix.haskellPackages.purescript(If you uncomment the last line, you can also build a completely statically-linked version of As a follow-up for trying to get haskell.nix building spago statically linked, I haven't made any progress since my last comment. However, a comment on @nh2's static-haskell-nix repo seemed to indicate that |
The reason for that is the following:
|
|
There is a new fix on haskell.nix that appears to fix the problem I was seeing above: input-output-hk/haskell.nix#448 I should try this fix with the code from above (as well as nh2's suggestion to use |
|
Hi, I've got no idea what this repo is and what it does, but if you want a simple, clean example of building working and small static binaries for Linux with |
|
@balsoft Thanks a lot for the link! I'll definitely check it out. |
|
Hello, Everyone! My setup: The logs: On journalctl: May I help with something? -- Edit: -- Edit 2: |
|
@naripok cool, thanks for the report. It looks like the process crashes while trying to reach the internet - at the moment where it crashes it's supposed to reach for GitHub to ask for the latest release - and on Ubuntu this is usually solvable by installing Btw, have you tried installing |
|
Hey, @f-f, -- Edit: From yay, searching for these files: And I can confirm that the relevant files are present: |
|
Right. It could be that the |
|
@f-f I haven't uploaded a static release for 0.15.2 yet. However, I think at one point you were saying that the current release process creates a In the past, I've just deleted this file, and uploaded a new I haven't done this yet for the 0.15.2 release. Could this be the problem that @naripok is seeing? @naripok If you're interested in looking into this, you could check out the statically-linked binary for 0.14.0, and seeing if that works for you. |
|
@cdepillabout oh right, thanks for noticing, I totally forgot about that 🙈 I just removed the broken static binary in the latest release, and opened #620 so that we won't upload it anymore in future releases |
|
I wonder if we could just use this instead of Nix? https://github.com/utdemir/ghc-musl |
|
@f-f Yeah, that might be much easier to get working and maintain than a Nix-based approach. |
|
It depends on whether you have native dependencies (e.g. use libraries written in C or other languages). WIthout native dependencies, it's usually enough to just have a musl-linked GHC and the docker approach is enough. With native dependencies, the nix approach is often better, because it is easier to override your native dependencies to provide static libraries. |
|
Update: I just noticed that Stack started building static binaries for linux too, PR here: commercialhaskell/stack#5267 They are building with Alpine, maybe we can pull off something like that too by just adding another Travis build based on an Alpine image? |
|
In the end I feel that the main reason why we needed static builds was to avoid all the issues with |
|
I've sent a proof-of-concept PR getting static builds of |
We should publish statically-linked binaries for our Linux releases. The reason for doing this is that various Linux distros ship different versions of some native libraries we link to, and dynamically linking means that the binary doesn't work out of the box on some of them, and people have to install dependencies with their package manager, etc etc.
It would be nice if we could instead build and publish a statically linked binary. This has been tried in the past (e.g. see #437), but failed because we were still linking to
glibc, while to properly link statically we should link tomusl.The easiest way to compile Haskell statically with no hassle is probably with
static-haskell-nix.So to put this into use we should:
This kind of opens up the question of "why use the whole
stackthing if we can just build with Nix", which is relevant because I wouldn't want to maintain two build systems that need attention when doing routine things like updating a dependency. So we could figure out a way to reduce this burden - e.g. haskell.nix might help with this?cc @cdepillabout
The text was updated successfully, but these errors were encountered: