Implement installation and binary/source distribution rules

[snowleopard]

There is no support for this in the build system yet. See install, sdist, binary-dist rules in makefiles.

Here is the todo list:

• Installation
• Source distribution (done, thanks @KaiHa!)
• Binary distribution

[snowleopard]

@KaiHa Thank you for starting to work on this issue. Is it OK if I add you as a collaborator on the project? Otherwise I can't assign this issue to you.

[KaiHa]

Is it OK if I add you as a collaborator on the project?

Yes, thats OK

[snowleopard]

@KaiHa Done! Please don't push directly to the repository, keep using pull requests, so that I can keep track of what's going on.

[KaiHa]

Done! Please don't push directly to the repository

Thanks, hopefully this will prevent me from pushing into your repository by accident.

[KaiHa]

@snowleopard if I need a function like wildCheckCase, can I add a build dependency to MissingH, or do you prefer a different approach?

[ndmitchell]

@KaiHa MissingH is very much a library from a different era - most of the things it can do are done better by other things. As an example, Shake provides ?== which is used pervasively throughout the rest of Hadrian. If you need any help, give me the example of what you'd pass to wildCheckCase.

[snowleopard]

@KaiHa I agree with Neil, so far Shake's ?== was sufficient for most things. Where it wasn't I introduced simple functions like matchVersionedFilePath instead of adding new package dependencies.

[KaiHa]

Thanks @snowleopard, @ndmitchell, ?== is exactly what I was looking for. @ndmitchell did you notice that in hackage the documentation of ?== is missing although it is in the source code.

[KaiHa]

As @thomie suggested some files need to be generated by alex and happy before they can get included into the source distribution. Where should I put these generated files?

If I put them into the same place as in the legacy tarball it would be easy to spot differences between tarballs generated by make and hadrian. But hadrian will need extra rules if building from a source distribution. Should we add alex and happy as build dependencies for building from a source distribution and do not add these generated files at all? @snowleopard what do you think?

[ndmitchell]

@KaiHa - I noticed last night when I wrote a documentation scanner for all my Haskell projects, and fixed it. Interesting how things often seem to be discovered independently and simultaneously! Thanks for pointing it out.

[snowleopard]

@KaiHa Hadrian already generates these files and puts them into _build directory. I guess if you need these files built for the sdist rule, you can simply need them.

Have a look at src/Oracles/ModuleFiles.hs and into the top post of #210 (updated definition).

Using contextFiles you can find all source files corresponding to modules of a given Context. These source files may include Alex/Happy sources, e.g. compiler/parser/Lexer.x.

Using haskellSources you can find all Haskell files corresponding to modules of a given Context, some of which are generated and therefore live in _build directory. It looks like this is what you are after in sdist build rule, so you could do something like:

...
srcs <- haskellSources context
putBuild "| Build generated source files..."
need srcs -- here Hadrian builds all generated Haskell files for you
putBuild "| Copy source files..."
for_ srcs $ \src -> do
    copyFile src ... -- copy to a source distribution
...

Does this help? Let me know if you have any other questions.

[thomie]

@KaiHa: the problem with adding alex and happy as build dependencies is that it breaks existing build scripts (Linux distributions etc). This might cause more repair work and annoyances than it saves on Hadrian implementation.

I've learned that even small changes such as placing the user's guide in share/doc/ghc-<version> instead of share/doc/ghc can cause busywork for others (fpco/ghc-rc-stackage@e96839c#commitcomment-16196551).

[KaiHa]

@snowleopard can you have a look at 9495da2? It is a native implementation of sdist-ghc, but it has its flaws. I put some additional libraries into the tar-ball, because otherwise the build from the tar-ball fails (didn't investigate further).

Also I am not happy (no pun intended) with putting generated files into the tar-ball. The ghc ./aclocal.m4 script needs a patch if we leave the generated .hs files in the _build/ directory, otherwise ./configure will complain if alex or happy is missing. Furthermore I believe hadrian itself must learn how to deal with missing alex and happy.

[snowleopard]

@KaiHa I apologise for the long delay: I'm currently amidst business travel and can't find time for a proper review. In any case I suggest to raise a PR and we'll go from there. I should be able to respond with more substance during the weekend. Thank you!

[KaiHa]

Don't worry about the delay @snowleopard. What would be the best way to get the paths to the produced program binaries and libraries for usage in the install rule?

[snowleopard]

What would be the best way to get the paths to the produced program binaries and libraries for usage in the install rule?

@KaiHa For binaries, see programPath :: Context -> Maybe FilePath from src/GHC.hs. This function is a big ad-hoc mess and I'm not happy about it at all, but I think you should use it for now.

For libraries, see pkgLibraryFile :: Context -> Action FilePath from src/Settings/Paths.hs.

[snowleopard]

Thanks to @KaiHa we now have working source distribution rules! There are a few minor issues, but I think we can tick this box in the todo list (see the top of the issue).

[bgamari]

What is the status of bindist support? This would be great to have as it would allow us to move the new CI infrastructure over to Hadrian.

[snowleopard]

@izgzhen Is bindist rule on the to-do list of your Summer of Haskell project?

[izgzhen]

@izgzhen Is bindist rule on the to-do list of your Summer of Haskell project?

It is not, but I plan to take over it after summer. But feel free to do it yourself or assign to others :)

[snowleopard]

@izgzhen Thanks! I've unassigned you for now, as the installation rule is almost complete (except for #345). If noone signs up for it I'll try to take care of bindist in August.

[bgamari]

What is the status of bindist support?

[snowleopard]

@bgamari I didn't have time to look into this yet.

It would be more productive is someone familiar with what bindist should do gives this a try, so I've been waiting for a volunteer :) But if noone shows up I'll self-assign.

[bgamari]

I am looking at this. As far as I can tell there are two pieces to this (in the unix case at least),

1. Building the bindist tarball
2. Implementing the installation logic

In principle (1) is straightforward, being very similar to the source distribution case.

In the case of (2), however, there are a few possible directions to take,

1. Continue using the current build system for its make install rule
2. Build a shell script or Makefile specifically for installation
3. Build a Haskell installation executable

To me it seems like (3) is a bit more complexity than we need and will bloat binary distribution sizes unnecessarily. I suspect (2) is the right choice, but I'm curious to hear what your thoughts are, @snowleopard .

[snowleopard]

@bgamari I agree. We can rely on option (1) while Make is still in the tree, and (2) will be a good long-term solution. We don't want to build Hadrian or any other executable just to make a binary installation, a script should be sufficient.

[bgamari]

Ideally we would be able to reuse Hadrian's install logic while implementing the bindist logic. It would be a shame to have to duplicate this logic. For instance, one might write down a small DSL for installation actions which could be interpreted by Hadrian directly to implement the current hadrian install rule, or interpreted to produce a bindist install script.

[chitrak7]

Hi @snowleopard @izgzhen.
I would like to help in completing this task. Can someone please give me a headstart on where to begin. Thanks.

[alpmestan]

Regarding bindists: I wrote a preliminary binary distribution rule in #445 that I will soon adapt and submit in a pull request, once #531 is merged.

[snowleopard]

@chitrak7 Thank you! It looks like there is already some progress with this. I suggest we come back to this issue when #531 and #445 are merged.

[izgzhen]

@chitrak7 also, it looks like the install rules is temporarily disabled in #531, so it would be nice if you can talk with @alpmestan about fixing that part as well :)

[angerman]

I don’t think we want to actually have “install” in hadrian. If you ship the binary distribution you want something portable that doesn’t require to build hadrian (or ship a fully static hadrian binary).

The linked PR puts everything in stage1/{bin,lib} and building the binary distribution ends up being a simple zip/tar archive of those two folders. By being relocatable for the major platforms you can just unpack it and move it where you like. For convenience a short configure script is supplied, such that configure —prefix and make install just work.

[izgzhen]

@angerman I see, sounds good!

[alpmestan]

@chitrak7 We can talk (IRC, email?) about what comes after #531 (the binary distribution rule, wrapper generation, etc). What I have is quite simple and doesn't cover all cases, as I take advantage of the relocatability that comes with #531.

Maybe I should put together a patch on top of #531 that adds the binary distribution generation logic, and we can see where you can take it from there (in particular, you might want to look into generating wrapper scripts only when they're required, as well as maybe embedding some more content in the bindists -- I currently only pack bin, lib, docs and a couple of files necessary for running make install on the system where you want to install the bindist, IIRC).

[chitrak7]

Hi @snowleopard @alpmestan @izgzhen
PR #531 currently has implemented an elementary version of binary distribution. The current version ships contents of stage1 compilers “bin” and “libs” and some other files sufficient for installation. Although this is sufficient for ghc to work, few files may be missing. If I am not wrong, the future work in this issue can be summarised as:

1. Current “bindist” rule needs “phony” rules. We have to instead use the original files and executables that we have to ship.
2. Add an additional rule to figure out the installation directory on its own depending on the platform on which it is being installed.
3. Compare with the binary distribution supplied by ghc to check for files/executables that are still not supplied.
4. Currently, most of platforms do not need wrapper scripts anymore. I will implement rules to figure out the need of generating wrapper files or not depending upon the platform.
5. Add more features to the makefile, to match those which are currently supplied with the make-based ghc binary distribution.

[alpmestan]

Just a small comment: the implementation is not in #531 but will come in a separate PR. For now it's just in a branch (based on #531): alpmestan/hadrian@bye-ghc-cabal...alpmestan:alp/bindist

[alpmestan]

Since #558 we have a binary distribution rule.

[snowleopard]

@alpmestan Yes, I think we should close this issue as soon as we fix Hadrian and test the binary distribution rule on all platforms.

[alpmestan]

Yes, I didn't mean to comment to say "let's close this", just a status update. I also think it would perhaps make sense to leave it open until the bindist rule and the accompanying Makefile become a little more feature complete (compared to the current bindist rules in the make build system).

[chitrak7]

@alpmestan @snowleopard Can we close this issue now? Or is there still some work required?

[alpmestan]

@chitrak7 Do we want to be a little smarter in the bindist code as well, like we're planning to do in the testsuite? And just need whatever ways and packages and what not are dictated by the --flavour that we pass to hadrian, instead of assuming just vanilla (like we do now, in Rules.BinaryDist.pkgTarget, for libraries) or any other arbitrary choice like that?

Maybe @bgamari has an opinion on this, as he's our main user for the bindist rule I think. :-)

[chitrak7]

@alpmestan I think binary distributions do not allow this level of customization. Having said that, I just checked with ghc-8.4.2 binary and saw that we need dynamic libraries too, as well as more rts ways. So maybe we build the binary distribution adding all these ways. We can make a bindist flavour and build that, but need not provide customization of flags.

[alpmestan]

Yes we need vanilla, profiling and dynamic at the very least I think.

[chitrak7]

I think we need to solve #626 before we can address this.

[snowleopard]

We now have Rules.Bindist, so I think we can close this issue and open more specific ones if needed.