Depend on Hlint as a library

[fendor]

This problem is discussed in HIE #1143.

If you depend on hlint as a library, it demands that the location of its data-files are known at compile-time.
This is fine, if you are using binary packages of hlint or build it with cabal/stack from source.
However, if you depend on hlint as a library, and want to distribute a static executable of your application, hlint will usually fail, because the location of its data-files is known at compile-time and baked into the application. Thus, it might happen that the data-files are not present on the installation system at the expected location.

To solve this problem, the following approaches can be chosen:

• The application is responsible for distributing the data-files and configuring hlint correctly to find it.
  • Problem: How to configure the paths correctly?
• Patch hlint to provide the data-files via source code, e.g. translate the hlint.yaml into a haskell source record and drop the data-files required for hlint. For customisation, a hlint.yaml can be written to $XDG_CONFIG_HOME on startup.
  • Problem: Might bloat the executable?
• Applications should not depend on hlint as a library and create a run-time dependency for it.
  • Problem: Opens applications up for version conflicts.

What is your view on this issue? Which way would you suggest, and is there another, possibly better, way to address this issue?

[ndmitchell]

My current mental model is the first one. I believe HLint already looks at several places for the data files, and there is a flag --datadir which let's you specify where to find data files. There are also places in the API that let you specify where files are.

I'm not such a fan of anything TH related as my experience is TH makes things deeply unpleasant - failures on users systems, slow compile, version changes etc.

Does that work for you?

[fendor]

We can make it work, it is a bit complicated though, since we have to distribute the data-files somehow.
I was thinking about using file-embed, to embed the data-files at compile-time and write them into some location and configure hlint accordingly. But this requires TH and, afaik, increases compile-time even more.
Another way, we can transform the data-files into default values ourselves in the project and write them to disk when we need them and configure hlint accordingly. This might, however, introduce a new hazard to keep hlint configurations up to date with upstream.
Which one seems to be the more sensible way?

To patch hlint, I wouldn't have thought about using TH, but rather replacing the data-files needed at run-time with default values in hlint library.

[ndmitchell]

The data values are quite large, so that would be a little unpleasant. file-embed does seem to be the cleanest approach... Where is the difficulty in just distributing the HLint data files directly? (I'm sure there is, I'd just like to understand it)

[fendor]

Maybe I got something wrong: When you use data-files and use cabal to find them at run-time, the data-file location is compiled into the executable, e.g. ~/.cabal/store/package/data/.... In the case of stack, it might also be <project>/.stack-work/.... I think it can be different locations as well, but dont know how to specify that. If you delete the data folder, or move the executable to a different system, then these libraries are no longer present and the program fails. Moreover, it exposes information about the build system.

[ndmitchell]

@fendor - somewhat. You can:

• Use an environment variable hlint_datadir to override the data directory at runtime. That's available on all programs.,
• You can pass --datadir=... to HLint and it will use that location instead. That's a feature I built into HLint to make it easier.
• HLint will also look in . if it doesn't find good stuff in datadir.

[fendor]

Wasn't aware of the first option, that can be very helpful, thank you!
This is a good start, but for a fully functioning binary distribution still a hassle. It requires us to redistribute the data-files ourselves, which is, of course, a valid solution. This is the reason I created this issue, to learn the possibilities.
However, it still has the problem of exposing parts of the build environment, e.g. compile-time file location, if I see that correctly?
The last option is not really an alternative, since the defaults are not known then, right? Moreover, we have found that if the default files are not present, hlint throws an IOException. Is that correct or did we do something wrong?

[ndmitchell]

The behaviour when HLint can't find its files is unspecified. If you want a particular behaviour I can change it, but my guess is it's not super useful regardless of what it does.

For HIE don't you typically compile it on a users computer? If so, isn't HLint not a problem? Or are you trying to change that?

[fendor]

We are trying to change that since the compile-times are quite long. Thus, we want to be able to distribute binaries, see #1068.
Hlint was identified of having run-time dependencies (data-files) that we want to remove. You are right, it is not a problem if hie has been built from source.

[ndmitchell]

Makes sense. I don't have a good way to solve this. If HLint had a command to copy its data files to some location, could you use that when compiling to generate a redistributable package that worked nicely?

[fendor]

I am not sure.
What would that imply?
Would it be required to install hlint as an executable on the user system? This is actually another totally valid solution, turning hlint into a full-fledged run-time dependency for hie.
I am also not sure about having a separate hlint.yaml for a potential global hlint and the hlint that is used by hie.

[ndmitchell]

I meant during HIE building you build HLint in as a library. Run hlint --gather-deps=hie_install_dir, and then pass --data-dir=hie_install_dir when running on a users machine (or the equivalent in API form).

[fendor]

Yes, this would work, but I think this is already possible, right? With more template haskell, though.
Still, is it acceptable for the user to have two hlint.yaml's that influnece their lints, in your opinion?

[ndmitchell]

I think HLint uses the first hlint.yaml it finds, and doesn't worry about the others, so that should be fine.

[samuelpilz]

@ndmitchell I will try to add hlint --gather-deps=hie_install_dir to the packaging-process and let hie set the data-dir flag correctly.

[ndmitchell]

Note there is no --gather-deps at the moment. There could be though (PR welcome). Or maybe you'd just want a mode that prints out the data dir and exits, letting you do a recursive copy.

[mpickering]

This option seems a bit unnecessary. You can get the data-files directly by unpacking the release tarball. (cabal get hlint)

[samuelpilz]

I will propably solve this using cabal get and --datadir=. Feel free to close.

[ndmitchell]

Thanks, let me know if it doesn't work nicely.

[Avi-D-coder]

@ndmitchell would you reconsider the TH approach?

It's not just nicer for using hlint as a library it's nicer when using the executable too. I'm on a 256G laptop and I delete ~/.stack when ever it gets past 10G. I install hlint using stack install, so when I delete ~/.stack hlint has to be rebuilt, just to get 3 data files.

Here's a commit using file-embed. The TH is relegated to one file. The executable actually gets slightly smaller 48117184 to 48020968 (GHC 8.8).