How to distribute to Linux and OSX?

[heetbeet]

Maybe we should initially bypass the Linux bureaucracy by creating an sh installer similar to Anaconda.

Here is the Anaconda installer header from curl -r 0-26557 https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh:
Anaconda3-2021.05-Linux-x86_64.sh.zip

Here is an initial attempt into making it into something a bit more generic:
https://github.com/heetbeet/BashTemplateInstaller/blob/main/bash_installer.sh

This header seems to work for both Linux and OSX. I'm not sure though how the copyright will works, if it's a problem, it's easy enough to write something from scratch.

[davidanthoff]

Yeah, I like that approach! Rust also does that, so we might also be able to copy from them. And with the rewrite in Rust we will have just two very small binaries and that should actually be enough to get started.

[davidanthoff]

So I think where we are landing now is this: juliaup itself can be distributed either a) by a third party deployment tool or b) standalone.

The currently implemented third party option is the Windows Store, but down the road we should also try to get it into homebrew for Mac, and maybe eventually even the various native Linux distribution options. I'd still like sudo apt install julia to just work at some point :) Other options in this vein might also be conda or pip or anything else. What all of these options have in common is that if you get juliaup in that way, those third party distribution options are in charge of keeping juliaup itself updated, i.e. none of the self commands described below will work. Same for uninstall, if you got juliaup via a third part installer, we'll disable our native uninstall functionality.

The second option is standalone. In that case the user starts out with the kind of shell command @heetbeet mentioned above. The setup flow in that case would be this:

1. user executes the kind of command @heetbeet mentioned above.
2. the shell script downloads the juliaup binary and drops it into ~/.julia/juliaup/bin. Probably should take JULIA_DEPOT into account, though?
3. It runs juliaup self install, i.e. at that point it hands all control over to juliaup and the rest of the logic is written in Rust.
4. juliaup self install does then continue with the following steps:
5. Downloads the julialauncher binary and drops it next to juliaup into ~/.julia/juliaup/bin.
6. Gets both juliaup and julialauncher onto the PATH. My preferred solution on Linux/Mac would be that it creates symlinks to these two binaries in some known folder that is already on the PATH, i.e. that it does not modify PATH at all. One benefit of this is that we can keep julialauncher named julialauncher, and then just name the symlink julia. So for example we might create two symlinks like /usr/local/bin/julia -> ~/.julia/juliaup/bin/julialauncher and /usr/local/bin/juliaup -> ~/.julia/juliaup/bin/juliaup (not sure whether those are the right locations for the symlinks, though). If for some reason the symlink option doesn't work, then we would have to add ~./julia/juliaup/bin to the PATH in this step.
7. It runs the code for initial setup that essentially prepares the release channel.

One question is whether step 5 should really be part of juliaup, or do we move that part into the shell script? Not clear to me what is easier/more robust... We could go crazy and embed julialauncher as a binary blob in juliaup...

For the standalone scenario we would then also have commands juliaup self update and juliaup self uninstall.

[simeonschaub]

I have created a juliaup build script for arch here in case anyone is interested. For now, that will need to be updated manually, but I think it should also be possible to do this automatically as part of CI.

[davidanthoff]

Oh, and one other major question: how do we "brand" ourselves in third party stores? In the Windows Store we are just Julia, not juliaup. I think that if we decide that this will become the official Julia setup story, then we should try to register everywhere as just julia. After all, this package does put julia on the PATH, and it just also ships a helper binary that is called juliaup.

We might need a transition phase, though, where we register as juliaup until everything has gotten more stable.

[KristofferC]

Couldn't juliaup just update itself?

[davidanthoff]

Couldn't juliaup just update itself?

So in the case of a standalone install, this would do that. Or were you thinking auto-update itself? But when distributed via a third-party mechanism, then I'm not so sure whether we want it to self-update, right? Certainly for the Windows Store that won't work (because it doesn't allow us to change the files it distributes), and presumably other things like brew, conda etc would also not like that?

[KristofferC]

But when distributed via a third-party mechanism, then I'm not so sure whether we want it to self-update, right?

I am not sure. I think with pip I still do pip --upgrade but I am pretty sure I never installed pip "manually". This is what rustup does: https://rust-lang.github.io/rustup/basics.html#keeping-rustup-up-to-date. So they allow a different configuration of rustup where self update can be disabled by certain vendors.

[heetbeet]

I've been playing around to see how a sh installer (à la Anaconda) would work out: https://github.com/heetbeet/proof-of-concept-juliaup-installer/releases/tag/0.0.1

This is still a work-in-progress, but If we want to go this route for user-level installers (i.e. doesn't require sudo), it wouldn't need much more work to get something stable going. The installer can scale all the way from

1. No-payloads - download all required files from the internet
2. Only juliaup payload - download Julia from the internet via juliaup usual mechanism
3. Self-contained - installs juliaup with BundledJulia on computers without internet

It can be made to work on OSX and Linux.

Here is an example run

Let me know if this is something I should continue to work on or if an alternative is preferred to this.

[davidanthoff]

Fantastic!

I think generally we probably don't need the BundledJulia case for this. The only reason I'm doing that with the Windows Store version is that we can't run any setup script when the user installs Julia from the store. The first time we have a hook to run anything is when the user manually starts one of the binaries. So there is then a scenario that a user installs Julia from the store, then goes offline, and then starts Julia for the first time. I want that to work, and we can only pull that off with the bundled Julia version. But I think for your installer script that is not an issue at all: by definition that will run at install time when the user is online, and so we can guarantee that at some point juliaup self install gets called and that can then just use the built-in mechanism to download an initial version of Julia and install it. The only scenario I can think of where a bundled Julia might still be good is something where someone tries to install on a machine that is not connected to the internet at all, I guess that would actually work with option 3, right? While most juliaup features would obviously not work in that case, that might still be interesting, to enable things like julia-vscode/julia-vscode#2374... But not sure.

I think one benefit of option 1 would be that we could later easier add support for things like the "release preview" channel for juliaup itself. For example I could imagine an env variable (or command line argument) to the shell script that installs the release preview version of juliaup, rather than the release version. So I think I'm probably tending towards option 1 right now, but maybe we could keep the other options around if we change our mind later?

I think Mac and Linux is perfectly fine. rustup also has a completely separate solution for Windows (namely a small .exe that does the same thing). Plus we have the store version on Windows for now.

I see that the shell script right now has the logic to create the symlinks. I'm wondering whether it wouldn't be better to put that into juliaup itself? If we had a new command juliaup self install that 1) created these symlinks and 2) added the release channel (and thereby downloaded a Julia version) and then ran juliaup self install at the end of the script, that should work, right? The benefit of that would be that folks could also just download juliaup manually, put it in some folder and then call juliaup self install and it would also work.

Oh, and the reason it is not picking up the BundledJulia stuff is that juliaup and that folder need to be in the same folder at the moment, but I think in your example juliaup is in bin, right?

[heetbeet]

After some thought, I think you are correct - it's best to stick with (1.) and possibly never support an offline wizard/q&a installer, or do it only when all the other installation features are done. My own thinking

• It'll bypass quite a bit of support complexity for the user (one script to rule them all)
• If a company/someone requires an offline installer, they can download the juliaup binaries directly and run your suggested juliaup self install

I think I'm going to fork the rustup-init.sh installer and work from there. It seems like there is quite a bit of tribal knowledge integrated into that installer. Before we set up a proper release channel, I'll use https://api.github.com/repos/JuliaLang/juliaup/releases/latest and https://github.com/JuliaLang/juliaup/releases/latest and try to match the "target triple" via something like https://gist.github.com/steinwaywhw/a4cd19cda655b8249d908261a62687f8#gistcomment-3705619
I'll pass the rest of the logic to juliaup self install to do the heavy lifting from there. This will require:

Symlinks: Yes lets make it juliaup's problem. This is for doing a symlink for julialauncher -> julia in the installed bin directory. This will allow the user to add this directory to the PATH (in cases where symlinking to a global directory is not possible). Some users might also want to keep juliaup portable, then at least they can access julia in that directory. One thing to look our for is that Windows users rarely have symlink privileges without admin rights. Juliaup should opt for a copy when unsuccessful symlinking ensues. Juliaup should then also remember on updating etc. to replace the copy...

Add to PATH: I think this is also something Juliaup can help the user with. On linux it can ask the user if they want to symlink juliaup and julialauncher to /usr/bin with a warning that it will prompt for sudo access, or if they want to add to bashrc. Most software skips the former and just focus on the latter since it is the common denominator for most platforms. (EDIT: and bashrc only adds it for that user under that user's sh session, and not for other users sharing the computer, which might not have access to the installation location of that user)

BundledJulia: Let's then rather not open this can of worms... and let juliaup always download a brand new julia online.

What I can action on:

• Fork the repo containing rustup-init.sh
• Rename Rustup-specific stuff to Juliaup-specific
• Bypass Rustup download URL logic and reroute to use https://github.com/JuliaLang/juliaup/releases/latest

What Juliaup can action on:

• Implement juliaup self install

[heetbeet]

To which location do we want to install juliaup? Prior art: rustup installs to $HOME/.cargo, Anaconda installs to $HOME/anaconda3

[KristofferC]

Perhaps .julia/juliaup?

Edit: Oh, I see that's where the executables are put.

[heetbeet]

@KristofferC I also thought about a location within ~/.julia, but the problem is that Juliaup superposes the DEPOT_PATH. Juliaup can be seen as a seperate piece of software that evaluates where the current DEPOT_PATH is and then do all it's dirty operations within that DEPOT_PATH. I.e. if you change the shell variable $JULIA_DEPOT_PATH, then juliaup will start it's dirty work afresh within that new DEPOT_PATH. So if we add the Juliaup binaries to ~/.julia location, it might be misleading and confusing.

But, with that said, it might not be a strong enough reason not to just add it to the currently evaluated DEPOT_PATH (like maybe DEPOT_PATH/juliaupbin or the likes), since most default systems will only ever use one DEPOT_PATH and it will likely be ~/.julia. And if the DEPOT_PATH moves, nothing will break, the location of the binaries might then just seem a tad awkward.

[KristofferC]

Juliaup can be seen as a seperate piece of software that evaluates where the current DEPOT_PATH is and then do all it's dirty operations within that DEPOT_PATH.

That makes sense. In that case, it is maybe best to have it as its own thing "above" .julia.

[davidanthoff]

I think the most consistent option would probably be ~/.juliaup, right?

There is another benefit to that: for scenarios where juliaup is distributed by a platform deployment tool (like the Windows Store, or maybe homebrew down the road etc.), the juliaup and julialauncherbinaries would also not be inside.julia, so in some way we would align these two scenarios a bit: for platform deployment tools, juliaupandjulialauncherend up somewhere in the system that is platform dependent, for our shell script installer they end up in~/.juliaup, but the layout and structure of .julia` is the same for both scenarios.

[davidanthoff]

I think I'm going to fork the rustup-init.sh installer and work from there. It seems like there is quite a bit of tribal knowledge integrated into that installer.

Yes, I got the same impression. It seemed to cover corner cases that I certainly would have never thought of :)

This is for doing a symlink for julialauncher -> julia in the installed bin directory. This will allow the user to add this directory to the PATH (in cases where symlinking to a global directory is not possible). Some users might also want to keep juliaup portable, then at least they can access julia in that directory. One thing to look our for is that Windows users rarely have symlink privileges without admin rights. Juliaup should opt for a copy when unsuccessful symlinking ensues. Juliaup should then also remember on updating etc. to replace the copy...

Ah, I didn't think about this case... The idea with the julia symlink next to the julialauncher is nice, because then the process would still show up as julialauncher in a list of processes. If this ends up a problem on Windows, we could also just drop a julia.bat in the bin folder, right? Or as you said just copy.

to symlink juliaup and julialauncher to /usr/bin

Is there a similar user specific location on Linux like /usr/bin?

[heetbeet]

I'm making really good headway on this. The juliaup version of rustup can already download os-specific binaries and extract it to ~/.juliaup (hardcoded for now) and then run ~/.juliaup/bin/julliaup self install (which then obviously crashes because it's not implemented yet):

• changes to original
• script for trying out

Re: julia.bat: Batch scripts screw around a bit with the flow of a program, if you press Ctrl-C it asks for "Terminate batch job (Y/N)”, which is really annoying. (I have a few .bat heavy workflows)

Re: /usr/bin: I don't think there is a user-level directory like /usr/bin, unfortunately.

[heetbeet]

Added a windows version to the scripts

• Linux / OSX / Cygwin / MinGW: juliaup-init.sh
• Windows : juliaup-init.cmd

We can then do a similar one-liner installation as rustup does

Or the user can download the script and run it natively.

Proof of concept:

Windows:

Powershell -NoP -C iex (New-Object System.Net.WebClient).DownloadString('https://github.com/heetbeet/rustup/raw/master/juliaup-init.cmd')

Linux / OSX / Cygwin / MinGW:

curl --proto '=https' --tlsv1.2 -sSf -L https://github.com/heetbeet/rustup/raw/master/juliaup-init.sh | sh

When the script is run with args, I forward those args to juliaup self install [args] (except for --help/-h -- and I want to implement --path/-p on the installer's side still.)

We should set up some kind of contract between the installer and juliaup in order for the installer to construct a correct help menu, and for juliaup self install ... to ignore script-specific args like --path/-p, since it's quite difficult to remove them from bash before passing them on to juliaup self install ....

[davidanthoff]

curl --proto '=https' --tlsv1.2 -sSf https://julialang-s3.julialang.org/juliaup/juliaup-init-releasepreview.sh | sh

now works. Still very experimental, but hey :) And then juliaup selfupdate should keep juliaup updated, and that should also work.

@heetbeet do we actually need the --proto and --tlsv1.2 arguments? Things seem to work for me without them?

[heetbeet]

It might not be necessary, I copied it from rust. It might be there to cater for some corner case, but I have no idea what that would be

[davidanthoff]

I think I'll close this issue. In principle it is clear now how that would work and that is implemented. There is still a lot of small things to fix, but we can track that in new issues.