Add CLI

[FFY00]

Closes #52

Signed-off-by: Filipe Laíns [email protected]

[FFY00]

@pradyunsg how do you want to implement the bytecode generation? I was thinking a custom destination, or making installer.install return the written records. Let me know how you want to proceed.

[FFY00]

I went with returning the records from installer.install. Please let me know if it looks good to you,

[FFY00]

@pradyunsg I think I have a pretty complete draft, let me know if the implementation looks good to you so that I can start working on the tests.

[FFY00]

@pradyunsg friendly ping.

[flying-sheep]

Ah cool! great to see someone working on it! I hope we have this soon, it fills such a big gap!

[takluyver]

This looks like a good start to me. I've left some comments inline.

Should there also be a --prefix option or similar, to install a package to a different location from the default?

[takluyver]

Should there be an option to skip this check as well? Or use the existing skip-dependency-check option to skip this as well?

I'm pretty sure there are times when I've bumped Requires-Python to reflect what I test and care about, even though there's a good chance code will still run on an older Python version.

[FFY00]

Sure.

[eli-schwartz]

I don't have strong opinions one way or the other about a python version check, but FWIW when I originally read this code I assumed the dependency check had a skip option because it was assumed people might often be installing a wheel when its dependencies are not currently installed, but will be by the time it is expected to be used.

In contrast, skipping the python version check is saying that you think the wheel metadata is flat out wrong. Which I guess you're saying it might sometimes be...

[takluyver]

Yup. If you want a concrete example, see h5py/h5py#1727 - CI is slow, so we had an incentive to stop testing on Python 3.6. When we did, we could either declare that it requires Python 3.7 (which would be wrong at least for a while), or declare that it still supports 3.6 - until someone one day realises that that's wrong and files an issue.

Being overly restrictive is generally easier for maintainers - someone who does a simply pip install h5py on Python 3.6 will automatically get an older version that was tested on 3.6. And if someone really needs a newer h5py, they can bypass the check and see if it works.

[jameshilliard]

In contrast, skipping the python version check is saying that you think the wheel metadata is flat out wrong. Which I guess you're saying it might sometimes be...

The main thing here I think is that version checks need to be done against the target interpreter sysconfig(the one that will be used with the package being installed) and not the one actually executing installer and doing the installation(which can be entirely different for example when cross compiling)

[FFY00]

Should there also be a --prefix option or similar, to install a package to a different location from the default?

How do you know that would be the correct path for the interpreter you are targeting?

[takluyver]

It would be up to the user to provide a useful prefix. As I understand it, this is meant to be a low-level tool for expert users such as downstream packagers, so it doesn't need to be totally foolproof.

I don't know if the option would be necessary, though. Of downstream packaging systems I've got some familiarity with:

• Linux distros may have a split between things they package in /usr and things installed separately in /usr/local - I don't know quite how this is controlled, but a prefix option would be one possible way.
• Spack appears to use --prefix when packaging, but it also specifies individual target directories, so maybe it won't use this CLI anyway.
• Conda doesn't appear to need it

[FFY00]

IMO, overwriting the prefix is in most cases a very bad way to handle things. The only reason I can think of where one might want to overwrite the prefix is if you are targeting a different interpreter altogether, but doing that this way is very flimsy, prone to breaking, and limited in situations it might actually work (often you'll need to overwrite more than just the prefix). Even if you are an expert, I would not recommend this tool for that. For that scenario, I would recommend writing a new tool using the API this project provides that allows to properly customize the install locations -- either overwrite or allow to evaluate the schemes with all supported config options, not just the prefix.

In most cases the best way to handle customizations in the install locations is by adding new schemes to sysconfig. This PR does not implement it yet, but I would like to have an option to select the scheme for which you want to install.

Linux distros may have a split between things they package in /usr and things installed separately in /usr/local - I don't know quite how this is controlled, but a prefix option would be one possible way.

This should be done via a sysconfig prefix IMO.

Spack appears to use --prefix when packaging, but it also specifies individual target directories, so maybe it won't use this CLI anyway.

Which demonstrates the limitations of overwriting the prefix, and would be require a different tool like the one I described above. But if spack is the one shipping the Python interpreter, I would recommend adding a sysconfig scheme instead 😅

Conda doesn't appear to need it

Conda needs much more control over the environment, not just the locations supported by wheels/sysconfig, so it brings its own logic.

[takluyver]

Fair enough. Installing to a specified prefix is a familiar pattern on Unix-y platforms, though, both for Python packages and more generally. So I'd hope we could at least offer some kind of explanation about "if you're looking for a prefix option, here's what to do instead". 🙂

Conda needs much more control over the environment, not just the locations supported by wheels/sysconfig, so it brings its own logic.

Conda packages of Python code are often made using pip install (example), and I imagine they may well use this CLI in the future. Conda's own logic comes in once the desired package is installed in the temporary environment.

[eli-schwartz]

I think there is a big difference between:

• destdir support, which is about installing to the current interpreter's defined install scheme but actually placing the physical files in a staging directory which is usually then added to a distro packaging manifest, turned into an archive, and then later merged directly into /,
• prefix, which is actually meant to modify how the install scheme works

IMHO prefix should not be supported, unless the CLI also intends to somehow offer support for installing into a python interpreter that is not sys.executable. In this case, I'd expect, say, a --python option which then runs the target python to dump sysconfig values and install into those instead.

[eli-schwartz]

Alternatively, to cover the /usr vs /usr/local case, you'd need a python interpreter patched to properly support this (in other words, it actually adds that directory to sys.path and allows importing the code you just installed), e.g. the debian package for python, which should probably be exposed in the CLI as a --scheme option to select which install scheme you want to utilize. This would also allow installing in --user mode by diverting from posix_prefix to posix_user.

[takluyver]

Fair enough. So the "what to do instead" would be to run installer from the target Python so it picks up that prefix, IIUC.

[flying-sheep]

Linux distros may have a split between things they package in /usr and things installed separately in /usr/local - I don't know quite how this is controlled, but a prefix option would be one possible way.

If you’re interested, this is my current understanding:

1. /usr/local/{bin,lib,…} is allowed to be modified by any tool (e.g. by an administrator running make install)
2. /usr/{bin,lib,…} is sole domain of the distro package manager. Only the contents of a .deb, .rpm, .pkg.tar.*, … may end up there
3. $some_prefix/usr/{bin,lib,…} is how packaging tools usually prepare the files going into such a .deb, .rpm, .pkg.tar.*, …

I don’t think a simple path prefix (=destdir) is the right abstraction for /usr vs /usr/local, as e.g. it’s /etc and /usr/local/etc I think, but a destdir (or chroot) is necessary for 3.

[FFY00]

That is not exactly accurate, /usr/local may be not be used by programs under /usr, so really it is its own separate prefix and should have no influence here.

[FFY00]

@pradyunsg can you have a look at this? If everything looks alright to you, I will start working on the tests.

[takluyver]

This might be controversial, but I'd actually favour not installing a script, and having the CLI only available as python -m installer.

This avoids some possible forms of confusion - e.g. if you activate a virtualenv where installer is not installed, but python-installer is on PATH from some other location. And it simplifies bootstrapping, because a wheel for a pure-Python, library-only project can be unzipped into site-packages to install it, whereas creating scripts is more complex. Given that we're suggesting installer as the answer to that for downstream packagers, it would be nice to make the installation of installer itself easy.

[FFY00]

I concur, and happy to remove that from the PR.

And it simplifies bootstrapping, because a wheel for a pure-Python, library-only project can be unzipped into site-packages to install it, whereas creating scripts is more complex. Given that we're suggesting installer as the answer to that for downstream packagers, it would be nice to make the installation of installer itself easy.

I don't think removing the entrypoint changes much in relation to bootstrapping, as most people that need this tool still need to build the wheel from source. And if you have access to the source, you could already just run it from source to install itself instead of unpacking, which is much much more quirky.

[takluyver]

That's true. Though I'm guessing downstream packagers would implement the 'unzip a wheel' bit for bootstrapping flit_core and tomli before installer, in any case. The alternative would be to have all 3 of those pieces importable from source together to bootstrap them. But I'm just going on guesswork.

[FFY00]

FWIW, in https://github.com/FFY00/python-bootstrap I am just running everything from source. Well, kinda, setuptools needs some special handling 🙃

[pradyunsg]

Thanks for filing this, and for your patience here! I've had a bunch of review comments written down for months, but life stuff kept preventing me from having time to sit down and writing this blurb.

There's a few aspects of this that I'm wary of, which can be summarized down to a difference between what various folks think this package is.

• IMO it is providing one very narrowly defined thing: logic to properly unpack wheels into the specified directories with all the right launchers.
• ISTM folks expect this to be a drop-in replacement for pip install {options} {wheel} with other sorts of useful functionality.

This has various implications relevant to this PR:

• I don't want to get into the business of figuring out what the "right" directories are for unpacking a wheel in this package.

  Various redistributors do things differently, and I don't want to be involved in handling those differences here, beyond having them specify the right paths to this package. To paint a fuller picture on this -- pip now uses sysconfig on 3.10+ and distutils for <3.9. However, there's an opt-in environment variable to adopt sysconfig. There are multiple kinds of patches flying around, on pip and Python. There's also a private only-for-redistributors mechanisms to change whether pip uses sysconfig or disutils under consideration, that I certainly don't want to be replicating all of that work here. These are the sorts of details that redistributors need to understand and make decisions about on/at their end, and they are in the best position to specify the relevant paths for each Python installation they provide.

• I do not want tie the usage to the currently running version of Python.

  While pip can only install packages into the running interpreter, this package does not have the same limitations in the way it is implemented. There's no reason that someone can't install a wheel into Python 2.7 or 3.3 or whatever, while running this library on Python 3.10. I don't think the CLI for this package should imply that either.

• I'm not sure I would want this package to handle checking extra things.

  Functionality like checking dependencies or the python_version are inherently tied to the running interpreter. Other than the what-is-the-scope aspects of this, this is also at odds with the note above, about not being tied to the running interpreter. Some of this stuff sounds like the sort of thing that can (and should) be handled at a different level, i.e. a higher level tool. This package isn't intended to be used by "end users", i.e. a regular user of Python software -- things like dependency management and ensuring that the installs are happening properly are a concern for higher level tooling, that's built around or on top of this package.

Because this is fairly long already... there's broadly two directions here:

• Make a case that this project should be broader than "unpack this wheel correctly" and why that is a better, more sustainable and more maintainable position for the overall ecology here.
• Rework this PR to fit into the smaller scope of "unpack this wheel correctly" (or file a separate one).

[pradyunsg]

Move these into an cli extra -- that's the mechanism for declaring optional dependencies. I don’t think all users need these to be installed.

[pradyunsg]

And... assuming that we're reducing scope, we don't need either dependency.

[pradyunsg]

I would strongly prefer to only provide python -m installer or even just a scripts/install_wheel.py in the sdist.

Both of them:

• are unambigous about the Python executable being used.
• do not result in conflicting executables given that these are unqualified.
• do not require generating a script wrapper during installation of this package.

[jameshilliard]

Yeah, it would also be good to have cli flags for settings stuff like the installation --root similar to setuptools, for example our setuptools target build/installs have to set a bunch of stuff so that things are built against the right interpreter and installed in the right place:

# Target setuptools-based packages
PKG_PYTHON_SETUPTOOLS_ENV = \
	_PYTHON_SYSCONFIGDATA_NAME="$(PKG_PYTHON_SYSCONFIGDATA_NAME)" \
	PATH=$(BR_PATH) \
	$(TARGET_CONFIGURE_OPTS) \
	PYTHONPATH="$(if $(BR2_PACKAGE_PYTHON3),$(PYTHON3_PATH),$(PYTHON_PATH))" \
	PYTHONNOUSERSITE=1 \
	_python_sysroot=$(STAGING_DIR) \
	_python_prefix=/usr \
	_python_exec_prefix=/usr

PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS = \
	--prefix=/usr \
	--executable=/usr/bin/python \
	--single-version-externally-managed \
	--root=$(TARGET_DIR)

[pradyunsg]

Please see #58.

[pradyunsg]

…

If you really think pycompile is such an important feature, then I’d rather that this be implemented in the SchemeDictionaryDestination object directly. I’m not convinced that it is, but I’m open to having my mind changed on this.

[layday]

Shouldn't the bytecode files be added to the record? I was under the impression pip did this too to have a "complete" record upon installation. In addition, some package managers (i.e. Nix) do not allow modifications to installed files and directories so if we don't compile them here then they'll either have to do it themselves or start-up performance will suffer rather substantially. The complication is that bytecode files are Python versioned so ideally these should be generated by the target interpreter.

[pradyunsg]

See final bullet in https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl.

[layday]

Oh thanks, I missed that. I wonder why pip runs compileall then?

[pradyunsg]

This sort of stuff is honestly why I am wary of providing a CLI that automatically determines the right paths for you. :)

[jameshilliard]

ISTM folks expect this to be a drop-in replacement for pip install {options} {wheel} with other sorts of useful functionality.

I think it probably makes sense to go even further and have this allow for sdist/repo based build+install workflows rather than just wheel based workflows, now this could be done in installer itself by adding and vendoring a build dependency tree(I think) or by creating a new package that depends on installer+build and vendors the full dependency tree there, but distros really could use a single package that has similar semantics to the existing setuptools/distutils style staged build+install workflow. The important part is that this build tool would need to vendor a lot of stuff like pip does in order to not have circular dependencies as distro build systems often have limitations that prevent resolution of circular dependencies.

FWIW, in https://github.com/FFY00/python-bootstrap I am just running everything from source. Well, kinda, setuptools needs some special handling 🙃

Maybe it would make sense to have python-bootstrap vendor all of its dependencies(with full pip style namespace isolation) for bootstrapping purposes. Right now it's not a great option for distro maintainers since it would likely introduce namespace conflicts due to just using submodules for dependencies.

While pip can only install packages into the running interpreter, this package does not have the same limitations in the way it is implemented. There's no reason that someone can't install a wheel into Python 2.7 or 3.3 or whatever, while running this library on Python 3.10. I don't think the CLI for this package should imply that either.

Yep, this is very important for cross compilation where we have a host toolchain that has a python interpreter built for the host but needs to install for a target interpreter(which can not be run on the host)

Functionality like checking dependencies or the python_version are inherently tied to the running interpreter. Other than the what-is-the-scope aspects of this, this is also at odds with the note above, about not being tied to the running interpreter.

Well using sysconfig may work in some cases when targeting a different interpreter, for cross compilation we hack up the sysconfig for target package builds so that stuff like c extensions gets built against the target interpreter rather than the host interpreter.

[jameshilliard]

Maybe use sysconfig.get_python_version() instead since that should generally work better for cross compilation scenarios I think.

[FFY00]

Good point, but unfortunately sysconfig.get_python_version does not contain the minor version.

[jameshilliard]

Docs indicate it does:

Return the MAJOR.MINOR Python version number as a string.

[FFY00]

Sorry! I meant the patch version.

[FFY00]

Maybe it would make sense to have python-bootstrap vendor all of its dependencies(with full pip style namespace isolation) for bootstrapping purposes. Right now it's not a great option for distro maintainers since it would likely introduce namespace conflicts due to just using submodules for dependencies.

python-bootstrap is not a package, but rather a script. The idea is that it automates the bootstrap process, you just fetch the source (clone the git repo, or fetch a tarball with the submodules, etc.) and you can use it to build and install all tooling necessary to build and install Python packages, which is not available (that's what we are trying to build) 😅

[jameshilliard]

python-bootstrap is not a package, but rather a script.

Yeah, i realize that, I mean it would potentially make sense to make it a package, or make a meta package with namespace vendored dependencies.

[FFY00]

Yeah, i realize that, I mean it would potentially make sense to make it a package, or make a meta package with namespace vendored dependencies.

This is getting a bit off-topic, so it would be better to open an issue in its repo, but I don't really see why it would be beneficial to make it a package. Note, by "package" here I mean a Python installable and distributable package, not a package as in Python module at the interpreter level (it already is that, as it is a folder with a __init__.py files, not a single file module .py).

[takluyver]

@jameshilliard Pradyun wants to keep the scope of this package tightly focused, and it currently (in main) has no dependencies. So I'm pretty sure that suggesting that it should also handle building, acquire several extra dependencies, and then deal with vendoring them all, is not going to achieve anything besides us all spending more time writing comments to each other. If you want a package that combines the functionality of build & installer in a single place with no external dependencies, feel free to make that yourself. I think there's value in having a project that focuses purely on the 'install from wheel' step.

---

I'd say a CLI is useful even if it requires users to specify the same information as the Python API - all 6 destination directories (--purelib /path/to/abc --platlib /another/path ...), the Python executable path, and the script kind (though I'd be tempted to default to POSIX scripts, as I imagine there are far more people repackaging Python packages in Posix-y environments than Windows). I think a lot of packaging tooling is often based on shell scripts, or things like Makefiles that naturally use shell syntax, so having even a verbose command line interface is nicer than having to embed Python code.

It's tempting to say it should then extend to more traditional methods of describing install locations, i.e. --root and --prefix options. But if you really want to leave nothing implicit, nothing being guessed from the running Python, you'd need quite a few more inputs. Looking at sysconfig.py, a posix_prefix installation has variables base (i.e. prefix), platbase, installed_base, installed_platbase, platlibdir, py_version_short and abiflags. That's more inputs than passing in the 6 paths, although admittedly they're probably shorter. So I think @pradyunsg is probably right to say no to this.

I can imagine it might be useful to have a shortcut to install for the currently running Python, if this were 100% explicit, e.g. python -m installer.this_python foo-1.0-py3-none-any.whl. It shouldn't be a default which you have to override a dozen ways to do something different. But I'd want feedback from actual downstreams about whether that would be useful.

Maybe it's worth asking @hroncok. He mentioned on pypa/flit#483 that Fedora would like to move to using build & installer if possible, so he can probably say if a python -m installer style CLI would be useful to them, and if so how they would use it.

[flying-sheep]

Totally agree. I was also thinking that “do current python by default and allow a way to specify others” might be the sweet spot.

Future consideration: Hmm, there should be some kind of database that contains all those paths for known OSs and distros. Or a file format standard so people can build a database like this easily themselves and we could supply all options at once by pointing to a file like that.

It's tempting to say it should then extend to more traditional methods of describing install locations, i.e. --root and --prefix options.

I thought we already agreed that --destdir or so is a good idea because it simply prefixes every single path with a changed root and makes a system packager’s job easy? On Arch, doing python -m build -w and then python -m installer --destdir="$pkgdir" dist/*.whl would be the dream!

Agreed about other, more complex options, but this one is both trivial and useful.

[FFY00]

Future consideration: Hmm, there should be some kind of database that contains all those paths for known OSs and distros. Or a file format standard so people can build a database like this easily themselves and we could supply all options at once by pointing to a file like that.

These paths can be dynamic as of 3.10, so this is not possible.

[layday]

I've toyed with some of the ideas that have been thrown around here at pyproject-install.

Rewiring the installation prefix feels like a hack and sysconfig provides no guarantee that its schemes will be located under the same prefix or that they can be parametrised. The replacement strings are not standardised. I feel sysconfig works better for venvs which are self-contained by design. But hey, the posix_prefix scheme is sensible, and that's probably the only one anybody cares about when they talk about installing under a different prefix.

[takluyver]

I've had a go in #92 at an alternative CLI. It requires much more verbose input than this one, because it doesn't use any information from the Python which runs installer. I hope this can be a starting point, and we can then discuss what, if any, convenience shortcuts would be useful on top of that.

[jameshilliard]

It requires much more verbose input than this one, because it doesn't use any information from the Python which runs installer.

Yeah, this approach seems like a good starting point as we want to customize this anyways in general.

[jameshilliard]

For that scenario, I would recommend writing a new tool using the API this project provides that allows to properly customize the install locations -- either overwrite or allow to evaluate the schemes with all supported config options, not just the prefix.

FYI this can't really be done in a separate tool package without creating a dependency cycle as this hypothetical tool wouldn't be able to depend on installer as installer would need said tool to install itself.

[pradyunsg]

Closing this out, since #94 is merged now! :)