Add support for configuring Poetry via env vars (POETRY_*)

[claireramming]

I have several applications that use poetry and several of the packages in each app need to be installed via a private repository. Previously I was able to install these in my apps using a requirements.txt that installed poetry, then a post_compile script that set poetry config vars for http basic sign-in to our private repository before calling poetry install.

With the new buildpack updates that allow poetry to be used out of the box, I'm struggling to figure out how to correctly pass my private repository user/password since I can no longer call my config command directly before poetry install is called (I don't think pre_compile will work here since poetry needs to be installed first). According to poetry I could use a config.toml file, but since one of the vars is a pw I don't want to expose that. Another option via poetry is that I should be able to set ENV vars to replace the config command that I used to call before running poetry install (poetry doc here) but poetry in Heroku does not seem to be picking up on my POETRY_ env vars, even though from what I can tell, it should be passing them... Previously with the post_compile config command I was setting the user/pass also with env vars (ex: poetry config http-basic.my-pypi "$PYPI_USERNAME" "$PYPI_PASSWORD") and it was working fine.

[edmorley]

@claireramming Hi! Thank you for filing this :-)

The Python buildpack hasn't historically exposed app config vars (apart from PIP_EXTRA_INDEX_URL) when running the pip/Pipenv package manager install steps, since some apps have broken env vars set (for example invalid values for PYTHONHOME, PYTHONPATH etc) that would cause the package manager to fail in hard to debug ways. As such, when adding support for Poetry I chose to match that approach for parity.

However, I would like to see if we can change this in the future (which would allow you to use the POETRY_HTTP_BASIC_<NAME>_USERNAME and POETRY_HTTP_BASIC_<NAME>_PASSWORD env vars) - it will just require careful thought about what env vars are potentially problematic - and for the buildpack to either filter those out, or hard error with an error message saying those broken env vars should be unset.

(The buildpack already has some rudimentary filtering here, however, we also need to check for additional env vars, plus cover the case where env vars have been set by a duplicate Python buildpack.)

In the meantime you could create a Poetry config file manually in a bin/pre_compile script. (The poetry config command behind the scenes writes out the config as toml and saves it to the locations documented at https://python-poetry.org/docs/configuration/).

For example in bin/pre_compile:

#!/usr/bin/env bash

set -euo pipefail

# See: https://python-poetry.org/docs/configuration/
POETRY_CONFIG_DIR="${HOME}/.config/pypoetry"

echo "Creating Poetry auth.toml..."

mkdir -p "${POETRY_CONFIG_DIR}"

cat >"${POETRY_CONFIG_DIR}/auth.toml" <<EOF
[http-basic.<EXAMPLE_INDEX_NAME>]
username = "${EXAMPLE_PRIVATE_PYPI_USERNAME}"
password = "${EXAMPLE_PRIVATE_PYPI_PASSWORD}"
EOF

(This approach should only be used for secrets; for any other Poetry config, add it to a project-local poetry.toml in the root of the app source so it can be managed via Git, rather than using bin/pre_compile.)

Alternatively, it should also be possible to configure credentials via a .netrc file (which Poetry should pick up automatically) - for which there may be some third party buildpacks available to write out the .netrc file.

[edmorley]

(Tracking adding support for configuring Poetry via env vars internally in GUS-W-17309193)

[claireramming]

Thanks @edmorley! I had a feeling I was underestimating the pre-compile step. That should work for us for now. We actually had gone with the poetry.toml file as a temporary workaround, but the pre-compile should be more secure.