Option to look for pip.conf in conda environment

[jakirkham]

Related to issue ( conda-forge/conda-forge.github.io#518 ).

Noted that there was some special handling with virtualenv to figure out the pip.conf file to use related to that environment. Was wondering if the same thing might be possible for a conda environment. Perhaps something like $CONDA_PREFIX/pip/pip.conf or something similar for both Unix and Windows (guessing with .ini as the extension). Though others may have thoughts on where this should live and how it should be named.

cc @kalefranz @msarahan @pradyunsg

[jakirkham]

Related point that we may want to consider is using a different configuration file for when conda-build is running vs. not.

[jakirkham]

Should add the conda package building considerations are independent from the active conda environment considerations.

[kalefranz]

I'm probably fine with it being at os.path.join(sys.prefix, 'pip.conf'), just like with virtualenv.

FYI, maybe not relevant here, but the canonical way to tell if a directory is a conda environment (and the way conda itself uses) is os.path.isfile(os.path.join(sys.prefix, 'conda-meta', 'history')).

[jakirkham]

Thoughts @pfmoore? Would this be something pip would be interested in and/or accept a patch for?

[pfmoore]

@jakirkham I assume you're referring to the running_under_virtualenv function in pip? But I'd appreciate a more explicit pointer to the code that you mean.

Having sys.base_prefix differ from sys.prefix is defined by Python as the way to expose the fact that you're in a virtual environment (see here). The use of sys.real_prefix is historically how virtualenv did this, and we handle that because prior to core Python support for virtual environments, virtualenv was the only solution around. IMO virtualenv should now switch to the standard base_prefix approach, and in the longer term, pip could drop checking for real_prefix.

I'd suggest that conda environments follow the core Python standard and set base_prefix, and then there would be no need for pip to special-case them, and other tools should also work without any special considerations.

[jakirkham]

Honestly don't really know the pip codebase well enough to propose where it should go. Was reading the docs, which sparked the question. :)

Interesting. Not sure to what extent using base_prefix is possible, but agree that sharing an API is a good idea if at all workable. This might work in some cases (e.g. base_prefix points at the python used by conda and prefix points to the python in the current environment if there is one). Not sure what to do when we are talking about the environment conda lives in (i.e. root later base) where both of these are the same though 😕 (added to the list below).

Can think of a few tricky cases though. There may be more. Not totally sure the implications off the top of my head, but we'd have to be sure we understand what base_prefix and prefix mean in these cases.

1. Activating virtualenv installed into a conda environment.
2. Environment stacking. ( support stacking environments conda/conda#5159 )
3. Using build/host in conda-build.
4. Using pip.conf/pip.ini in the root/base environment of conda.

[pfmoore]

Honestly don't really know the pip codebase well enough to propose where it should go. Was reading the docs, which sparked the question. :)

Cool, I thought you were referring to what's in the code. I assume you were looking in the docs here. Pip doesn't look at $VIRTUAL_ENV, but what it does is basically equivalent - the virtual environment config file is identified by pip as:

        # finally virtualenv configuration first trumping others
        if running_under_virtualenv():
            yield kinds.VENV, [venv_config_file]

venv_config_file is simply {sys.prefix}/pip.conf, and running_under_virtualenv simply tests if sys.prefix differs from sys.base_prefix (or if sys.real_prefix is set).

[jakirkham]

That sounds right.

Ah ok thanks for the info.

What does pip do on Python 2 to detect a virtualenv?

So if we just drop a config file in sys.prefix, should that be enough for pip? Is there a way we can verify what config file it is using? Does being in a virtualenv actually affect this?

[pfmoore]

What does pip do on Python 2 to detect a virtualenv?

sys.real_prefix. As I say, virtualenv sets that so pip will use that. But for conda, it's better to set sys.base_prefix as that's more forward compatible (you can set sys.base_prefix from user code in Python, both 2 and 3).

So if we just drop a config file in sys.prefix, should that be enough for pip? Is there a way we can verify what config file it is using? Does being in a virtualenv actually affect this?

From the code I quoted, such a config file will only be used if you're in a virtualenv (i.e. sys.base_prefix is set and is not equal to sys.prefix). You can't verify "what config file pip is using", because it uses all of them - there's a hierarchy of files and config settings are taken from the highest-priority place the specific setting you're looked at is defined in. From the docs:

If multiple configuration files are found by pip then they are combined in the following order:

1. Firstly the site-wide file is read, then
2. The per-user file is read, and finally
3. The virtualenv-specific file is read.

Each file read overrides any values read from previous files, so if the global timeout is specified in both the site-wide file and the per-user file then the latter value is the one that will be used.

[jakirkham]

So I think it is important to point out that conda and virtualenv are coming at this from two different angles. Sorry if what follows is already obvious, want to establish a baseline before asking more questions.

AFAIK virtualenv allows one to create a separate environment for Python packages that links in an external Python (with batteries included) for use in this otherwise clean environment. Please correct me if I'm getting any of that wrong.

OTOH conda allows users to install Conda packages into an environment. Those may be (and a lot of them are) Python packages as well, but they are more similar to Python packages from a system package manager (e.g. apt-get). However a user can actually install a Conda package with a different Python interpreter altogether. In fact I have several environments with different Python interpreters and this is a common use case. It can also install dependencies of Python (e.g. OpenSSL) or things completely unrelated (e.g. Boost).

As a result of this, it's quite easy to have a version of Python installed in the base environment in conda that does not match the one in my active environment. Honestly one could be CPython and the other PyPy. So one of the things I'm trying to wrap my head around is what happens when base_prefix and prefix are not somehow related to the same Python. Namely what if base_prefix and prefix point to totally different, possibly incompatible Pythons? Related what happens when we mess with things like base_prefix under the hood give site.py seems to try and configure it (amongst other things)?

On a different point, the base environment in conda is technically just another environment. There's some special things that go into other environments to make sure that the conda CLI can be run when other environments are active (@kalefranz feel free to correct me if I'm missing anything here). Otherwise the base environment is no different than the other ones. The definitions of base_prefix and prefix really breakdown here as they should be the same. However we wouldn't want base to be treated as a somehow special environment from the other conda environments as it would break things for users not expecting this difference. How can we solve this issue?

Looking at the search priority might help to highlight the conceptual differences. Conda users typically think of each environment as being its own siloed /usr (or /usr/local if you would rather). The Anaconda and conda-forge stacks are even built up that way. For example, we provide our own certificates, which our copies of OpenSSL, curl, git, etc. are taught to look for. Users are free to build their own certificate package and replace ours, but it would still be expected to inhabit the same location relative to the prefix environment. So different environments could even have different certificates. Digression aside, we'd ideally want pip.conf/pip.ini in a conda environment to be treated as a site-wide file. Is that something we could achieve with the virtualenv approach or could we try another approach?

[dstufft]

If I understand conda correctly, I don't think that the requested feature is really specific to conda (even though the original ask suggests that it is). I think it might be better worded as adding a config file that is specific to the actual Python install. On a Windows machine with the Python.org installers, this might be located at C:\Python36\pip.conf, on a Linux machine it might be /usr/lib/python3.6/pip.conf, in a Conda environment it'd be relative to that conda environment.

[jakirkham]

Thanks @dstufft. Agree that Python installers (batteries included) seems like an overlapping use case with similar end goals ( even if I wasn't thinking of that initially ;).

[pradyunsg]

So, should we just unconditionally have {sys.prefix}/pip.conf as a configuration file location? I'm cool with that. Should be easy enough to implement too.

@pfmoore @dstufft Would you be fine with changing the "venv" to "site" for pip config, in 10.0.0? Otherwise, we could have a transition to remove venv references in configuration over the standard deprecation cycle, removing venv references in pip 11/12, if we want. I'm fine with either.

[pfmoore]

@pradyunsg I'm not sure that unconditionally having {sys.prefix}/pip.conf is right. Currently we only do that if we're in a venv - doing so when not isn't a change that anyone has requested as far as I know. Conda should be able to report itself as a virtual environment, by setting sys.base_prefix (that's how core Python expects virtual environments to report themselves as such).

This is not something I'm wiling to grant an exception for to put it in 10.0.0, as it's not a bug fix but a new feature. And regardless, I don't see why we should change venv -> site. The feature is there for virtual environments after all. (And I see conda environments as just another approach to virtual environments, so I'm using the term generically here, not limited to virtualenv/venv).

[pradyunsg]

I got that feeler (unconditional addition) from Donald's comment. I guess I'm trying to get too much into 10.0.0, as I said, I'm fine with this happening after pip 10.0.0. =)

…

On Sun, 1 Apr 2018, 14:22 Paul Moore, ***@***.***> wrote: @pradyunsg <https://github.com/pradyunsg> I'm not sure that *unconditionally* having {sys.prefix}/pip.conf is right. Currently we only do that if we're in a venv - doing so when not isn't a change that anyone has requested as far as I know. Conda should be able to report itself as a virtual environment, by setting sys.base_prefix (that's how core Python expects virtual environments to report themselves as such). This is not something I'm wiling to grant an exception for to put it in 10.0.0, as it's not a bug fix but a new feature. And regardless, I don't see why we should change venv -> site. The feature is there for virtual environments after all. (And I see conda environments as just another approach to virtual environments, so I'm using the term generically here, not limited to virtualenv/venv). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5060 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADH7SRgnZmxgaaVphT4QTmbC-bEYGwYBks5tkJVUgaJpZM4Sjm2w> .

[dstufft]

I've had a few of the Linux distros ask for something similiar over time, so I think it's something that folks want (to let them provide some default configuration for pip). I don't think that it needs to get added to 10.0 though, and I don't know that {sys.prefix}/pip.conf is the right location for it either. I think this isn't a replacement for the virtual env config file though, and would rather be something that gets added before it in the resolution chain (and virtualenv config files override it).

[jakirkham]

Conda should be able to report itself as a virtual environment, by setting sys.base_prefix (that's how core Python expects virtual environments to report themselves as such). ... And I see conda environments as just another approach to virtual environments, so I'm using the term generically here, not limited to virtualenv/venv

Unfortunately that's not really an accurate understanding of conda. Please read comment in detail if you haven't already. :)

The key take away is virtualenv's link an external Python into their environment. Conda environments install a totally different Python into their environment. So the distinction of where other Python's may exist relative to the active conda environment is irrelevant as they have no effect. The point gets even more muddied if we consider the environment where the conda executable lives (base).

[pfmoore]

OK, so conda environments are more like entire Python installations. OK. Pip currently doesn't have a per-installation config file. As @dstufft says, that's something we could add.

[jakirkham]

Right. The Linux package manager analogue is a pretty good one.

[IPetrik]

Has there been any progress on this? I'm in a related situation where my team has a separate (non-conda) installation of python, because we don't have root access on the machine, and would like a global pip config for our installation. I really expected it to look in {sys.prefix}/etc/pip.conf, or something. Is there a possibility of this being added?

[pradyunsg]

Not really.

Looking at the discussion above again, ISTM the conclusion was that a per-installation configuration file has to be added, with its appropriate location, such that it's loaded just before venv configuration file.

I guess what's left is figuring out the location and actually doing the implementation.

[IPetrik]

Thanks @pradyunsg. So how do we go about deciding on the location?

[zooba]

I want this badly enough to do the work :) (especially if/when I implement #6248, but right now for overriding the default index-url for a deployed environment - most of our deployments are full copies of Python that aren't tied to a system install at all but are wholly local to the directory they are in).

I assume it should be checking one (or more) of the paths that get listed when you do python -m sysconfig (even if we actually take it directly from sys), though I'm afraid I don't have enough info on a Windows install to decide which one 😄

base = "C:\Python37_x64"
exec_prefix = "C:\Python37_x64"
installed_base = "C:\Python37_x64"
installed_platbase = "C:\Python37_x64"
platbase = "C:\Python37_x64"
prefix = "C:\Python37_x64"
projectbase = "C:\Python37_x64"

With a venv there's a bit more variety:

base = "C:\Projects\pip\.env2"
exec_prefix = "C:\Projects\pip\.env2"
installed_base = "C:\Python37_x64"
installed_platbase = "C:\Python37_x64"
platbase = "C:\Projects\pip\.env2"
prefix = "C:\Projects\pip\.env2"
projectbase = "C:\Projects\pip\.env2\Scripts"

So I'm thinking one of base, prefix or exec_prefix. But I don't actually know how those vary on other platforms and whether it matters. sys.prefix would have been my gut instinct.

[jakirkham]

FWIW here's the output from a Conda install on macOS (in the base environment). This is essentially the same for a Conda install on Linux.

data = "/zopt/conda3"
include = "/zopt/conda3/include/python3.6m"
platinclude = "/zopt/conda3/include/python3.6m"
platlib = "/zopt/conda3/lib/python3.6/site-packages"
platstdlib = "/zopt/conda3/lib/python3.6"
purelib = "/zopt/conda3/lib/python3.6/site-packages"
scripts = "/zopt/conda3/bin"
stdlib = "/zopt/conda3/lib/python3.6"

Here's the output from the system Python on macOS.

data = "/System/Library/Frameworks/Python.framework/Versions/2.7"
include = "/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7"
platinclude = "/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7"
platlib = "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages"
platstdlib = "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7"
purelib = "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages"
scripts = "/System/Library/Frameworks/Python.framework/Versions/2.7/bin"
stdlib = "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7"

Also here's the output from CentOS 7's system Python.

data = "/usr"
include = "/usr/include/python2.7"
platinclude = "/usr/include/python2.7"
platlib = "/usr/lib64/python2.7/site-packages"
platstdlib = "/usr/lib64/python2.7"
purelib = "/usr/lib/python2.7/site-packages"
scripts = "/usr/bin"
stdlib = "/usr/lib64/python2.7"

Thus far ${data}/etc looks like a good place to search. Though users are probably not messing with the internals of the macOS framework build (case 2).

[zooba]

${data}/etc has no cross-platform basis, unfortunately. What are the paths included in the variables section of the output (to match the ones that I copied)?

Another idea is to simply copy "whatever system is used for sitecustomize.py", since there's a good chance anyone customizing pip configuration here is also customizing Python configuration too.

[jakirkham]

Ah missed those, sorry. There was a lot of noise in the variables section (is there a good way to filter that?), but it seems base is the same as data in all 3 cases.

[zooba]

What about prefix and exec_prefix?

[jakirkham]

Both of those also appear to be the same as base and data in all 3 cases.

[pradyunsg]

I'm probably fine with it being at os.path.join(sys.prefix, 'pip.conf'), just like with virtualenv.

That's what we do now. :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.