Isolating conda environments from ~/.local #7173

ghost · 2018-04-16T20:32:25Z

I think, I am seeing a bug in conda create command where I am not able to create an isolated conda environment, which doesn't inherit packages from the base environment, in Ubuntu 16.04.

The reproducible steps for the bug are as follows.

Create a new Ubuntu 16.04 VM or any Ubuntu 16.04 machine.
Install azure-cli
pip install azure-cli

Let's assume the python version is 2.7 here, as that is the default python in Ubuntu 16.04

This command installs packages ~/.local/lib/python2.7

Install miniconda in Ubuntu 16.04
conda create -n testenv python=2.7
pip freeze

In pip freeze, I see the azure-cli packages from ~/.local/lib/python2.7 , which indicates to me that the testenv conda environment is not isolated, as it inherited the packages from the base environment.

I only checked this for azure-cli, but my guess is that it is true for other packages too.

For python virtual env, this is not an issue, I get an isolated python virtual environment. I also don't get this issue on windows. I only get this on Linux and Mac machines.

Please take a look at the issue and let me know if I am doing something wrong or it is a genuine bug.

Thanks

kalefranz · 2018-04-17T15:32:32Z

@mingwandroid What are your latest thoughts here? CPython has explicit cutouts and code paths to handle virtualenv. Maybe we should patch our cpython to basically always be in virtualenv mode? Of course, that's going to make people mad who are accustomed to the current behavior. Either way, I think there will be a portion of people unhappy. I'm not sure how to thread this needle.

ghost · 2018-04-17T15:40:56Z

One way I figured out to create an isolated conda environment is to 1) Create a python virtualenv, which is isolated. 2) Then clone that python virtualenv using conda create --clone option. Clone option doesn't have this problem.

kalefranz · 2018-04-17T15:58:38Z

Yeah the virtualenv you created is hooking into that hard-coded cpython behavior. I think the difference something related to sys.prefix vs sys.exec_prefix. Don't remember the exact details without digging into it a bit more.

ghost · 2018-04-17T16:04:20Z

https://github.com/pypa/pip/blob/master/src/pip/_internal/utils/misc.py#L283 is the line where pip has special condition for python virtual envs, which is not true for conda. In case that helps.

mingwandroid · 2018-04-17T16:52:11Z

@adyada, can we clarify a point here please, when you said:

In pip freeze, I see the azure-cli packages from ~/.local/lib/python2.7 , which indicates to me that the testenv conda environment is not isolated, as it inherited the packages from the base environment

Do you not mean that it inherited the packages from your system Python instead?

base environment means the packages in your conda base environment, to me, at least.

ghost · 2018-04-17T16:55:18Z

Yeah, I mean that the testenv conda environment inherited the packages from the system python. I didn't check for the base conda environment.

mingwandroid · 2018-04-17T17:03:55Z

@kalefranz, these deliberate exclusions are part of pip, as @adyada mentions, but really it's this part that seems to do the avoid user_site stuff (which is what ~/.local is called in the code AFAIK).

We could patch our pip to ignore user_site, but half the pip using world would maybe prefer we didn't? Or do you both contend that any exclusion done in pip for a virtualenv env should also always apply to a conda env? What about when freezing the conda base env? Should it still look in user_site?

Then there are those who say that conda should never ever look in user_site, and I have at least one foot in that camp (maybe we should invent an env var called CONDA_ECOSYSTEM_ISOLATION for this purpose? I face the same thing in the R world FWIW).

Whatever happens it should be consistent when freezing as when running and I think that's an uncontroversial statement?

kalefranz · 2018-04-17T18:01:34Z

Then there are those who say that conda should never ever look in user_site

I personally have a foot there too. And I'd go so far as saying this should be the default for any conda-installed python (given that any conda-install python will by definition always be in a conda environment). The environment variable I'd propose then would be to open up the interpreter to look in user_site if people actually want that behavior.

That's my personal opinion though, and why this issue is marked as a discussion topic. If we make any type of change here, we need to make sure it's thoroughly discussed and we have consensus on the behavior change.

ghost · 2018-04-17T18:18:31Z

I think, including user_site packages also caused a bug for us, when some people tried installing and using some packages and got errors. That's how the bug was reported to me. I will reproduce that bug and post here by today or tomorrow.

ghost · 2018-04-18T16:45:48Z

The error that I get when user_site packages are also included in an conda environment is that the import of some modules fail, even if that module shows up in pip freeze.
I am just pasting the output of pip freeze in the conda env and a command that shows the import failure.

The important lines are shown as bold lines.

So, backports.tempfile shows up in pip freeze, but then its import fails.

Apologies for pasting such a long output. Actually, I didn't install most of these packages in the conda environment, these were inherited from the user_site location, that is ~/.local/python2.7

I don't see this issue when I am able to create an isolated python environment using python virtual env

So, I am also on the side that we should not be including user_site packages in a conda env. Probably, might be better to have the conda behavior same as the python virtual env behavior in terms of isolation of environments.

(testenv) testuser@testVMadyada2: $ python -m pip freeze
adal==0.5.1
applicationinsights==0.11.2
argcomplete==1.9.4
asn1crypto==0.24.0
azure-batch==4.1.1
azure-batch-extensions==1.0.1
azure-cli==2.0.31
azure-cli-acr==2.0.23
azure-cli-acs==2.0.31
azure-cli-advisor==0.5.1
azure-cli-appservice==0.1.31
azure-cli-backup==1.1.1
azure-cli-batch==3.2.0
azure-cli-batchai==0.2.0
azure-cli-billing==0.1.8
azure-cli-cdn==0.0.14
azure-cli-cloud==2.0.13
azure-cli-cognitiveservices==0.1.12
azure-cli-command-modules-nspkg==2.0.1
azure-cli-configure==2.0.15
azure-cli-consumption==0.3.0
azure-cli-container==0.1.22
azure-cli-core==2.0.31
azure-cli-cosmosdb==0.1.20
azure-cli-dla==0.0.19
azure-cli-dls==0.0.21
azure-cli-eventgrid==0.1.12
azure-cli-eventhubs==0.1.2
azure-cli-extension==0.0.12
azure-cli-feedback==2.1.1
azure-cli-find==0.2.9
azure-cli-interactive==0.3.19
azure-cli-iot==0.1.19
azure-cli-keyvault==2.0.21
azure-cli-lab==0.0.21
azure-cli-monitor==0.1.5
azure-cli-network==2.0.28
azure-cli-nspkg==3.0.2
azure-cli-profile==2.0.22
azure-cli-rdbms==0.2.1
azure-cli-redis==0.2.12
azure-cli-reservations==0.1.2
azure-cli-resource==2.0.27
azure-cli-role==2.0.22
azure-cli-servicebus==0.1.2
azure-cli-servicefabric==0.0.12
azure-cli-sql==2.0.25
azure-cli-storage==2.0.31
azure-cli-vm==2.0.30
azure-common==1.1.9
azure-datalake-store==0.0.19
azure-graphrbac==0.40.0
azure-keyvault==0.3.7
azure-mgmt-advisor==1.0.1
azure-mgmt-authorization==0.40.0
azure-mgmt-batch==4.1.0
azure-mgmt-batchai==1.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-cdn==1.0.0
azure-mgmt-cognitiveservices==1.0.0
azure-mgmt-compute==4.0.0rc1
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==0.4.0
azure-mgmt-containerregistry==1.0.1
azure-mgmt-containerservice==3.0.1
azure-mgmt-cosmosdb==0.3.1
azure-mgmt-datalake-analytics==0.2.0
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.2.0
azure-mgmt-devtestlabs==2.0.0
azure-mgmt-dns==2.0.0rc1
azure-mgmt-eventgrid==0.4.0
azure-mgmt-eventhub==1.2.0
azure-mgmt-iothub==0.4.0
azure-mgmt-iothubprovisioningservices==0.1.0
azure-mgmt-keyvault==0.40.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-monitor==0.5.0
azure-mgmt-msi==0.1.0
azure-mgmt-network==2.0.0rc2
azure-mgmt-nspkg==2.0.0
azure-mgmt-rdbms==1.1.0
azure-mgmt-recoveryservices==0.1.0
azure-mgmt-recoveryservicesbackup==0.1.1
azure-mgmt-redis==4.1.0
azure-mgmt-reservations==0.1.0
azure-mgmt-resource==1.2.1
azure-mgmt-servicebus==0.4.0
azure-mgmt-servicefabric==0.1.0
azure-mgmt-sql==0.8.6
azure-mgmt-storage==1.5.0
azure-mgmt-trafficmanager==0.40.0
azure-mgmt-web==0.35.0
azure-multiapi-storage==0.2.0
azure-nspkg==2.0.0
azure-storage==0.34.3
azureml-core==0.1.0.806005
backports.tempfile==1.0
backports.weakref==1.0.post1
bcrypt==3.1.4
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
colorama==0.3.9
configparser==3.5.0
cryptography==2.2.2
entrypoints==0.2.3
enum34==1.1.6
funcsigs==1.0.2
futures==3.2.0
humanfriendly==4.10
idna==2.6
ipaddress==1.0.22
isodate==0.6.0
jmespath==0.9.3
jsonpickle==0.9.6
keyring==12.0.1
knack==0.3.2
mock==2.0.0
monotonic==1.4
msrest==0.4.27
msrestazure==0.4.26
ndg-httpsclient==0.4.4
oauthlib==2.0.7
paramiko==2.4.1
pathlib==1.0.1
pathlib2==2.3.0
pathspec==0.5.6
pbr==4.0.2
prompt-toolkit==1.0.15
pyasn1==0.4.2
pycparser==2.18
pydocumentdb==2.3.1
Pygments==2.2.0
PyJWT==1.6.1
PyNaCl==1.2.1
pyOpenSSL==17.5.0
python-dateutil==2.7.2
pytz==2018.4
PyYAML==3.12
requests==2.18.4
requests-oauthlib==0.8.0
ruamel.ordereddict==0.4.13
ruamel.yaml==0.15.37
scandir==1.7
scp==0.10.2
SecretStorage==2.3.1
six==1.11.0
sshtunnel==0.1.3
tabulate==0.8.2
urllib3==1.22
vsts-cd-manager==1.0.1
wcwidth==0.1.7
websocket-client==0.47.0
Whoosh==2.7.4
xmltodict==0.11.0
(testenv) testuser@testVMadyada2: $
(testenv) testuser@testVMadyada2: $
(testenv) testuser@testVMadyada2: $ python
Python 2.7.14 |Anaconda, Inc.| (default, Mar 27 2018, 17:29:31)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from azureml.core.project import Project
Traceback (most recent call last):
File "", line 1, in
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/core/project.py", line 15, in
from azureml.execution.execution_api import ExperimentExecution
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/execution/execution_api.py", line 6, in
from azureml.execution import _commands
File "/home/testuser/miniconda3/envs/testenv/lib/python2.7/site-packages/azureml/execution/_commands.py", line 37, in
from backports import tempfile as temp_dir_back
ImportError: cannot import name tempfile

kalefranz · 2018-04-19T06:32:45Z

Is pip freeze what's reaching into ~/.local here then? I'm guessing conda list doesn't show the packages... because those packages aren't actually in site-packages within the environment prefix, correct?

ghost · 2018-04-20T17:15:05Z

yeah, conda list doesn't show those packages.

This is the output of conda list.

(testenv) testuser@testVMadyada2:~$ conda list

packages in environment at /home/testuser/miniconda3/envs/testenv:

kalefranz · 2018-04-21T00:20:45Z

I'd be in favor of by default putting the kibosh on all the .local stuff for any python in a conda environment (and, to reiterate, that's ALL conda-installed python), but I'm certain there'll be some vocal griping if we do.

ghost · 2018-04-21T04:05:36Z

So, I could reproduce this behavior on windows too.

I think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages. This way conda won't break the behavior for existing users and there would "--no-user-site" argument to not include any user_site packages if someone wants it.

mingwandroid · 2018-04-21T10:27:39Z

I think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages.

The problem is interfacing this conda environment parameter to Python itself which is what is in control of whether ~/.local is added to the sys.path. Using an env. var is sub-optimal for external tooling that does not use the correct conda activate procedure. I guess we could put this in a file in the root of the environment and then modify our python executable to always look for this? Changing the behaviour of the python executable isn't something to do lightly though!

kalefranz · 2018-04-21T14:05:25Z

How does the python executable know if it’s been called from a virtualenv? Maybe we should be flipping that switch for all conda envs too? But then I think that’s going to have other ramifications than just removing .local from sys.path. Definitely something we’d need to be extremely careful about. Actually should probably just reach out to some cpython core maintainers for their thoughts.

…

Sent from my iPhone

On Apr 21, 2018, at 5:27 AM, Ray Donnelly ***@***.***> wrote: think, there could be some people who would be depending on user_site packages in a conda environment. So, a solution is to have an argument like "--no-user-site" in the conda create command, which then doesn't include user_site packages. The problem is interfacing this conda environment parameter to Python itself which is what is in control of whether ~/.local is added to the sys.path. Using an env. var is sub-optimal for external tooling that does not use the correct conda activate procedure. I guess we could put this in a file in the root of the environment and then modify our python executable to always look for this? Changing the behaviour of the python executable isn't something to do lightly though! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

mbargull · 2018-04-21T14:26:06Z

xref: pypa/pip#5060

mingwandroid · 2018-04-21T14:52:54Z

I think we would like to be able to configure the root env to also not look in ~/local, and if so in that case you just want to check a file relative to sys.prefix and virtual/base shouldn't matter.

ghost · 2018-04-21T15:22:38Z

It looks like python virtualenv after changing the sys.prefix, sets a sys.base_prefix attribute, which is then used to determine in pip that there is a virtualenv active.
https://github.com/pypa/pip/blob/master/src/pip/_internal/locations.py#L39

jakirkham · 2018-05-13T19:07:54Z

Please don’t disable ~/.local. 😟

takluyver · 2018-05-15T14:37:38Z

I'd support disabling user site packages for Python in conda. I always do this in my own conda environments, and I've certainly seen it cause confusion for users, e.g. when a package they just now conda-installed is shadowed by a version they installed with pip ages ago. User site packages is valuable when using system Python, to install packages without sudo, but conda is typically installed in your user account anyway, so it's just an extra complication.

I would do this by setting ENABLE_USER_SITE = False in site.py (this is what I set in conda envs locally):

https://github.com/python/cpython/blob/v3.6.5/Lib/site.py#L79-L81

I think that with this set, pip should refuse to do a --user install, though I haven't checked whether it currently does so.

Edit: My views on this got more complex, see my comment below.

kalefranz · 2018-07-25T17:25:49Z

xref: #448

kousu · 2022-04-10T02:36:56Z

@less-and-less-bugs wrote:

vim path of site.py
Then set ENABLE_USER_SITE False. Then all the packages in ./local would not be found by used environment now

This is exactly what we used to do:

https://github.com/spinalcordtoolbox/spinalcordtoolbox/blob/e6eb4299fa7d9e4a2fb8f6362c26908847a49ec8/install_sct#L608

But then I figured out (spinalcordtoolbox/spinalcordtoolbox#3606) that this was a little more reliable:

https://github.com/spinalcordtoolbox/spinalcordtoolbox/blob/7198415f186385c9d3d2e4246d58da4b0bf7e262/install_sct#L605

So maybe that'll help you.

I still am surprised every time that conda doesn't do this by default, since it's main feature (at least, as seen by the zeitgeist) is to provide reproducibility.

niander · 2022-04-15T20:28:22Z

This creates issues when using tox-conda plugin since it will use conda to create an environment but pip to install dependencies. Unfortunately, I am not sure how to inject a pyvenv.cfg when tox is handling the environments.

mfansler · 2022-04-15T20:55:08Z

@niander what is the problem exactly? Using pip is fine - it defaults to installing into the active Python's prefix. The issue is pip install --user installs. Is that what tox is using?

niander · 2022-04-15T21:09:38Z

No, they use regular pip install. From logs I can see that first tox-conda create a conda environment. Then, it installs pytest because that is one of the dependencies in my testenv. pytest have attrs >= 19.3.0 as dependency but pip doesn't install it because there is already a newer version of attrs installed in the user site-packages location. Since this is Windows, the location is %APPDATA%\roaming\python\python37\site-packages.

However, when later tox calls pytest, it will fail with ModuleNotFoundException.
So pip found the package in site-packages but when calling pytest it can't find it anymore.

I don't think it is script pointing to a different python issue because I can see that pytest and pip are executed from the same created conda environment.

mfansler · 2022-04-15T21:18:58Z

@niander Okay, makes sense. Have you tried including conda-ecosystem-user-package-isolation package under conda_deps? That should isolate the Python installation from user space.

niander · 2022-04-15T23:05:34Z

@mfansler I tried but then later I got an issue with calling pip install directly inside a toxenv (I do this to install a local package in dev mode). pip complains it can't find ssl library.

Nonetheless, I am still failing to understand why when pytest executes it can't find attr if this is installed in the site-packages. If I start the python inside the conda environment, I can do import attr, but the pytest executed by tox doesn't. Also, if I start pytest from the conda environment created by tox I don't get the same error. I am confused by that.

niander · 2022-04-16T01:00:47Z

I figured it out. It is because of the way tox executes commands. The subprocess that tox creates to execute the test commands doesn't include the environment %APPDATA%.

The code in site.py looks for the user site-packages and if that exists adds that to the python path (unless it is a venv, but not applicable here since it is a conda env and there is no pyvenv.cfg). The logic in site.py for Windows is to see if %APPDATA% exists, if not, uses ~ (%USERPROFILE%).

So, in this case, running inside tox, because the real user site-packages exists only inside %APPDATA% and not ~, site.py will assume that there is no user site-packages and won't add it to the python path.

What I am not sure is why then when tox (or tox-conda) is just installing dependencies the user site-packages is found and added to python path.

Edit: Adding setenv =, APPDATA = {env:APPDATA:~} to tox.ini resolved the situation. Maybe this is a tox issue since APPDATA is an important OS env?

github-actions · 2023-04-16T04:02:06Z

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

Verify that you can still reproduce the issue at hand
Comment that the issue is still reproducible and include:
- What OS and version you reproduced the issue on
- What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

itcarroll · 2023-11-15T17:45:28Z

So this issue has been ongoing for a long time, and github bot keeps closing issues that are not resolved:

Workarounds described include the conda-ecosystem-user-package-isolation package or setting PYTHONNOUSERSITE manually. Here is a case where this workaround is not useful.

A command line tool installed outside a conda environment, using pip install --user, goes in ~/.local/bin/ and includes as its first line #!/path/to/python. THAT Python is outside the conda environment. With PYTHONNOUSERSITE however, site packages are disabled for ALL Python interpreters. This breaks the command line tool.

New workaround.

Although conda does not create a virtual environment, a pyvenv.cfg file is checked by the site module regardless. So, do what venv does. Create a pyvenv.cfg file for your conda environment with the line include-system-site-packages = false.

$ conda activate <name>
$ cat > $CONDA_PREFIX/pyvenv.cfg
include-system-site-packages = false
$

I don't know what other repercussions that has involving sys.prefix, but so far so good. Your mileage may vary.

I strongly feel the conda dev team should provide the --no-site-packages option requested a decade ago in #394.

mfansler · 2023-11-15T18:53:22Z

Although conda does not create a virtual environment, a pyvenv.cfg file is checked by the site module regardless. So, do what venv does. Create a pyvenv.cfg file for your conda environment with the line include-system-site-packages = false.

This sounds like an excellent solution. The code for that mechanism has been stable since Python v3.3.

This could also be implemented as a Conda Forge package that literally just delivers a pyvenv.cfg to the $PREFIX with that line. I'd be happy to support it (feel free to tag me on staged-recipes to review).

jezdez · 2023-11-16T09:49:03Z

@itcarroll Please open a new ticket with your summary, it's no use piling on this 5-year-old ticket with yet more discussion where most of the current maintainers were not around.

jezdez · 2023-11-16T10:17:21Z

Full disclosure, I've opened conda/infrastructure#850 to lock older issue sooner, to make sure we're able to encourage actionable issues instead of reviving old ones.

itcarroll · 2023-11-16T14:04:21Z

@jezdez Done! You never know whether maintainers are more averse to duplication or re-activation, so thank you for the clear guidance.

kalefranz added pending::discussion contains some ongoing discussion that needs to be resolved prior to proceeding source::partner created by or for an Anaconda, Inc. partner company labels Apr 17, 2018

kalefranz changed the title ~~Unable to create an isolated conda environment on ubuntu 16.04~~ Isolating conda environments from ~/.local Apr 30, 2018

This was referenced May 8, 2018

Installing a package with conda-forge last in channels list, forces reinstall and downgrade of already installed packages #7261

Closed

Default to --user pypa/pip#1668

Closed

nehaljwani mentioned this issue Jul 1, 2018

"conda activate" does not set PYTHONOUSERSITE ContinuumIO/anaconda-issues#9688

Open

nehaljwani mentioned this issue Jul 19, 2018

sys.path contains --local installed python paths for a conda env #6018

Closed

kalefranz mentioned this issue Jul 25, 2018

conda incompatible with local installations? #448

Closed

lulmer mentioned this issue May 16, 2022

TypeError: forward() got an unexpected keyword argument 'return_dict' UKPLab/sentence-transformers#1553

Open

github-actions bot added the stale [bot] marked as stale due to inactivity label Apr 16, 2023

kenodegard added this to 🧭 Planning Apr 17, 2023

github-project-automation bot moved this to 🆕 New in 🧭 Planning Apr 17, 2023

github-actions bot added the stale::closed [bot] closed after being marked as stale label May 16, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale May 16, 2023

github-project-automation bot moved this from 🆕 New to 🏁 Done in 🧭 Planning May 16, 2023

jaimergp reopened this Nov 16, 2023

github-project-automation bot moved this from 🏁 Done to 🏗️ In Progress in 🧭 Planning Nov 16, 2023

jezdez closed this as completed Nov 16, 2023

github-project-automation bot moved this from 🏗️ In Progress to 🏁 Done in 🧭 Planning Nov 16, 2023

itcarroll mentioned this issue Nov 16, 2023

isolating Python in conda environments from site-packages #13337

Open

2 tasks

itcarroll mentioned this issue Nov 21, 2023

Isolate conda python from site packages, like venv does by default conda-forge/staged-recipes#24506

Closed

10 tasks

grahamgower mentioned this issue Nov 28, 2023

Problem installing genomatnn conda environment from .yml file grahamgower/genomatnn#61

Open

wbvguo mentioned this issue Apr 1, 2024

prioritize conda's site-packages before the .local library's site-packages #13746

Open

2 tasks

github-actions bot added the locked [bot] locked due to inactivity label Aug 22, 2024

github-actions bot locked as resolved and limited conversation to collaborators Aug 22, 2024

Isolating conda environments from ~/.local #7173

Isolating conda environments from ~/.local #7173

Comments

ghost commented Apr 16, 2018 • edited by kalefranz Loading

kalefranz commented Apr 17, 2018

ghost commented Apr 17, 2018

kalefranz commented Apr 17, 2018

ghost commented Apr 17, 2018

mingwandroid commented Apr 17, 2018

ghost commented Apr 17, 2018

mingwandroid commented Apr 17, 2018 • edited Loading

kalefranz commented Apr 17, 2018

ghost commented Apr 17, 2018

ghost commented Apr 18, 2018

kalefranz commented Apr 19, 2018

ghost commented Apr 20, 2018

packages in environment at /home/testuser/miniconda3/envs/testenv:

Name Version Build Channel

kalefranz commented Apr 21, 2018

ghost commented Apr 21, 2018

mingwandroid commented Apr 21, 2018 • edited Loading

kalefranz commented Apr 21, 2018 via email

mbargull commented Apr 21, 2018

mingwandroid commented Apr 21, 2018

ghost commented Apr 21, 2018

jakirkham commented May 13, 2018

takluyver commented May 15, 2018 • edited Loading

kalefranz commented Jul 25, 2018

kousu commented Apr 10, 2022 • edited Loading

niander commented Apr 15, 2022

mfansler commented Apr 15, 2022

niander commented Apr 15, 2022 • edited Loading

mfansler commented Apr 15, 2022 • edited Loading

niander commented Apr 15, 2022 • edited Loading

niander commented Apr 16, 2022 • edited Loading

github-actions bot commented Apr 16, 2023

itcarroll commented Nov 15, 2023

mfansler commented Nov 15, 2023

jezdez commented Nov 16, 2023

jezdez commented Nov 16, 2023

itcarroll commented Nov 16, 2023

ghost commented Apr 16, 2018 •

edited by kalefranz

Loading

mingwandroid commented Apr 17, 2018 •

edited

Loading

mingwandroid commented Apr 21, 2018 •

edited

Loading

takluyver commented May 15, 2018 •

edited

Loading

kousu commented Apr 10, 2022 •

edited

Loading

niander commented Apr 15, 2022 •

edited

Loading

mfansler commented Apr 15, 2022 •

edited

Loading

niander commented Apr 15, 2022 •

edited

Loading

niander commented Apr 16, 2022 •

edited

Loading