Install with -e in requirements.txt results in missing dependency

[johnbender]

Hi!

Up front I'd like to say thanks for all hard work on the build pack. I'd also like say that my experience managing dependencies with python is minimal so I apologize if I've done something to cause the issue myself.

We have two submodules in our project repository that we install using the following convention in our requirements.txt (reproduced from our repo):

# this is the default included as a part of the python build pack from heroku
bob-builder==0.0.5

# install the softix submodules
-e lib/softix/core
-e lib/softix/cli

Previously this worked just fine but now when attempting to make use of the CLI tool from our app we're getting the following stack trace:

$ heroku run bash                         
Running bash on  wastastaging... up, run.4739 (Free)
~ $ softix --help
Traceback (most recent call last):
  File "/app/.heroku/python/bin/softix", line 6, in <module>
    from pkg_resources import load_entry_point
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 3017, in <module>
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 3003, in _call_aside
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 3030, in _initialize_master_working_set                                                                                                 File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 659, in _build_master
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 967, in require
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 853, in resolve
pkg_resources.DistributionNotFound: The 'softixcli' distribution was not found and is required by the application

To aid in reproduction of the issue I've created a repository that can be deployed to a new instance:

https://github.com/johnbender/heroku-buildpack-python-issue

I noticed that I had to run the deploy once or twice and also attempt to run the command once or twice to get the above error. :(

Our current workaround is to reference an older version of the buildpack:

$ heroku buildpacks                                                                                                                                                 
=== Buildpack URLs
1. https://github.com/heroku/heroku-buildpack python#1bccc03b30d31c314452f2648ebb0296f1e5f155
2. heroku/nodejs

[kennethreitz]

Thanks for the report. There was a bug over the weekend with all -e builds, including ones pointed to GitHub. I'm working on a fix today to fix this type today as well. Sorry for the inconvenience. The fix should be easy.

[johnbender]

@kennethreitz

No problem at all! That's why we have a staging environment. :)

[ian-axelrod]

Thanks!

[kennethreitz]

can you try again?

[johnbender]

With the following:

$ heroku buildpacks
=== Buildpack URLs
1. https://github.com/heroku/heroku-buildpack-python#324ebc92234ffbbbdde802fc87b1d76a1d86e9e2
2. heroku/nodejs

I'm not seeing the dependency resolution error anymore (ie, it works).

I'm not clear on what exactly the canonical buildpack heroku/python points at but I tried that also and it doesn't work. I assume you have some release process that points it at a given commit when you're ready.

[kennethreitz]

Correct — ideally, https://github.com/heroku/heroku-buildpack-python works perfectly, and can be deployed tomorrow.

[ryanhiebert]

I was seeing this same issue, but this fix hasn't worked for me. Using buildpack https://github.com/heroku/heroku-buildpack-python. It looks like it's already deployed, because it's failing in the same way as heroku/python, but in a new way.

Now, during my build, it's giving me this error. This is the same commit that I have previously deployed successfully. It's different that the error is given during the build, instead of after, but it seems to otherwise be a very similar issue.

Note that the -e package in question, aspiredu, is successfully installed.

remote: -----> Python app detected
remote:      $ pip install -r requirements.txt
remote:        Obtaining file:///tmp/build_1efda2a3bb544c9902d13ce667028fc7 (from -r /tmp/build_1efda2a3bb544c9902d13ce667028fc7/requirements.txt (line 9))
remote:        Installing collected packages: aspiredu
remote:          Running setup.py develop for aspiredu
remote:        Successfully installed aspiredu
remote:
remote:      $ python manage.py collectstatic --noinput
remote:        Traceback (most recent call last):
remote:          File "manage.py", line 10, in <module>
remote:            execute_from_command_line(sys.argv)
remote:          File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
remote:            utility.execute()
remote:          File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/__init__.py", line 302, in execute
remote:            settings.INSTALLED_APPS
remote:          File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 55, in __getattr__
remote:            self._setup(name)
remote:          File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 43, in _setup
remote:            self._wrapped = Settings(settings_module)
remote:          File "/app/.heroku/python/lib/python2.7/site-packages/django/conf/__init__.py", line 99, in __init__
remote:            mod = importlib.import_module(self.SETTINGS_MODULE)
remote:          File "/app/.heroku/python/lib/python2.7/importlib/__init__.py", line 37, in import_module
remote:            __import__(name)
remote:        ImportError: No module named aspiredu.settings

[kennethreitz]

@ryanbrainard can you try against master again? I just changed the order for you, so the files should be available during the collectstatic step.

Once you confirm this change fixes the problem, I can deploy this.

[ryanhiebert]

Same thing. Here's the commands that I ran to switch the buidpack. Did I screw something up?

$ heroku buildpacks:remove https://github.com/heroku/heroku-buildpack-python'#1bccc03b30d31c314452f2648ebb0296f1e5f155' -r bld
Buildpack removed. Next release on aspiredu-bld-ryan will use heroku/nodejs.
Run git push heroku master to create a new release using this buildpack.
$ heroku buildpacks:add https://github.com/heroku/heroku-buildpack-python -r bld
Buildpack added. Next release on aspiredu-bld-ryan will use:
  1. heroku/nodejs
  2. https://github.com/heroku/heroku-buildpack-python

[kennethreitz]

@ryanbrainard can you paste your requirements.txt file?

[ryanhiebert]

Here's the top, slightly redacted:

cat requirements.txt | head -n 20
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
--extra-index-url https://pypi.fury.io/--redacted--/aspiredu/

-e .
amqp==1.4.9               # via kombu
anyjson==0.3.3            # via kombu
babel==2.3.4              # via django-phonenumber-field
beautifulsoup4==4.5.1     # via robobrowser
billiard==3.3.0.23        # via celery
boto==2.45.0
celery==3.1.25            # via django-celery
certifi==2016.9.26
cffi==1.9.1               # via cryptography
contextlib2==0.5.4        # via raven
cryptography==1.7.1       # via paramiko

[ryanhiebert]

The -e . is how aspiredu is found and collected. There is a setup.py, and the source files are in /src/python from the setup.py in the root of my repo.

[kennethreitz]

@ryanhiebert would you mind opening a support ticket and asking that I (Kenneth Reitz) take a look at it? This will give me permission to look at your slug and see what's going on

http://support.heroku.com

[ryanhiebert]

Will do.

[kennethreitz]

@ryanhiebert link me to the ticket here so I can look at it ASAP

[ryanhiebert]

https://help.heroku.com/tickets/434553

[ryanhiebert]

Thanks @kennethreitz , the latest master seems to be working for me!