Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pip's built-in caching mechanism to improve repeat builds #926

Open
6 tasks
CaseyFaist opened this issue Mar 12, 2020 · 0 comments
Open
6 tasks

Use pip's built-in caching mechanism to improve repeat builds #926

CaseyFaist opened this issue Mar 12, 2020 · 0 comments

Comments

@CaseyFaist
Copy link
Contributor

12-factor applications should be robust to redeploy, and take steps to boot quickly. This is broken in #909, but after resolving that bug, are there improvements to be made?

Pip by default caches the packages it installs: https://pip.pypa.io/en/stable/reference/pip_install/#caching

But this is turned off for the buildpack currently:
https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/pip-install#L50

We do try to cache requirements files and validate updates ourselves on repeat builds:
https://github.com/heroku/heroku-buildpack-python/blob/master/bin/steps/pip-uninstall

But it's built on tooling that has not been maintained:
https://github.com/heroku-python/pip-pop/blob/master/bin/pip-diff

Rather than maintain it by default, investigate what pip supports to see if better tools are now available:
https://pip.pypa.io/en/stable/reference/pip_install/#wheel-cache
https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
https://pip.pypa.io/en/stable/reference/pip_install/#hashes-from-pypi
https://pip.pypa.io/en/stable/reference/pip_check/

  • Proof of concept to investigate if caching-dir will support heroku path requirements (build in temp, copy to /app/.heroku/*)
  • Speed test - installing from wheel cache faster than installing from pypi?
  • Security review - more secure to cache wheels?
  • Proof of concept - do repeat builds in ephemeral containers allow for hashing?
  • Proof of concept/Security review - what do we gain checking pypi's hashes against packages?
  • Proof of concept - Verify installed packages have compatible dependencies (which we don't currently do)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants