diff --git a/source/guides/index-mirrors-and-caches.rst b/source/guides/index-mirrors-and-caches.rst index 9fdea0a1d..45eadcb30 100644 --- a/source/guides/index-mirrors-and-caches.rst +++ b/source/guides/index-mirrors-and-caches.rst @@ -4,20 +4,25 @@ Package index mirrors and caches ================================ -:Page Status: Incomplete -:Last Reviewed: 2014-12-24 +:Page Status: Ongoing +:Last Reviewed: 2023-11-08 -Mirroring or caching of PyPI can be used to speed up local package installation, -allow offline work, handle corporate firewalls or just plain Internet flakiness. +Mirroring or caching of PyPI (and other package indexes) can be used to speed +up local package installation, allow offline work, work with corporate +firewalls or handle just plain internet flakiness. -Three options are available in this area: +There are multiple classes of options in this area: -1. pip provides local caching options, -2. devpi provides higher-level caching option, potentially shared amongst - many users or machines, and -3. bandersnatch provides a local complete mirror of all PyPI :term:`packages - `. +1. local/hosted caching of package indexes. + +2. local/hosted mirroring of a package index. A mirror is a (whole or + partial) copy of a package index, which can be used in place of the + original index. + +3. private package index with fall-through to public package indexes (for + example, to mitigate dependency confusion attacks), also known as a + proxy. Caching with pip @@ -47,17 +52,227 @@ documentation for getting started`__. __ https://devpi.net/docs/devpi/devpi/latest/+d/quickstart-pypimirror.html +devpi has additional functionality, such as mirroring package indexes, running +multiple indexes with a concept of inheritance, syncing between multiple +servers, index replication and fail-over, and package upload. + +* `devpi on PyPI `_ +* `devpi source `_ + Complete mirror with bandersnatch ---------------------------------- -bandersnatch will set up a complete local mirror of all PyPI :term:`packages +bandersnatch will set up a complete local (or `AWS S3`_) mirror of all PyPI +:term:`packages ` (externally-hosted packages are not mirrored). See the `bandersnatch documentation for getting that going`__. -__ https://github.com/pypa/bandersnatch/ +__ https://bandersnatch.readthedocs.io/en/latest/ A benefit of devpi is that it will create a mirror which includes :term:`packages ` that are external to PyPI, unlike bandersnatch which will only cache :term:`packages ` hosted on PyPI. + +* `bandersnatch on PyPI `_ +* `bandersnatch source `_ + + +Other package index servers +--------------------------- + +simpleindex +^^^^^^^^^^^ + +Routes URLs to multiple package indexes (including PyPI), serves local (or +cloud-hosted, for example `AWS S3`_, with a custom plugin) directory of +packages, no caching without custom plugins, no mirroring. + +* `simpleindex on PyPI `_ +* `simpleindex source / documentation + `_ + +pypiserver +^^^^^^^^^^ + +Serves local directory of packages, no fall-through to package indexes +(including PyPI), supports package upload. + +* `pypiserver on PyPI `_ +* `pypiserver source / documentation + `_ + +pypiprivate +^^^^^^^^^^^ + +Serves local (or `AWS S3`_-hosted) directory of packages, no fall-through to +package indexes (including PyPI). + +* `pypiprivate on PyPI `_ +* `pypiprivate source / documentation + `_ + +PyPI Cloud +^^^^^^^^^^ + +PyPI server, backed by `AWS S3`_, another cloud storage service, or local +files. Supports redirect/cached proxying, authentication and authorisation, no +mirroring. + +* `PyPI Cloud on PyPI `_ +* `PyPI Cloud source `_ +* `PyPI Cloud documentation `_ + +pywharf +^^^^^^^ + +.. warning:: Not maintained, project archived + +PyPI server, backed by GitHub or local files. No proxy or mirror. + +* `pywharf on PyPI `_ +* `pywharf source `_ + +Python package index plugin for Pulp +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Supports local/`AWS S3`_ mirrors, package upload, proxying to multiple indexes, +no caching. + +* `pulp_python on PyPI `_ +* `pulp_python documentation `_ +* `pulp_python source `_ + +pip2pi +^^^^^^ + +Manual syncing of specific packages, no proxy. + +* `pip2pi on PyPI `_ +* `pip2pi source / documenation `_ + +proxpi +^^^^^^ + +Package index caching proxy, supports multiple indexes, no mirroring. + +* `proxpi on PyPI `_ +* `proxpi source `_ + +nginx_pypi_cache +^^^^^^^^^^^^^^^^ + +Package index caching proxy using `nginx `_ + +* `nginx_pypi_cache source `_ + +Flask-Pypi-Proxy +^^^^^^^^^^^^^^^^ + +.. warning:: Not maintained, project archived + +Caches PyPI. No cache size limit, no caching index pages. + +* `Flask-Pypi-Proxy on PyPI `_ +* `Flask-Pypi-Proxy documentation + `_ +* `Flask-Pypi-Proxy source `_ + +http.server +^^^^^^^^^^^ + +Standard-library, hosts directory exactly as laid out, no proxy to package +indexes (eg PyPI). See more in :ref:`Hosting your Own Simple Repository`. + +* `http.server documentation + `_ + +Apache +^^^^^^ + +Using +`mod_rewrite `_ and +`mod_cache_disk +`_, +you can cache requests to package indexes through an Apache server. + +Gemfury +^^^^^^^ + +Hosted and managed solution. Private indexes are not free, documentation +doesn't say anything about fall-through. + +* `Host Python packages on Gemfury `_ +* `Gemfure PyPI documentation `_ + +Artifactory +^^^^^^^^^^^ + +Hosted and managed solution. Proxy (with caching) multiple package indexes, and +host a new package index (supporting upload) with fall-through. Can be +self-hosted (not for free). + +* `JFrog Artifactory `_ +* `PyPI Repositories on Artifactory documentation + `_ + +Nexus Repository Manager +^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Proxy (with caching) multiple package indexes, and +host a new package index (supporting upload) with fall-through. + +* `Sonatype Nexus repository + `_ +* `PyPI documentation for Nexus + `_ + +Coherent Minds PyPI Filter +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Only filters requests, redirecting to PyPI if not +filtered, and blocking requests otherwise. + +* `Coherent Minds PyPI filter `_ + +GitLab Package Registry +^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private and public package index with +optional fall-through, permissioning. + +* `GitLab documentation + `_ + +AWS CodeArtifact +^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with optional cached +fall-through to PyPI. + +* `AWS CodeArtifact `_ +* `Python packages on CodeArtifact documentation + `_ + +Azure Artifacts +^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with optional fall-through. + +* `Azure Artifacts + `_ +* `Python packages on Azure Artifacts documentation + `_ + +Google Artifact Registry +^^^^^^^^^^^^^^^^^^^^^^^^ + +Hosted and managed solution. Private package index with no fall-through nor +mirroring. + +* `Google Artifact Registry `_ +* `Python packages on Artifact Registry documentation + `_ + +.. _`AWS S3`: https://aws.amazon.com/s3/