-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache Requirement Object Creation #10550
Conversation
… case there were only ~200 unique requirement objects created in ~5-10 minutes of resolution time
44a7076
to
bb88244
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if this should be a part of 21.3; leaving the decision to RM. @pradyunsg
Co-authored-by: Pradyun Gedam <[email protected]>
Is that just a sometimes flaky test? the diff between the second to last and last commits is just adding a max-size parameter on the cache. |
Yea. That test checks that a timeout of 0.001 seconds causes timeout errors. Apparently the network is too quick in some cases. :) |
Do I need to push an empty commit or anything? or is it just ok if there are some failed tests on the pr? |
Maybe re-triggering the builds could help. I will close and reopen the PR to do it. |
Thanks for the PR @jbylund! |
Bumps [pip](https://github.com/pypa/pip) from 21.2.4 to 21.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>21.3 (2021-10-11)</h1> <h2>Deprecations and Removals</h2> <ul> <li>Improve deprecation warning regarding the copying of source trees when installing from a local directory. (<code>[#10128](pypa/pip#10128) <https://github.com/pypa/pip/issues/10128></code>_)</li> <li>Suppress location mismatch warnings when pip is invoked from a Python source tree, so <code>ensurepip</code> does not emit warnings on CPython <code>make install</code>. (<code>[#10270](pypa/pip#10270) <https://github.com/pypa/pip/issues/10270></code>_)</li> <li>On Python 3.10 or later, the installation scheme backend has been changed to use <code>sysconfig</code>. This is to anticipate the deprecation of <code>distutils</code> in Python 3.10, and its scheduled removal in 3.12. For compatibility considerations, pip installations running on Python 3.9 or lower will continue to use <code>distutils</code>. (<code>[#10358](pypa/pip#10358) <https://github.com/pypa/pip/issues/10358></code>_)</li> <li>Remove the <code>--build-dir</code> option and aliases, one last time. (<code>[#10485](pypa/pip#10485) <https://github.com/pypa/pip/issues/10485></code>_)</li> <li>In-tree builds are now the default. <code>--use-feature=in-tree-build</code> is now ignored. <code>--use-deprecated=out-of-tree-build</code> may be used temporarily to ease the transition. (<code>[#10495](pypa/pip#10495) <https://github.com/pypa/pip/issues/10495></code>_)</li> <li>Un-deprecate source distribution re-installation behaviour. (<code>[#8711](pypa/pip#8711) <https://github.com/pypa/pip/issues/8711></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Replace vendored appdirs with platformdirs. (<code>[#10202](pypa/pip#10202) <https://github.com/pypa/pip/issues/10202></code>_)</li> <li>Support <code>PEP 610 <https://www.python.org/dev/peps/pep-0610/></code>_ to detect editable installs in <code>pip freeze</code> and <code>pip list</code>. The <code>pip list</code> column output has a new <code>Editable project location</code> column, and the JSON output has a new <code>editable_project_location</code> field. (<code>[#10249](pypa/pip#10249) <https://github.com/pypa/pip/issues/10249></code>_)</li> <li><code>pip freeze</code> will now always fallback to reporting the editable project location when it encounters a VCS error while analyzing an editable requirement. Before, it sometimes reported the requirement as non-editable. (<code>[#10410](pypa/pip#10410) <https://github.com/pypa/pip/issues/10410></code>_)</li> <li><code>pip show</code> now sorts <code>Requires</code> and <code>Required-By</code> alphabetically. (<code>[#10422](pypa/pip#10422) <https://github.com/pypa/pip/issues/10422></code>_)</li> <li>Do not raise error when there are no files to remove with <code>pip cache purge/remove</code>. Instead log a warning and continue (to log that we removed 0 files). (<code>[#10459](pypa/pip#10459) <https://github.com/pypa/pip/issues/10459></code>_)</li> <li>When backtracking during dependency resolution, prefer the dependencies which are involved in the most recent conflict. This can significantly reduce the amount of backtracking required. (<code>[#10479](pypa/pip#10479) <https://github.com/pypa/pip/issues/10479></code>_)</li> <li>Cache requirement objects, to improve performance reducing reparses of requirement strings. (<code>[#10550](pypa/pip#10550) <https://github.com/pypa/pip/issues/10550></code>_)</li> <li>Support editable installs for projects that have a <code>pyproject.toml</code> and use a build backend that supports :pep:<code>660</code>. (<code>[#8212](pypa/pip#8212) <https://github.com/pypa/pip/issues/8212></code>_)</li> <li>When a revision is specified in a Git URL, use git's partial clone feature to speed up source retrieval. (<code>[#9086](pypa/pip#9086) <https://github.com/pypa/pip/issues/9086></code>_)</li> <li>Add a <code>--debug</code> flag, to enable a mode that doesn't log errors and propagates them to the top level instead. This is primarily to aid with debugging pip's crashes. (<code>[#9349](pypa/pip#9349) <https://github.com/pypa/pip/issues/9349></code>_)</li> <li>If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones. (<code>[#9498](pypa/pip#9498) <https://github.com/pypa/pip/issues/9498></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Present a better error message, when a <code>file:</code> URL is not found. (<code>[#10263](pypa/pip#10263) <https://github.com/pypa/pip/issues/10263></code>_)</li> <li>Fix the auth credential cache to allow for the case in which the index url contains the username, but the password comes from an external source, such as keyring. (<code>[#10269](pypa/pip#10269) <https://github.com/pypa/pip/issues/10269></code>_)</li> <li>Fix double unescape of HTML <code>data-requires-python</code> and <code>data-yanked</code> attributes. (<code>[#10378](pypa/pip#10378) <https://github.com/pypa/pip/issues/10378></code>_)</li> <li>New resolver: Fixes depth ordering of packages during resolution, e.g. a dependency 2 levels deep will be ordered before a dependecy 3 levels deep. (<code>[#10482](pypa/pip#10482) <https://github.com/pypa/pip/issues/10482></code>_)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/abec8a701bfa66aa15fedf4c898011aa2d95f29e"><code>abec8a7</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/68a70486c9224f9d25be3cbf56c73d8a33c6a713"><code>68a7048</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/9f18a403ca41f4e42fbb89d286b6571a099cb54b"><code>9f18a40</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10481">#10481</a> from notatallshaw/prefer_failures</li> <li><a href="https://github.com/pypa/pip/commit/db496cbce518fa159476695db0cd4f1c1a8ab6f5"><code>db496cb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10563">#10563</a> from pradyunsg/shorter-timeout</li> <li><a href="https://github.com/pypa/pip/commit/4fac2b90a5d200b46e7b576013bb25f4ebb3f937"><code>4fac2b9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10550">#10550</a> from jbylund/joe/cache_requirement_creation</li> <li><a href="https://github.com/pypa/pip/commit/786957cf85a641d49b4cfcceef717ef229ac8238"><code>786957c</code></a> Use a shorter timeout, to ensure that this fails more often</li> <li><a href="https://github.com/pypa/pip/commit/1e3c127d4a938643aca1bbc25e6581493e316476"><code>1e3c127</code></a> Avoid passing <code>.</code> to vendoring</li> <li><a href="https://github.com/pypa/pip/commit/610424f9f8ad1f99d0a48bf9a53e7a9df4242304"><code>610424f</code></a> Quote "PreferenceInformation" to avoid runtime NameError</li> <li><a href="https://github.com/pypa/pip/commit/c01b5c6d8a4858cf733408b4b020933f902dda9e"><code>c01b5c6</code></a> Update a test for resolvelib 0.8.0</li> <li><a href="https://github.com/pypa/pip/commit/394a24eb1a5f9af5da7d4d2452ed5fe952de5db2"><code>394a24e</code></a> Upgrade resolvelib to 0.8.0</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.4...21.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.4&new-version=21.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Bumps [pip](https://github.com/pypa/pip) from 21.2.4 to 21.3. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>21.3 (2021-10-11)</h1> <h2>Deprecations and Removals</h2> <ul> <li>Improve deprecation warning regarding the copying of source trees when installing from a local directory. (<code>[#10128](pypa/pip#10128) <https://github.com/pypa/pip/issues/10128></code>_)</li> <li>Suppress location mismatch warnings when pip is invoked from a Python source tree, so <code>ensurepip</code> does not emit warnings on CPython <code>make install</code>. (<code>[#10270](pypa/pip#10270) <https://github.com/pypa/pip/issues/10270></code>_)</li> <li>On Python 3.10 or later, the installation scheme backend has been changed to use <code>sysconfig</code>. This is to anticipate the deprecation of <code>distutils</code> in Python 3.10, and its scheduled removal in 3.12. For compatibility considerations, pip installations running on Python 3.9 or lower will continue to use <code>distutils</code>. (<code>[#10358](pypa/pip#10358) <https://github.com/pypa/pip/issues/10358></code>_)</li> <li>Remove the <code>--build-dir</code> option and aliases, one last time. (<code>[#10485](pypa/pip#10485) <https://github.com/pypa/pip/issues/10485></code>_)</li> <li>In-tree builds are now the default. <code>--use-feature=in-tree-build</code> is now ignored. <code>--use-deprecated=out-of-tree-build</code> may be used temporarily to ease the transition. (<code>[#10495](pypa/pip#10495) <https://github.com/pypa/pip/issues/10495></code>_)</li> <li>Un-deprecate source distribution re-installation behaviour. (<code>[#8711](pypa/pip#8711) <https://github.com/pypa/pip/issues/8711></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Replace vendored appdirs with platformdirs. (<code>[#10202](pypa/pip#10202) <https://github.com/pypa/pip/issues/10202></code>_)</li> <li>Support <code>PEP 610 <https://www.python.org/dev/peps/pep-0610/></code>_ to detect editable installs in <code>pip freeze</code> and <code>pip list</code>. The <code>pip list</code> column output has a new <code>Editable project location</code> column, and the JSON output has a new <code>editable_project_location</code> field. (<code>[#10249](pypa/pip#10249) <https://github.com/pypa/pip/issues/10249></code>_)</li> <li><code>pip freeze</code> will now always fallback to reporting the editable project location when it encounters a VCS error while analyzing an editable requirement. Before, it sometimes reported the requirement as non-editable. (<code>[#10410](pypa/pip#10410) <https://github.com/pypa/pip/issues/10410></code>_)</li> <li><code>pip show</code> now sorts <code>Requires</code> and <code>Required-By</code> alphabetically. (<code>[#10422](pypa/pip#10422) <https://github.com/pypa/pip/issues/10422></code>_)</li> <li>Do not raise error when there are no files to remove with <code>pip cache purge/remove</code>. Instead log a warning and continue (to log that we removed 0 files). (<code>[#10459](pypa/pip#10459) <https://github.com/pypa/pip/issues/10459></code>_)</li> <li>When backtracking during dependency resolution, prefer the dependencies which are involved in the most recent conflict. This can significantly reduce the amount of backtracking required. (<code>[#10479](pypa/pip#10479) <https://github.com/pypa/pip/issues/10479></code>_)</li> <li>Cache requirement objects, to improve performance reducing reparses of requirement strings. (<code>[#10550](pypa/pip#10550) <https://github.com/pypa/pip/issues/10550></code>_)</li> <li>Support editable installs for projects that have a <code>pyproject.toml</code> and use a build backend that supports :pep:<code>660</code>. (<code>[#8212](pypa/pip#8212) <https://github.com/pypa/pip/issues/8212></code>_)</li> <li>When a revision is specified in a Git URL, use git's partial clone feature to speed up source retrieval. (<code>[#9086](pypa/pip#9086) <https://github.com/pypa/pip/issues/9086></code>_)</li> <li>Add a <code>--debug</code> flag, to enable a mode that doesn't log errors and propagates them to the top level instead. This is primarily to aid with debugging pip's crashes. (<code>[#9349](pypa/pip#9349) <https://github.com/pypa/pip/issues/9349></code>_)</li> <li>If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones. (<code>[#9498](pypa/pip#9498) <https://github.com/pypa/pip/issues/9498></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Present a better error message, when a <code>file:</code> URL is not found. (<code>[#10263](pypa/pip#10263) <https://github.com/pypa/pip/issues/10263></code>_)</li> <li>Fix the auth credential cache to allow for the case in which the index url contains the username, but the password comes from an external source, such as keyring. (<code>[#10269](pypa/pip#10269) <https://github.com/pypa/pip/issues/10269></code>_)</li> <li>Fix double unescape of HTML <code>data-requires-python</code> and <code>data-yanked</code> attributes. (<code>[#10378](pypa/pip#10378) <https://github.com/pypa/pip/issues/10378></code>_)</li> <li>New resolver: Fixes depth ordering of packages during resolution, e.g. a dependency 2 levels deep will be ordered before a dependecy 3 levels deep. (<code>[#10482](pypa/pip#10482) <https://github.com/pypa/pip/issues/10482></code>_)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/abec8a701bfa66aa15fedf4c898011aa2d95f29e"><code>abec8a7</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/68a70486c9224f9d25be3cbf56c73d8a33c6a713"><code>68a7048</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/9f18a403ca41f4e42fbb89d286b6571a099cb54b"><code>9f18a40</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10481">#10481</a> from notatallshaw/prefer_failures</li> <li><a href="https://github.com/pypa/pip/commit/db496cbce518fa159476695db0cd4f1c1a8ab6f5"><code>db496cb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10563">#10563</a> from pradyunsg/shorter-timeout</li> <li><a href="https://github.com/pypa/pip/commit/4fac2b90a5d200b46e7b576013bb25f4ebb3f937"><code>4fac2b9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10550">#10550</a> from jbylund/joe/cache_requirement_creation</li> <li><a href="https://github.com/pypa/pip/commit/786957cf85a641d49b4cfcceef717ef229ac8238"><code>786957c</code></a> Use a shorter timeout, to ensure that this fails more often</li> <li><a href="https://github.com/pypa/pip/commit/1e3c127d4a938643aca1bbc25e6581493e316476"><code>1e3c127</code></a> Avoid passing <code>.</code> to vendoring</li> <li><a href="https://github.com/pypa/pip/commit/610424f9f8ad1f99d0a48bf9a53e7a9df4242304"><code>610424f</code></a> Quote "PreferenceInformation" to avoid runtime NameError</li> <li><a href="https://github.com/pypa/pip/commit/c01b5c6d8a4858cf733408b4b020933f902dda9e"><code>c01b5c6</code></a> Update a test for resolvelib 0.8.0</li> <li><a href="https://github.com/pypa/pip/commit/394a24eb1a5f9af5da7d4d2452ed5fe952de5db2"><code>394a24e</code></a> Upgrade resolvelib to 0.8.0</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.4...21.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.4&new-version=21.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>
Requirement object creation is a large fraction of resolution time in some circumstances (~65% inclusive in the profile I'm looking at), and at least in this example there are very few distinct arguments to the
Requirement
constructor. Which I think makes this a decent candidate for caching.TODO: