Create utility to render and store transactional snapshot of simple detail page for a project #8586

ewdurbin · 2020-09-18T21:01:50Z

Needed for #8487.

Exposes packaging.utils.render_simple_detail for use in TUF integration.

def render_simple_detail(project, request, store=False):

Returns (content_hash, simple_detail_path)

The store flag can be used to control wether the resulting rendered/hashed simple detail page is pushed to our storage service.

ewdurbin · 2021-03-23T14:56:33Z

@di can I get a review and sanity check here before I go about implementing tests?

ewdurbin · 2021-03-23T14:56:38Z

cc @woodruffw

warehouse/legacy/api/simple.py

woodruffw

LGTM! One tiny question about when simple detail files are being stored 🙂

ewdurbin · 2021-08-25T16:24:54Z

Ready for review @di and @woodruffw.

warehouse/packaging/utils.py

woodruffw

LGTM!

warehouse/packaging/utils.py

di · 2021-08-25T22:02:04Z

warehouse/packaging/utils.py

+                },
+            )
+
+    return (content_hash, simple_detail_path)


Should we return content here instead, so the endpoint can directly render the simple page rather than having to fetch it from storage?

Initially, this is implemented so that we render simple detail pages in the same way that we currently do and this utility is available for TUF.

We'll need some transition plan before we start serving simple pages directly from object storage... which I consider material for a follow on PR.

Also as is, we don't ever fetch back simple pages from storage from warehouse.

Also as is, we don't ever fetch back simple pages from storage from warehouse.

Should we ever? Serving the request (and storing the rendered & hashed detail page) should happen every time cache is purged and a request hits the backend. The worse case is that we occasionally overwrite an existing file with itself.

For the hashed pages, we would need to serve those from storage of course.

ewdurbin · 2021-08-26T16:17:46Z

I'm not sure what the best next step is for this PR.

It seems we have two concerns at play in review:

Exposing the necessary utility for TUF
Establishing the full simple index in object storage

Initially this work is intended to provide only the first outcome, to unblock TUF integration into warehouse.

For the TUF concern, there appears to only be a decision regarding path names for files stored in the object store.

For the second one, I think we need the following:

Matching renderer for the simple_index view, and perhaps an hourly job to render and purge its cache?
A tool to backfill/regenerate simple indexes for all projects.
Some plan to transition

warehouse/packaging/utils.py

kairoaraujo

I am working on the following implementation PR on top of this PR (related to PEP 458, TUF).
I noticed that the simple HTML files generated by render_simple_details are empty (see my comment on warehouse/packaging/utils).

warehouse/packaging/services.py

warehouse/packaging/utils.py

…scussed also store unhashed index as normal

ewdurbin

I think this is good to go, it puts in place the mechanism for TUF to invoke render_simple_detail and get the hash/path for signing/service.

For now we'll remove serving all simple pages from storage from scope, and get this merged so TUF work can proceed when contributors are ready.

I'll merge this after configuring the new SIMPLE_BACKEND for test.pypi.org/pypi.org.

This commit enables the simple hash project index during the project upload file. This change will store the simple storage with the project hash index for new project uploads. This PR is a follow-up on PR pypi#8586, where it uses in the legacy file upload the render simple details as part of PEP 458. Signed-off-by: Kairo de Araujo <[email protected]>

…etail page for a project (pypi#8586) * helper function to render and hash simple detail for a specific project * use the existing Jinja2 environment * file storage * format with black * store hashed project indexes at `/simple/<HASH>.<PROJECT_NAME>` as discussed also store unhashed index as normal * add 'simple.backend' config for tests * add a jinja renderer to pyramid_request fixture * restore tests for existing functionality * test new SimpleStorage services * license * test render_simple_index utility * Update warehouse/packaging/utils.py * Update warehouse/packaging/utils.py * fix tests, store last serial information on metadata of simple files * reformat/lint * Remove print statement * Add flush * Fix tests * Simpflify duplication in storage services Co-authored-by: Dustin Ingram <[email protected]>

ewdurbin force-pushed the render_and_hash_simple_detail branch from b4f35ca to b967eba Compare September 18, 2020 21:03

di mentioned this pull request Sep 18, 2020

Pre-generate and serve simple index metadata #8487

Open

ewdurbin force-pushed the render_and_hash_simple_detail branch from b967eba to 581e20e Compare January 12, 2021 16:14

Base automatically changed from master to main January 21, 2021 18:39

ewdurbin force-pushed the render_and_hash_simple_detail branch from 581e20e to c76e4e7 Compare February 9, 2021 17:48

ewdurbin force-pushed the render_and_hash_simple_detail branch from c76e4e7 to 3745775 Compare February 23, 2021 16:14

ewdurbin force-pushed the render_and_hash_simple_detail branch from b142122 to 27e862a Compare March 8, 2021 17:09

ewdurbin force-pushed the render_and_hash_simple_detail branch from 27e862a to 1c16dbe Compare March 23, 2021 14:10

ewdurbin force-pushed the render_and_hash_simple_detail branch from 1c16dbe to 33a7b76 Compare March 23, 2021 14:57

ewdurbin commented Mar 23, 2021

View reviewed changes

warehouse/legacy/api/simple.py Outdated Show resolved Hide resolved

ewdurbin requested a review from di March 23, 2021 15:09

woodruffw approved these changes Mar 23, 2021

View reviewed changes

ewdurbin force-pushed the render_and_hash_simple_detail branch from 33a7b76 to 2de0757 Compare April 29, 2021 14:06

ewdurbin marked this pull request as ready for review August 25, 2021 16:24

woodruffw reviewed Aug 25, 2021

View reviewed changes

warehouse/packaging/utils.py Outdated Show resolved Hide resolved

woodruffw approved these changes Aug 25, 2021

View reviewed changes

di reviewed Aug 25, 2021

View reviewed changes

ewdurbin commented Aug 26, 2021

View reviewed changes

warehouse/packaging/utils.py Outdated Show resolved Hide resolved

ewdurbin commented Aug 26, 2021

View reviewed changes

warehouse/packaging/utils.py Outdated Show resolved Hide resolved

ewdurbin force-pushed the render_and_hash_simple_detail branch from 52635fd to 7a94d77 Compare November 9, 2021 14:32

di approved these changes Nov 9, 2021

View reviewed changes

kairoaraujo reviewed Jan 27, 2022

View reviewed changes

warehouse/packaging/services.py Outdated Show resolved Hide resolved

warehouse/packaging/utils.py Show resolved Hide resolved

ewdurbin added 4 commits February 1, 2022 10:25

helper function to render and hash simple detail for a specific project

5d3455d

use the existing Jinja2 environment

9769dc5

file storage

383da2f

format with black

b475183

ewdurbin and others added 13 commits February 1, 2022 10:26

store hashed project indexes at /simple/<HASH>.<PROJECT_NAME> as di…

2b3a956

…scussed also store unhashed index as normal

add 'simple.backend' config for tests

8ab2f22

add a jinja renderer to pyramid_request fixture

3b44537

restore tests for existing functionality

2f66a21

test new SimpleStorage services

4c26eb2

license

abe198b

test render_simple_index utility

b02e09c

Update warehouse/packaging/utils.py

14d8f37

Update warehouse/packaging/utils.py

16a9246

fix tests, store last serial information on metadata of simple files

de4b7d9

reformat/lint

c344e5b

Remove print statement

46af08d

Add flush

a5c60e0

di force-pushed the render_and_hash_simple_detail branch from 7a94d77 to a5c60e0 Compare February 1, 2022 16:07

di added 2 commits February 1, 2022 14:47

Fix tests

11aed13

Simpflify duplication in storage services

0292b6a

di mentioned this pull request Feb 1, 2022

Roadmap for PEP 458 #10672

Open

52 tasks

ewdurbin commented Feb 2, 2022

View reviewed changes

ewdurbin merged commit f116534 into main Feb 2, 2022

ewdurbin deleted the render_and_hash_simple_detail branch February 2, 2022 18:13

di mentioned this pull request Feb 2, 2022

Remove legacy path storage #10677

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create utility to render and store transactional snapshot of simple detail page for a project #8586

Create utility to render and store transactional snapshot of simple detail page for a project #8586

ewdurbin commented Sep 18, 2020 •

edited

Loading

ewdurbin commented Mar 23, 2021

ewdurbin commented Mar 23, 2021

woodruffw left a comment

ewdurbin commented Aug 25, 2021

woodruffw left a comment

di Aug 25, 2021

ewdurbin Aug 26, 2021

ewdurbin Aug 26, 2021

di Aug 26, 2021

ewdurbin Aug 27, 2021

ewdurbin commented Aug 26, 2021

kairoaraujo left a comment

ewdurbin left a comment

Create utility to render and store transactional snapshot of simple detail page for a project #8586

Create utility to render and store transactional snapshot of simple detail page for a project #8586

Conversation

ewdurbin commented Sep 18, 2020 • edited Loading

ewdurbin commented Mar 23, 2021

ewdurbin commented Mar 23, 2021

woodruffw left a comment

Choose a reason for hiding this comment

ewdurbin commented Aug 25, 2021

woodruffw left a comment

Choose a reason for hiding this comment

di Aug 25, 2021

Choose a reason for hiding this comment

ewdurbin Aug 26, 2021

Choose a reason for hiding this comment

ewdurbin Aug 26, 2021

Choose a reason for hiding this comment

di Aug 26, 2021

Choose a reason for hiding this comment

ewdurbin Aug 27, 2021

Choose a reason for hiding this comment

ewdurbin commented Aug 26, 2021

kairoaraujo left a comment

Choose a reason for hiding this comment

ewdurbin left a comment

Choose a reason for hiding this comment

ewdurbin commented Sep 18, 2020 •

edited

Loading