Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

setuptools: sdist reproducibility #105502

Closed
raboof opened this issue Nov 30, 2020 · 4 comments
Closed

setuptools: sdist reproducibility #105502

raboof opened this issue Nov 30, 2020 · 4 comments
Labels
0.kind: bug Something is broken 6.topic: python 6.topic: reproducible builds

Comments

@raboof
Copy link
Member

raboof commented Nov 30, 2020 •
edited
Loading

Describe the bug

The 'sdist' intermediate source derivation of the python setuptools is not currently reproducible.

To Reproduce
Steps to reproduce the behavior:

  1. nix-store --query --graph $(nix-instantiate '<nixpkgs>' -A python3Packages.setuptools) | grep sdist
  2. find the derivation path
  3. with that derivation path: nix-build <derivation-path> --check

Expected behavior

'check' should succeed.

We already untar and then retar the generated archive to control the timestamps.

While the version of setuptools we're on (47.3.1) is not very old (June 2020), upstream has since had several releases, up to v50.3.2. A quick test reveals 50.3.2 also does not yet produce stable timestamps. Upstream work on this has started (pypa/setuptools#2136, pypa/setuptools#1512, python/cpython#20331) but stalled.
@FRidh
Copy link
Member

FRidh commented Dec 1, 2020

50.3.1 is now in staging-next. #105584

As you linked to, this is in the end a distutils issue. That makes it a bit complicated, because distutils is moved out of cpython, which means any solution will likely be Python version dependent.

@raboof
Copy link
Member Author

raboof commented Dec 1, 2020

That makes it a bit complicated, because distutils is moved out of cpython

How do you mean that? It looks like https://github.com/python/cpython/blob/master/Lib/distutils/archive_util.py is still there? (I'm not too familiar with the ecosystem, might have missed something)

@FRidh
Copy link
Member

FRidh commented Dec 1, 2020

Still a draft though https://www.python.org/dev/peps/pep-0632/, I thought it was already accepted.

raboof added a commit to raboof/nixpkgs that referenced this issue Dec 2, 2020
@raboof
setuptools: stable file ordering for sdist
8e76a36 
We already re-pack the source distribution to make the
timestamps consistent, but the result is still not
entirely stable because by default tar does not sort
the files in the archive deterministically. This PR
makes sure the entries are sorted by name.

NixOS#105502
@raboof raboof mentioned this issue Dec 2, 2020
Merged
10 tasks
FRidh pushed a commit that referenced this issue Dec 8, 2020
@raboof @FRidh
setuptools: stable file ordering for sdist
17c8400 
We already re-pack the source distribution to make the
timestamps consistent, but the result is still not
entirely stable because by default tar does not sort
the files in the archive deterministically. This PR
makes sure the entries are sorted by name.

#105502
@raboof
Copy link
Member Author

raboof commented Mar 4, 2021

Long-closed by #105680

@raboof raboof closed this as completed Mar 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0.kind: bug Something is broken 6.topic: python 6.topic: reproducible builds
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@raboof @veprbl @FRidh