Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remote: Add zstd to encoding types #330

Merged
merged 1 commit into from
Nov 16, 2023
Merged

remote: Add zstd to encoding types #330

merged 1 commit into from
Nov 16, 2023

Conversation

joverlee521
Copy link
Contributor

@joverlee521 joverlee521 commented Nov 16, 2023

Allows us to skip gzip compression when uploading .zst files. Includes adding ".zst" to the mimetypes encoding types dictionary because it is not yet included in the standard library.¹

¹ https://github.com/python/cpython/blob/2ef3676a5bb8fba531fb8237ce50c27ebe37fb96/Lib/mimetypes.py#L411-L417

Checklist

  • Checks pass

Allows us to skip gzip compression when uploading .zst files.
Includes adding ".zst" to the mimetypes encoding types dictionary
because it is not yet included in the standard library.¹

¹ https://github.com/python/cpython/blob/2ef3676a5bb8fba531fb8237ce50c27ebe37fb96/Lib/mimetypes.py#L411-L417
@joverlee521
Copy link
Contributor Author

Tested locally with

nextstrain remote upload s3://nextstrain-data/files/dengue/trial/ /tmp/sequences_all.fasta.zst

Seems to have worked as expected

$ curl -I https://data.nextstrain.org/files/dengue/trial/sequences_all.fasta.zst
HTTP/2 200 
content-type: application/zstd

Copy link
Member

@tsibley tsibley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for adding this.

nextstrain/cli/remote/s3.py Show resolved Hide resolved
@joverlee521 joverlee521 merged commit 91b8423 into master Nov 16, 2023
41 checks passed
@joverlee521 joverlee521 deleted the zstd-encoding branch November 16, 2023 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants