Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set single thread of zstd and blosc #899

Merged
merged 3 commits into from
Oct 12, 2024
Merged

Conversation

dachengx
Copy link
Collaborator

What is the problem / what does the code in this PR do

Only XENONnT/outsource#213 can not limit the usage of CPU because zstd and blosc by default will use as many as CPU on the machine.

This PR replaces zstd with zstandard and sets a one-thread constraint of zstd and blosc.

The reason for replacing zstd with zstandard is listed here: https://python-zstandard.readthedocs.io/en/latest/projectinfo.html#comparison-to-other-python-bindings and https://github.com/indygreg/python-zstandard/blob/e27f2f49f187f5f698e0a6004bee2aed80fe39e3/docs/projectinfo.rst?plain=1#L30.

I hope this will not make the previously compressed data unable to be decompressed.

Can you briefly describe how it works?

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).

@dachengx dachengx marked this pull request as ready for review October 12, 2024 21:37
Copy link

Coverage Status

coverage: 89.918% (-0.3%) from 90.264%
when pulling 5b93e3c on _single_thread_compressor
into a528e4b on master.

@dachengx dachengx merged commit f8c3c6c into master Oct 12, 2024
7 of 8 checks passed
@dachengx dachengx deleted the _single_thread_compressor branch October 12, 2024 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant