Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn back to zstd because zstandard raise errors #902

Merged
merged 1 commit into from
Oct 13, 2024
Merged

Conversation

dachengx
Copy link
Collaborator

@dachengx dachengx commented Oct 13, 2024

What is the problem / what does the code in this PR do

Further fix #899

Without this PR, code (with straxen 9d2a6b6111b0e43051f53d19fd394c6861465fdb)

import shutil
import straxen
from straxen.test_utils import nt_test_run_id


shutil.rmtree("./strax_test_data", ignore_errors=True)
st = straxen.test_utils.nt_test_context()
st.set_context_config({
    "allow_multiprocess": True,
    "timeout": 120,
})

st.make(
    nt_test_run_id,
    "peaklets",
    processor=processor,
)

will raise error

2024-10-13 12:28:58,123 - utilix - DEBUG - Token exists at /home/xudc/.dbtoken
2024-10-13 12:28:58,124 - utilix - DEBUG - Token is valid.
/home/xudc/straxen/straxen/config/preprocessors.py:16: UserWarning: From straxen version 2.1.0 onward, URLConfig parameters will be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
  warnings.warn(
/home/xudc/straxen/straxen/plugins/records/records.py:467: RuntimeWarning: invalid value encountered in divide
  means = baseline_buffer / count
/home/xudc/straxen/straxen/plugins/records/records.py:470: RuntimeWarning: invalid value encountered in divide
  res["baseline_rms_mean"][:] = (baseline_rms_buffer / count)[:]
/home/xudc/strax/strax/processing/general.py:396: UserWarning: endtime of things is not sorted! touching_windows will return the indices of the first and last things which are touching the container.
  warnings.warn(
Exception in thread save_0:pulse_counts:
Traceback (most recent call last):
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 980, in _bootstrap_inner
Exception in thread save_0:peaklets:
Traceback (most recent call last):
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 917, in run
    self.run()
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xudc/strax/strax/storage/common.py", line 709, in save_from
    self._target(*self._args, **self._kwargs)
  File "/home/xudc/strax/strax/storage/common.py", line 709, in save_from
    source.throw(e)
  File "/home/xudc/strax/strax/storage/common.py", line 693, in save_from
    source.throw(e)
  File "/home/xudc/strax/strax/storage/common.py", line 693, in save_from
    new_f = self.save(chunk=chunk, chunk_i=chunk_i, executor=executor)
  File "/home/xudc/strax/strax/storage/common.py", line 736, in save
    new_f = self.save(chunk=chunk, chunk_i=chunk_i, executor=executor)
  File "/home/xudc/strax/strax/storage/common.py", line 736, in save
    bonus_info, future = self._save_chunk(
  File "/home/xudc/strax/strax/storage/files.py", line 331, in _save_chunk
    bonus_info, future = self._save_chunk(
  File "/home/xudc/strax/strax/storage/files.py", line 331, in _save_chunk
    filesize = strax.save_file(fn, **kwargs)
  File "/home/xudc/strax/strax/io.py", line 84, in save_file
    filesize = strax.save_file(fn, **kwargs)
  File "/home/xudc/strax/strax/io.py", line 84, in save_file
    result = _save_file(write_file, data, compressor)
  File "/home/xudc/strax/strax/io.py", line 93, in _save_file
    result = _save_file(write_file, data, compressor)
  File "/home/xudc/strax/strax/io.py", line 93, in _save_file
    d_comp = COMPRESSORS[compressor]["compress"](data)
zstd.ZstdError: cannot compress: Src size is incorrect
    d_comp = COMPRESSORS[compressor]["compress"](data)
zstd.ZstdError: cannot compress: Src size is incorrect
Caught error while saving pulse_counts!
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
convert_channel::	changed channel
convert_channel_like::	update area_per_channel
convert_channel_like::	update saturated_channel
Traceback (most recent call last):
  File "/home/xudc/t.py", line 16, in <module>
    st.make(
  File "/home/xudc/strax/strax/context.py", line 1755, in make
    for _ in self.get_iter(
  File "/home/xudc/strax/strax/context.py", line 1646, in get_iter
    generator.throw(e)
  File "/home/xudc/strax/strax/context.py", line 1613, in get_iter
    for n_chunks, result in enumerate(strax.continuity_check(generator), 1):
  File "/home/xudc/strax/strax/chunk.py", line 363, in continuity_check
    for chunk in chunk_iter:
  File "/home/xudc/strax/strax/processors/threaded_mailbox.py", line 316, in iter
    raise s.got_exception
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/cvmfs/xenon.opensciencegrid.org/releases/nT/el9.2024.09.1/anaconda/envs/XENONnT_el9.2024.09.1/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xudc/strax/strax/storage/common.py", line 709, in save_from
    source.throw(e)
  File "/home/xudc/strax/strax/storage/common.py", line 693, in save_from
    new_f = self.save(chunk=chunk, chunk_i=chunk_i, executor=executor)
  File "/home/xudc/strax/strax/storage/common.py", line 736, in save
    bonus_info, future = self._save_chunk(
  File "/home/xudc/strax/strax/storage/files.py", line 331, in _save_chunk
    filesize = strax.save_file(fn, **kwargs)
  File "/home/xudc/strax/strax/io.py", line 84, in save_file
    result = _save_file(write_file, data, compressor)
  File "/home/xudc/strax/strax/io.py", line 93, in _save_file
    d_comp = COMPRESSORS[compressor]["compress"](data)
zstd.ZstdError: cannot compress: Src size is incorrect

This is related to zstandard for some unknown and deep reason. So we turn back to zstd but only constraint it to a single core.

Maybe the error will occur when the data is too small to be compressed. But I do not really know.

Can you briefly describe how it works?

Please see the comments in io.py.

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).

@dachengx dachengx marked this pull request as ready for review October 13, 2024 17:34
@coveralls
Copy link

Coverage Status

coverage: 90.267% (+0.002%) from 90.265%
when pulling 9eea8ef on still_use_zstd
into 04e0930 on master.

@dachengx dachengx merged commit dca3545 into master Oct 13, 2024
7 of 8 checks passed
@dachengx dachengx deleted the still_use_zstd branch October 13, 2024 17:49
rsriya added a commit to XENONnT/base_environment that referenced this pull request Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants