Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save multiple output when iteration stops in PostOffice #852

Merged
merged 2 commits into from
Jul 2, 2024

Conversation

dachengx
Copy link
Collaborator

@dachengx dachengx commented Jul 1, 2024

What is the problem / what does the code in this PR do

When testing SingleThreadProcessor, run the codes below:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

import straxen
from straxen.test_utils import nt_test_run_id

st = straxen.test_utils.nt_test_context()

assert st.is_stored(nt_test_run_id, 'raw_records')
# st.make(nt_test_run_id, 'records')
st.make(nt_test_run_id, 'event_info', processor='single_thread')

In strax_test_data/012882-pulse_counts-exzzquvjyt_temp, run ls -lah,

total 12K
drwxr-xr-x  2 xudc xudc 4.0K Jul  1 17:17 .
drwxrwxrwx 23 xudc xudc 4.0K Jul  1 17:18 ..
-rw-r--r--  1 xudc xudc 1.7K Jul  1 17:17 pulse_counts-exzzquvjyt-metadata.json

So no chunk is saved. This is because after a topic stops its iteration and if it is from a multiple output plugin like PulseProcessing, other provided data_types are not saved:

self._ack_topic_exhausted(topic)

Can you briefly describe how it works?

I made PostOffice._multi_output_topics a dictionary. The keys of it are the topics, and the values of it are all the provides of the plugin which provides the topic. Whenever a StopIteration is encountered PostOffice._fetch_new, all messages(outputs) will be sent to PostOffice._ack_topic_exhausted.

Can you give a minimal working example (or illustrate with a figure)?

As the codes above shown, after this PR, in strax_test_data/012882-pulse_counts-exzzquvjyt, run ls -lah,

total 16K
drwxr-xr-x  2 xudc xudc 4.0K Jun 29 17:06 .
drwxrwxrwx 23 xudc xudc 4.0K Jun 29 17:06 ..
-rw-r--r--  1 xudc xudc 3.9K Jun 29 17:06 pulse_counts-exzzquvjyt-000000
-rw-r--r--  1 xudc xudc 2.2K Jun 29 17:06 pulse_counts-exzzquvjyt-metadata.json

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).

@dachengx dachengx requested a review from JelleAalbers July 1, 2024 22:27
@dachengx dachengx marked this pull request as ready for review July 1, 2024 22:39
@dachengx dachengx requested a review from MerzJohannes July 1, 2024 22:39
@coveralls
Copy link

Coverage Status

coverage: 90.521% (+0.002%) from 90.519%
when pulling 2f593ce on multi_output
into 40f85a3 on master.

@coveralls
Copy link

Coverage Status

coverage: 90.507% (+0.002%) from 90.505%
when pulling 37c49f6 on multi_output
into 0ab921b on master.

@dachengx dachengx merged commit 2d772f1 into master Jul 2, 2024
9 checks passed
@dachengx dachengx deleted the multi_output branch July 2, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants