Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save subruns information of hyperrun #869

Merged
merged 10 commits into from
Aug 18, 2024
Merged

Save subruns information of hyperrun #869

merged 10 commits into from
Aug 18, 2024

Conversation

dachengx
Copy link
Collaborator

@dachengx dachengx commented Aug 16, 2024

What is the problem / what does the code in this PR do

Hyperrun "process" subruns into hyperrun. This is different from the superrun workflow, where the subruns are only concatenated in Rechunker. But hyperrun concatenates the subruns in Plugin._fetch_chunk and split subruns in Plugin.iter. Previously the concatenation does not collect the information from different subrus so the metadata of subruns is not stored in metadata of hyperrun.

Can you briefly describe how it works?

This PR changes:

  1. Add attribute superrun to Chunk to store the subruns information: "start" and "end".
  2. Add functions _merge_superrun_in_chunk, _merge_runs_in_chunk, and _continuity_check to handle the metadata of subruns in attribute superrun, such as merge different subruns and split metadata while splitting data.
  3. Check whether we need to assign superrun to chunk in Plugin.do_compute. If there are multiple subruns in Chunk.superrun, we must assign the Chunk.subruns to the Chunk.superrun because hyperrun's subruns information is only handled here.

Can you give a minimal working example (or illustrate with a figure)?

Please look at the test_hyperrun_chunk_properties test function in tests/test_hyperruns.py.

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).

@coveralls
Copy link

coveralls commented Aug 16, 2024

Coverage Status

coverage: 89.802% (+0.2%) from 89.607%
when pulling 3268c8f on save_hyperrun_subruns
into 66e974b on master.

@dachengx dachengx marked this pull request as ready for review August 18, 2024 03:49
@dachengx dachengx merged commit 95f8ca2 into master Aug 18, 2024
8 checks passed
@dachengx dachengx deleted the save_hyperrun_subruns branch August 18, 2024 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants