Optimize synapse loading. #68
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is a WIP. The intention is to create an short-term release that can be used in simulations that have a large impact on GPFS. Therefore, the following topics are out-of-scope:
|
This comment has been minimized.
This comment has been minimized.
|
The corresponding libsonata changes: BlueBrain/libsonata#298 |
1uc
force-pushed
the
1uc/hotfix-edge-loading
branch
from
721d3bd to
9a1b913
Compare
This comment has been minimized.
This comment has been minimized.
1uc
force-pushed
the
1uc/hotfix-edge-loading
branch
3 times, most recently
from
c67ef5a to
eef0918
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
1uc
force-pushed
the
1uc/hotfix-edge-loading
branch
2 times, most recently
from
597f7fe to
fa1eb33
Compare
added 3 commits
October 23, 2023 19:39
This implements merging of edge IDs, reading larger chunks, then extracting the required data. The current implementation is a WIP, and has a few obvious weaknesses.
Advanced load balancing scatters node ids. However, contiguous node IDs is a prerequisite for merge-read-extract to work. Moreover, if the node IDs are highly scattered the current implementation will likely go OOM.
1uc
force-pushed
the
1uc/hotfix-edge-loading
branch
from
fa1eb33 to
42478a6
Compare
|
Logfiles from GitLab pipeline #164280 (:no_entry:) have been uploaded here! Status and direct links: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Reading synapses from edge files is done in many small, scattered reads.
Scope
Review