Optimize graph object buffering and flushing #395
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
austinkelleher
requested review from
ctdio,
aiwilliams,
ndowmon and
mknoedel
ndowmon
reviewed
Dec 16, 2020
...integration-sdk-runtime/src/storage/FileSystemGraphObjectStore/FileSystemGraphObjectStore.ts
Show resolved
Hide resolved
ctdio
suggested changes
Dec 16, 2020
...-runtime/src/storage/FileSystemGraphObjectStore/__tests__/FileSystemGraphObjectStore.test.ts
Outdated
Show resolved
Hide resolved
ctdio
previously approved these changes
Dec 16, 2020
Only write prettified files to the file system on local collection
Add tests for job state upload calls
Share graph object creation test utils across tests and cleanup
Support for continuous integration data uploads
ctdio
previously approved these changes
Dec 16, 2020
ndowmon
previously approved these changes
Dec 16, 2020
mknoedel
reviewed
Dec 17, 2020
| ) { | ||
| await this.flushRelationshipsToDisk(); | ||
| await this.flushRelationshipsToDisk(onRelationshipsFlushed); | ||
| } | ||
| } | ||
|
|
||
| async getEntity({ _key, _type }: GraphObjectLookupKey): Promise<Entity> { |
There was a problem hiding this comment.
Thought: what information is typically gotten from previous entities? My first thought would be that it would usually only be foreign keys, which is likely just an id. Is it worth considering when we flush InMemoryGraphObjectStore, we maintain a map similar to the implementation of DuplicateKeyTracker. That way we could likely store much more than 500 entities and relationships for most of the use-cases of getEntity.
Maybe we could get lucky and reduce roundtrips to the disk a good amount of the time.
There was a problem hiding this comment.
There are definitely lots of additional caching improvements that we can make!
mknoedel
reviewed
Dec 17, 2020
...integration-sdk-runtime/src/storage/FileSystemGraphObjectStore/FileSystemGraphObjectStore.ts
Show resolved
Hide resolved
mknoedel
reviewed
Dec 17, 2020
mknoedel
previously approved these changes
Dec 17, 2020
Improve uploader queue to respect queue size instead of waiting for idle
austinkelleher
dismissed stale reviews from mknoedel, ndowmon, and ctdio
via
e3d379a
ndowmon
previously approved these changes
Dec 18, 2020
mknoedel
previously approved these changes
Dec 18, 2020
- @jupiterone/[email protected] - @jupiterone/[email protected] - @jupiterone/[email protected] - @jupiterone/[email protected] - @jupiterone/[email protected] - @jupiterone/[email protected] - @jupiterone/[email protected]
austinkelleher
dismissed stale reviews from mknoedel and ndowmon
via
3cb5160
ndowmon
approved these changes
Dec 18, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
The motivation of this change is to allow our
FileSystemGraphObjectStoreto buffer more data in memory before performing a flush.Previously Implementation Details
Whenever we called
jobState.findEntity,jobState.iterateEntities, orjobState.iterateRelationships, a flush would be performed. Multiple steps are typically running at the same time in an integration. This means that flushing is happening all of the time.New Implementation Details
We now have two in-memory graph object lookup tables:
When we perform a
jobState.findEntity, we first check mapAto see if we have this graph object buffered in memory. If we do not, then we perform our traditional lookup on disk. (There is actually more we can do to optimize this. See here: #385 (comment))When we perform a
jobState.iterateEntitiesor ajobState.iterateRelationships, we first iterate over the data we have buffered in mapB, then we perform our traditional disk file iteration method.