Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of FileSystemGraphObjectStore getEntity #385

Closed
austinkelleher opened this issue Nov 30, 2020 · 2 comments
Closed

Improve performance of FileSystemGraphObjectStore getEntity #385

austinkelleher opened this issue Nov 30, 2020 · 2 comments
Assignees

Comments

@austinkelleher
Copy link
Contributor

The getEntity function in FileSystemGraphObjectStore is currently iterating every entity on disk to catch duplicate entity keys. We should trust the DuplicateKeyTracker and return the first entity with the given key. On integrations that ingest a large amount of data, we do not want to iterate every local file.

See: https://github.com/JupiterOne/sdk/blob/master/packages/integration-sdk-runtime/src/storage/FileSystemGraphObjectStore/FileSystemGraphObjectStore.ts#L67

@austinkelleher
Copy link
Contributor Author

This is one small change, but a significant improvement would be to create an in-memory Map that tracks the file path of the file by _key and the index of the entity in the file. That would drastically improve the performance of local entity searches.

@austinkelleher
Copy link
Contributor Author

Fixed in #454

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant