You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The getEntity function in FileSystemGraphObjectStore is currently iterating every entity on disk to catch duplicate entity keys. We should trust the DuplicateKeyTracker and return the first entity with the given key. On integrations that ingest a large amount of data, we do not want to iterate every local file.
This is one small change, but a significant improvement would be to create an in-memory Map that tracks the file path of the file by _key and the index of the entity in the file. That would drastically improve the performance of local entity searches.
The
getEntity
function inFileSystemGraphObjectStore
is currently iterating every entity on disk to catch duplicate entity keys. We should trust theDuplicateKeyTracker
and return the first entity with the given key. On integrations that ingest a large amount of data, we do not want to iterate every local file.See: https://github.com/JupiterOne/sdk/blob/master/packages/integration-sdk-runtime/src/storage/FileSystemGraphObjectStore/FileSystemGraphObjectStore.ts#L67
The text was updated successfully, but these errors were encountered: