fix: performance on large repositories #220

cfabianski · 2022-12-09T13:59:25Z

Description

Fixes performance issues seen on the Gitlab repository.

The cause was running the encrypted field processor for each detection. We now run the processor once with all detections.

Also:

Removes configurable processors, in practice the code only supported the encrypted field one anyway.
Adds some additional logging
Ignores symlinks
Increases max file size to 2mb

I've added test coverage that shows my fix or feature works as expected.
I've updated or added documentation if required.
I've included usage information in the description if CLI behavior was updated or added.
PR title follows Conventional Commits format

didroe · 2022-12-13T10:05:09Z

didroe force-pushed the fix/processing-of-structure-sql branch 3 times, most recently from 72f14e4 to ea2770a Compare December 13, 2022 09:13

didroe changed the title ~~wip: ignore symlink and extend to bigger files~~ fix: performance on large repositories Dec 13, 2022

didroe added the echoes/initiative: curio label Dec 13, 2022

cfabianski and others added 5 commits December 13, 2022 10:06

wip: ignore symlink and extend to bigger files

c995ffb

wip: improve processor efficiency

19edc46

refactor: improve performance of processors

5b3bd8e

refactor: cleanup

2f83b76

fix: use correct encrypted field module name and update snapshots

7b7bbe1

didroe force-pushed the fix/processing-of-structure-sql branch from c4c6baa to 7b7bbe1 Compare December 13, 2022 10:07

didroe marked this pull request as ready for review December 13, 2022 10:07

cfabianski merged commit 480b746 into main Dec 13, 2022

cfabianski deleted the fix/processing-of-structure-sql branch December 13, 2022 10:20