optimize various bits around scan profiles #4050
Conversation
|
Further improvements: |
There was a problem hiding this comment.
Nice work! It's clever to do all_declared_scan_profiles and assigned_scan_levels in one pass. There could be one step more for little optimization, and that is by creating source_scan_profile_references after the loop. You gain a little performance improvement because it's faster to setup a set on an existing list (or any sequence or iterable) than constructing it element by element via set.add (e.g. previous implementation source_scan_profile_references = {sp.reference for sp in all_declared_scan_profiles})
Example of benchmark code below.
import timeit
code1 = """
x = set()
l = []
for i in range(50000):
l.append(i)
x.add(i)
"""
print(timeit.timeit(code1, number=2000)) # 2.3204293749295175
code2 = """
l = [i for i in range(50000)]
x = set([i for i in l])
"""
print(timeit.timeit(code2, number=2000)) # 2.087212207959965But the performance gain is negligible and the suggested implementation is already fast and understandable, so I'll leave it up to you.
Yes, I was pondering this too, as set operations 'one by one' are slower than doing an update, or mass assignment. There's a bunch of other optimizations that can be done, but I'm focussing on not running the entire loop if its needed first, as thats the main win that would safe 90% or more for most installs. Furthermore Im guessing all of this will be undone once the nibbles start taking care of propagation, as they do all this taint tracking in a much more efficient way. |
Co-authored-by: Jeroen Dekkers <[email protected]>
underdarknl
changed the title
|
the tests are now failing on ill-formatted test-references. |
Checklist for QA:
What works:I think it works. Did identify some 'weird' things in the propagation, but those seem to already be on main. What doesn't work:n/a Bug or feature?:n/a |
|
Changes
I noticed we do a periodic recalculate_scan_profiles, which by itself loops over the list of profiles three times.
QA notes
Scan profiles should still be handled the exact same way. The code is functionally the same as before.
Code Checklist
.envchanges files if required and changed the.env-distaccordingly.Checklist for code reviewers:
Copy-paste the checklist from the docs/source/templates folder into your comment.
Checklist for QA:
Copy-paste the checklist from the docs/source/templates folder into your comment.