OSError: [Errno 24] Too many open files

[ohyeah521]

[ERROR] 2023-09-01 09:54:34 - libsast scan
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "D:\venv\lib\site-packages\libsast\core_matcher\pattern_matcher.py", line 90, in pattern_matcher
data = file_path.read_text('utf-8', 'ignore')
File "C:\Python3\lib\pathlib.py", line 1266, in read_text
with self.open(mode='r', encoding=encoding, errors=errors) as f:
File "C:\Python3\lib\pathlib.py", line 1252, in open
return io.open(self, mode, buffering, encoding, errors, newline,
File "C:\Python3\lib\pathlib.py", line 1120, in _opener
return self._accessor.open(self, flags, mode)
OSError: [Errno 24] Too many open files: 'M:\uploads\b1436c59b24ebc3ab8632f24aaa16cb4\java_source\com\taobao\login4android\membercenter\security\SecurityEntranceActivity.java'

res = scanner.scan()

File "D:\venv\lib\site-packages\libsast\scanner.py", line 58, in scan
results['pattern_matcher'] = PatternMatcher(
File "D:\venv\lib\site-packages\libsast\core_matcher\pattern_matcher.py", line 58, in scan
results = pool.map(
File "C:\Python3\lib\multiprocessing\pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\Python3\lib\multiprocessing\pool.py", line 771, in get
raise self._value
libsast.exceptions.RuleProcessingError: Rule processing error.

[ajinabraham]

This looks like a resource limiter exception from OSError: [Errno 24] Too many open files:
Might be Windows specific. I will cut a new release were you can reduce the number of multiprocessing pool via environment variable.

[ajinabraham]

Can you try out libsast 2.0.2 ?

In your Windows/Thread ripper setup, the workers are defaulted to minimum of 61.
Set an environment variable LIBSAST_WORKERS=10 and perform a scan.
If it works, you can increase the worker count until you see the OSError. The higher the worker count, the better the performance/speed of the scan. This environment variable allows you to use a worker count that gives you best performance without exceptions on windows.

Other wise, use something like WSL for Windows and set a higher ulimit for libsast scans.

[ohyeah521]

I tested it, but it still doesn't work properly

[ajinabraham]

Have you tired with different values for LIBSAST_WORKERS ?

[ohyeah521]

Sorry, I wrote the wrong environment variable when testing libsast 2.0.2 yesterday. After I corrected it, it was able to work normally. libsast 2.0.3 work properly.

[ohyeah521]

This error occurs again when using version 3.1.6 under Windows: [Errno 24] Too many open files