Change how the ansible-doc thread pooling works #89
Labels
documentation
Improvements or additions to documentation
enhancement
New feature or request
Partial
Some of the work for this has been merged
The configuration for threads currently allow 128 concurrent workers but with the ansible-doc parser we have significantly less than that. The reason is that we need to keep the worker count at or below 128 but in ansible-doc we're currently allocating workers twice. First, we allocate workers to process each of the 13 documentable plugin types. Then, inside of each of those 13, we spawn more workers to invoke ansible-doc and get the plugin documentation.
Because of that structure, we don't know how many workers we can allocate to each plugin type so we just divide them evenly between them. 128/13 is roughly 10 threads for each plugin type. Since modules vastly outnumber the other plugins, this means we spend the majority of the time with 10 threads waiting on ansible-doc IO after the other plugins have finished.
The right way to fix this is to restructure the work so that all the workers are allocated from the same place. That way they'll all just end up being allocated as one of the 128 threads becomes free. As a stopgap, we could simply give modules 70% or so of the threads and divide the remaining threads evenly amongst the other plugin types.
The text was updated successfully, but these errors were encountered: