spa: make read/write queues configurable

[robn]

Motivation and Context

We are finding that as customers get larger and faster machines (hundreds of cores, large NVMe-backed pools) they keep hitting relatively low performance ceilings. Our profiling work almost always finds that they're running into bottlenecks on the SPA IO taskqs. Unfortunately there's often little we can advise at that point, because there's very few ways to change behaviour without patching.

Description

This commit adds two load-time parameters zio_taskq_read and zio_taskq_write that can configure the READ and WRITE IO taskqs directly.

This achieves two goals: it gives operators (and those that support them) a way to tune things without requiring a custom build of OpenZFS, which is often not possible, and it lets us easily try different config variations in a variety of environments to inform the development of better defaults for these kind of systems.

Because tuning the IO taskqs really requires a fairly deep understanding of how IO in ZFS works, and generally isn't needed without a pretty serious workload and an ability to identify bottlenecks, only minimal documentation is provided. Its expected that anyone using this is going to have the source code there as well.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

Further discussion

I know there's not really much appetite for even more config parameters, but as it says in above, we keep running IO queue bottlenecks at customer sites, and most don't have a way to take a patch. This would at least let us offer them something.

I'm aware that this is exposing implementation details to a degree - its the point! To that end, I'd be more than happy somehow marking this that makes it clear the operator is taking matters into their own hand. Maybe you have to set a zfs_debug flag first, or maybe it logs "dragons enabled!", or something else? Your suggestion welcome.

How Has This Been Tested?

By hand, and with our internal performance test suite. The most important thing is that when not in use, nothing changes.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

I've often wanted to be able to easily experiment with exactly this kind of tuning, and adding one more module option is a reasonable way to handle it. If you can just fix up the CentOS 7 and FreeBSD 13 build failures we can move this forward.

[robn]

Alright, pending test results that should do it. CentOS 7 was the bug in old GCC where you have to initialise arrays of complex types with {{0}} instead of {0}. The FreeBSD one was that parameter code is not really generalised at all for PARAM_CALL parameters, so I had to learn how to write that, and then spent hours down a deep rabbithole discovering that (maybe) you there's no memory allocator available in early module load, so it all has to be done on the stack. So that's another thing on my list of someday cleanups, oh well.

(ignore me; just felt an afternoon of faffing about so close to Christmas deserved a paragraph 😅)