Expand fragmentation table to reflect larger possibile allocation sizes

[pcd1193182]

Motivation and Context

When you are using large recordsizes in conjunction with raidz, with incompressible data, you can pretty reliably be making 21 MB allocations. Unfortunately, the fragmentation metric in ZFS considers any metaslabs with 16 MB free chunks completely unfragmented, so you can have a metaslab report 0% fragmented and be unable to satisfy an allocation. When using the segment-based metaslab weight, this is inconvenient; when using the space-based one, it can seriously degrade performance.

Description

We expand the fragmentation table to extend up to 1GB, and redefine the table size based on the actual table, rather than having a static define. We also tweak the one variable that depends on fragmentation directly.

The one caveat for this change is that on pools with small disks (less than 200GB), once a metaslab is dirtied at all it will always report fragmented. This is because at our target of 200 metaslabs, the whole metaslab is less than a gigabyte, so the largest possible free segment is less than a gigabyte. This may result in some user questions, but most users probably don't have disks that small installed. At larger sizes, the problem disappears. Users may note an increase in fragmentation when this change is released, but it doesn't actually reflect any on-disk changes, just a new measurement scale.

Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.

How Has This Been Tested?

Basic sanity testing only; passes the zfs test suite and zloop, and reports fragmentation correctly.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

Do you have any particular motivation to go as high as 1GB? IIRC 16MB is pretty hard block limit for ZFS now that is not going to change (soon). Sure you've shown that 16MB may be not enough, and free ranges close to it might not represent no fragmentation, since close sized allocations may produce significant amount of smaller fragments, exposing the hidden fragmentation. But I think those effects should rapidly diminish and could be neglected somewhere about 64-128MB. Also looking on logic of vdev_metaslab_set_size() it seems 512MB is the lowest metaslab size for the most of cases, which makes 128MB also a sweet spot to allow almost empty metaslabs to remain non-fragmented.