Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve RAIDZ expansion performance #15680

Closed
TurkeyMan opened this issue Dec 16, 2023 · 0 comments · Fixed by #16819
Closed

Improve RAIDZ expansion performance #15680

TurkeyMan opened this issue Dec 16, 2023 · 0 comments · Fixed by #16819
Labels
Type: Feature Feature request or new feature

Comments

@TurkeyMan
Copy link

Describe the feature would like to see added to OpenZFS

Performance of RAIDZ expansion could be improved.

How will this feature improve OpenZFS?

Currently the RAIDZ expansion performance is very slow and it can take several days to add a single large drive.
Improving expansion performance would result in a shorter period of degraded FS performance for users.
Expanding by mote than a single drive is not currently supported, so to add a few new drives to a pool can take weeks.

Additional context

There was discussion on this matter during development of the RAIDZ expansion feature where it was noted that performance is very slow, and the opinion of the developer was that it wouldn't be difficult to improve after the feature has landed.
Maybe it could be considered an item for some near-term follow-up on the landing of the new feature?
@TurkeyMan TurkeyMan added the Type: Feature Feature request or new feature label Dec 16, 2023
@behlendorf behlendorf mentioned this issue Dec 3, 2024
Merged
13 tasks
behlendorf pushed a commit to behlendorf/zfs that referenced this issue Dec 6, 2024
@amotin @behlendorf
Optimize RAIDZ expansion
7742e29 
- Instead of copying one ashift-sized block per ZIO, copy as much
as we have contiguous data up to 16MB per old vdev.  To avoid data
moves use gang ABDs, so that read ZIOs can directly fill buffers
for write ZIOs.  ABDs have much smaller overhead than ZIOs in both
memory usage and processing time, plus big I/Os do not depend on
I/O aggregation and scheduling to reach decent performance on HDDs.
 - Reduce raidz_expand_max_copy_bytes to 16MB on 32bit platforms.
 - Use 32bit range tree when possible (practically always now) to
slightly reduce memory usage.
 - Use ZIO_PRIORITY_REMOVAL for early stages of expansion, same as
for main ones.
 - Fix rate overflows in `zpool status` reporting.

With these changes expanding RAIDZ1 from 4 to 5 children I am able
to reach 6-12GB/s rate on SSDs and ~500MB/s on HDDs, both are
limited by devices instead of CPU.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by:	Alexander Motin <[email protected]>
Sponsored by:	iXsystems, Inc.
Closes openzfs#15680
Closes openzfs#16819
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Feature Feature request or new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optimize RAIDZ expansion amotin/zfs
1 participant
@TurkeyMan