c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2026-05-22 02:27:36 +03:00
Code Issues Packages Projects Releases Wiki Activity

looping in metaslab_block_picker impacts performance on fragmented pools

Browse Source 
On fragmented pools with high-performance storage, the looping in
metaslab_block_picker() can become the performance-limiting bottleneck.
When looking for a larger block (e.g. a 128K block for the ZIL), we may
search through many free segments (up to hundreds of thousands) to find
one that is large enough to satisfy the allocation. This can take a long
time (up to dozens of ms), and is done while holding the ms_lock, which
other threads may spin waiting for.

When this performance problem is encountered, profiling will show
high CPU time in metaslab_block_picker, as well as in mutex_enter from
various callers.

The problem is very evident on a test system with a sync write workload
with 8K writes to a recordsize=8k filesystem, with 4TB of SSD storage,
84% full and 88% fragmented. It has also been observed on production
systems with 90TB of storage, 76% full and 87% fragmented.

The fix is to change metaslab_df_alloc() to search only up to 16MB from
the previous allocation (of this alignment). After that, we will pick a
segment that is of the exact size requested (or larger). This reduces
the number of iterations to a few hundred on fragmented pools (a ~100x
improvement).

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Tony Nguyen <tony.nguyen@delphix.com>
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Serapheim Dimitropoulos <serapheim@delphix.com>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
External-issue: DLPX-62324
Closes #8877
This commit is contained in:
Matthew Ahrens
2019-06-13 13:06:15 -07:00
committed by Tony Hutter
parent 8805abb8fc
commit cbb9154958
2 changed files with 117 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
man/man5/zfs-module-parameters.5
+34
View File
@@ -325,6 +325,40 @@ Enable use of the fragmentation metric in computing metaslab weights.
Use \fB1\fR for yes (default) and \fB0\fR for no.
.RE
.sp
.ne 2
.na
\fBmetaslab_df_max_search\fR (int)
.ad
.RS 12n
Maximum distance to search forward from the last offset. Without this limit,
fragmented pools can see >100,000 iterations and metaslab_block_picker()
becomes the performance limiting factor on high-performance storage.
With the default setting of 16MB, we typically see less than 500 iterations,
even with very fragmented, ashift=9 pools. The maximum number of iterations
possible is: \fBmetaslab_df_max_search / (2 * (1<<ashift))\fR.
With the default setting of 16MB this is 16*1024 (with ashift=9) or 2048
(with ashift=12).
.sp
Default value: \fB16,777,216\fR (16MB)
.RE
.sp
.ne 2
.na
\fBmetaslab_df_use_largest_segment\fR (int)
.ad
.RS 12n
If we are not searching forward (due to metaslab_df_max_search,
metaslab_df_free_pct, or metaslab_df_alloc_threshold), this tunable controls
what segment is used. If it is set, we will use the largest free segment.
If it is not set, we will use a segment of exactly the requested size (or
larger).
.sp
Use \fB1\fR for yes and \fB0\fR for no (default).
.RE
.sp
.ne 2
.na