c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2024-12-26 03:09:34 +03:00
Code Issues Packages Projects Releases Wiki Activity
99573cc053
mirror_zfs/module/zfs
History
Prakash Surya 99573cc053 Timeout waiting for ZVOL device to be created 
We've seen cases where after creating a ZVOL, the ZVOL device node in
"/dev" isn't generated after 20 seconds of waiting, which is the point
at which our applications gives up on waiting and reports an error.

The workload when this occurs is to "refresh" 400+ ZVOLs roughly at the
same time, based on a policy set by the user. This refresh operation
will destroy the ZVOL, and re-create it based on a snapshot.

When this occurs, we see many hundreds of entries on the "z_zvol" taskq
(based on inspection of the /proc/spl/taskq-all file). Many of the
entries on the taskq end up in the "zvol_remove_minors_impl" function,
and I've measured the latency of that function:

Function = zvol_remove_minors_impl
msecs               : count     distribution
  0 -> 1          : 0        |                                        |
  2 -> 3          : 0        |                                        |
  4 -> 7          : 1        |                                        |
  8 -> 15         : 0        |                                        |
 16 -> 31         : 0        |                                        |
 32 -> 63         : 0        |                                        |
 64 -> 127        : 1        |                                        |
128 -> 255        : 45       |****************************************|
256 -> 511        : 5        |****                                    |

That data is from a 10 second sample, using the BCC "funclatency" tool.
As we can see, in this 10 second sample, most calls took 128ms at a
minimum. Thus, some basic math tells us that in any 20 second interval,
we could only process at most about 150 removals, which is much less
than the 400+ that'll occur based on the workload.

As a result of this, and since all ZVOL minor operations will go through
the single threaded "z_zvol" taskq, the latency for creating a single
ZVOL device can be unreasonably large due to other ZVOL activity on the
system. In our case, it's large enough to cause the application to
generate an error and fail the operation.

When profiling the "zvol_remove_minors_impl" function, I saw that most
of the time in the function was spent off-cpu, blocked in the function
"taskq_wait_outstanding". How this works, is "zvol_remove_minors_impl"
will dispatch calls to "zvol_free" using the "system_taskq", and then
the "taskq_wait_outstanding" function is used to wait for all of those
dispatched calls to occur before "zvol_remove_minors_impl" will return.

As far as I can tell, "zvol_remove_minors_impl" doesn't necessarily have
to wait for all calls to "zvol_free" to occur before it returns. Thus,
this change removes the call to "taskq_wait_oustanding", so that calls
to "zvol_free" don't affect the latency of "zvol_remove_minors_impl".

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: John Gallagher <john.gallagher@delphix.com>
Signed-off-by: Prakash Surya <prakash.surya@delphix.com>
Closes #9380
2019-10-01 12:33:12 -07:00
..
aggsum.c OpenZFS 9688 - aggsum_fini leaks memory 2018-10-19 12:08:03 -07:00
arc.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
blkptr.c Undo c89 workarounds to match with upstream 2017-11-04 13:25:13 -07:00
bplist.c Fast Clone Deletion 2019-07-26 10:54:14 -07:00
bpobj.c Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
bptree.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
bqueue.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
cityhash.c OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
dataset_kstats.c Fix panic on DilOS with kstat per dataset statistics 2019-09-03 12:12:31 -07:00
dbuf_stats.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dbuf.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
ddt_zap.c fat zap should prefetch when iterating 2019-06-12 13:13:09 -07:00
ddt.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dmu_diff.c diff_cb() does not handle large dnodes 2019-09-24 12:01:37 -07:00
dmu_object.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dmu_objset.c OpenZFS restructuring - zvol 2019-09-25 09:20:30 -07:00
dmu_recv.c Enable compiler to typecheck logging 2019-09-12 13:28:26 -07:00
dmu_redact.c Move objnode handling to common code 2019-09-12 13:31:09 -07:00
dmu_send.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dmu_traverse.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dmu_tx.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
dmu_zfetch.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dmu.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
dnode_sync.c Decrease contention on dn_struct_rwlock 2019-07-08 13:18:50 -07:00
dnode.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
dsl_bookmark.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
dsl_crypt.c Fix clone handling with encryption roots 2019-09-16 10:07:33 -07:00
dsl_dataset.c Enable compiler to typecheck logging 2019-09-12 13:28:26 -07:00
dsl_deadlist.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
dsl_deleg.c Update build system and packaging 2018-05-29 16:00:33 -07:00
dsl_destroy.c Enable compiler to typecheck logging 2019-09-12 13:28:26 -07:00
dsl_dir.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
dsl_pool.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
dsl_prop.c Update build system and packaging 2018-05-29 16:00:33 -07:00
dsl_scan.c Disabled resilver_defer feature leads to looping resilvers 2019-09-22 15:25:39 -07:00
dsl_synctask.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
dsl_userhold.c Enable compiler to typecheck logging 2019-09-12 13:28:26 -07:00
edonr_zfs.c DLPX-44812 integrate EP-220 large memory scalability 2016-11-29 14:34:27 -08:00
fm.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
gzip.c OpenZFS restructuring - move platform specific sources 2019-09-06 11:26:26 -07:00
hkdf.c Encryption patch follow-up 2017-10-11 16:54:48 -04:00
lz4.c Reword comment in lz4_compress_zfs 2019-05-02 16:46:04 -07:00
lzjb.c Change KM_PUSHPAGE -> KM_SLEEP 2015-01-16 14:41:26 -08:00
Makefile.in OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
metaslab.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
mmp.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
multilist.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
objlist.c Implement Redacted Send/Receive 2019-06-19 09:48:12 -07:00
pathname.c Disable unused pathname::pn_path* (unneeded in Linux) 2019-07-15 13:57:56 -07:00
range_tree.c Metaslab max_size should be persisted while unloaded 2019-08-05 14:34:27 -07:00
refcount.c Prevent race in blkptr_verify against device removal 2019-08-13 21:24:43 -06:00
rrwlock.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
sa.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
sha256.c OpenZFS restructuring - move platform specific sources 2019-09-06 11:26:26 -07:00
skein_zfs.c DLPX-44812 integrate EP-220 large memory scalability 2016-11-29 14:34:27 -08:00
spa_boot.c Add linux kernel module support 2010-08-31 13:41:58 -07:00
spa_checkpoint.c Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
spa_config.c Fix /etc/hostid on root pool deadlock 2019-09-10 13:42:30 -07:00
spa_errlog.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
spa_history.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
spa_log_spacemap.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
spa_misc.c Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
spa.c Prevent gcc -Werror=maybe-uninitialized warnings in spa_wait_common() 2019-09-16 10:46:02 -07:00
space_map.c Log Spacemap Project 2019-07-16 10:11:49 -07:00
space_reftree.c OpenZFS 7614, 9064 - zfs device evacuation/removal 2018-04-14 12:16:17 -07:00
THIRDPARTYLICENSE.cityhash OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
THIRDPARTYLICENSE.cityhash.descrip OpenZFS 8484 - Implement aggregate sum and use for arc counters 2018-06-06 09:35:59 -07:00
txg.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
uberblock.c MMP interval and fail_intervals in uberblock 2019-03-21 12:47:57 -07:00
unique.c Performance optimization of AVL tree comparator functions 2016-08-31 14:35:34 -07:00
vdev_cache.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev_indirect_births.c Fixes: #8934 Large kmem_alloc 2019-07-10 15:54:49 -07:00
vdev_indirect_mapping.c Get rid of space_map_update() for ms_synced_length 2019-02-12 10:38:11 -08:00
vdev_indirect.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev_initialize.c Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
vdev_label.c panic in removal_remap test on 4K devices 2019-06-13 13:12:39 -07:00
vdev_mirror.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev_missing.c Update vdev_ops_t from illumos 2019-06-20 18:29:02 -07:00
vdev_queue.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev_raidz_math_aarch64_neon_common.h OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math_aarch64_neon.c Linux 5.0 compat: SIMD compatibility 2019-07-12 09:31:20 -07:00
vdev_raidz_math_aarch64_neonx2.c Linux 5.0 compat: SIMD compatibility 2019-07-12 09:31:20 -07:00
vdev_raidz_math_avx2.c OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math_avx512bw.c OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math_avx512f.c OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math_impl.h codebase style improvements for OpenZFS 6459 port 2017-01-22 13:25:40 -08:00
vdev_raidz_math_scalar.c Linux 5.3: Fix switch() fall though compiler errors 2019-08-21 09:29:23 -07:00
vdev_raidz_math_sse2.c OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math_ssse3.c OpenZFS restructuring - move platform specific headers 2019-09-05 09:34:54 -07:00
vdev_raidz_math.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev_raidz.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
vdev_removal.c Device removal of indirect vdev panics the kernel 2019-09-16 10:46:59 -07:00
vdev_root.c Update vdev_ops_t from illumos 2019-06-20 18:29:02 -07:00
vdev_trim.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
vdev.c Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
zap_leaf.c Off-by-one in zap_leaf_array_create() 2019-01-18 09:58:46 -08:00
zap_micro.c fat zap should prefetch when iterating 2019-06-12 13:13:09 -07:00
zap.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
zcp_get.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zcp_global.c OpenZFS 8600 - ZFS channel programs - snapshot 2018-02-08 15:29:24 -08:00
zcp_iter.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zcp_synctask.c OpenZFS 9166 - zfs storage pool checkpoint 2018-06-26 10:07:42 -07:00
zcp.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
zfeature.c Consistently captialize GUID for features 2019-04-16 10:01:51 -07:00
zfs_byteswap.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zfs_fm.c Add zpool status -s (slow I/Os) and -p (parseable) 2018-11-08 16:47:24 -08:00
zfs_fuid.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_ioctl.c OpenZFS restructuring - zfs_ioctl 2019-09-27 10:46:28 -07:00
zfs_log.c Make module tunables cross platform 2019-09-05 14:49:49 -07:00
zfs_onexit.c Update build system and packaging 2018-05-29 16:00:33 -07:00
zfs_ratelimit.c Change checksum & IO delay ratelimit values 2018-03-04 17:34:51 -08:00
zfs_replay.c Fix zil replay panic when TX_REMOVE followed by TX_CREATE 2019-08-28 10:42:02 -07:00
zfs_rlock.c OpenZFS 9689 - zfs range lock code should not be zpl-specific 2018-10-11 10:19:33 -07:00
zfs_sa.c Project Quota on ZFS 2018-02-13 14:54:54 -08:00
zil.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
zio_checksum.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zio_compress.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zio_inject.c Fix typos in module/zfs/ 2019-09-02 17:56:41 -07:00
zio.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
zle.c Fix zle_decompress out of bound access 2018-02-09 10:08:05 -08:00
zrlock.c OpenZFS restructuring - move linux tracing code to platform directories 2019-09-11 14:25:53 -07:00
zthr.c Fast Clone Deletion 2019-07-26 10:54:14 -07:00
zvol.c Timeout waiting for ZVOL device to be created 2019-10-01 12:33:12 -07:00