Alexander Motin a41ef36858 DDT: Reduce global DDT lock scope during writes ... Before this change DDT lock was taken 4 times per written block, and as effectively a pool-wide lock it can be highly congested. This change introduces a new per-entry dde_io_lock, protecting some fields during I/O ready and done stages, so that we don't need the global lock there. According to my write tests on 64-thread system with 4KB blocks this significantly reduce the global lock contention, reducing CPU usage from 100% to expected ~80%, and increasing write throughput by 10%. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Rob Norris <robn@despairlabs.com> Signed-off-by: Alexander Motin <alexander.motin@TrueNAS.com> Closes #17960		2025-12-10 10:21:29 -08:00
..
abd.c	Prefer VERIFY0(n) over VERIFY3U(n, ==, 0)	2025-08-07 11:41:25 -07:00
aggsum.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
arc.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
blake3_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
blkptr.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bplist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
bpobj.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
bptree.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
bqueue.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
brt.c	Add BRT support to zpool prefetch command	2025-11-12 13:07:09 -08:00
btree.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dataset_kstats.c	ZIL: "crash" the ZIL if the pool suspends during fallback	2025-08-08 16:43:26 -07:00
dbuf_stats.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
dbuf.c	Fix caching of DDT log and BRT	2025-11-12 13:05:25 -08:00
ddt_log.c	Fix caching of DDT log and BRT	2025-11-12 13:05:25 -08:00
ddt_stats.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
ddt_zap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
ddt.c	DDT: Reduce global DDT lock scope during writes	2025-12-10 10:21:29 -08:00
dmu_diff.c	Allow physical rewrite without logical	2025-08-06 10:36:07 -07:00
dmu_direct.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dmu_object.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dmu_objset.c	dnode: add dn_dirtycnt, count of number of txgs this dnode is dirty on	2025-08-21 06:05:29 -07:00
dmu_recv.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dmu_redact.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
dmu_send.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dmu_traverse.c	Allow physical rewrite without logical	2025-08-06 10:36:07 -07:00
dmu_tx.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
dmu_zfetch.c	Wire O_DIRECT also to Uncached I/O (#17218)	2025-05-13 14:26:55 -07:00
dmu.c	Add BRT support to zpool prefetch command	2025-11-12 13:07:09 -08:00
dnode_sync.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dnode.c	Update dnode_next_offset_level to accept blkid instead of offset	2025-11-12 13:05:40 -08:00
dsl_bookmark.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dsl_crypt.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dsl_dataset.c	Prefer VERIFY0P(n) over VERIFY(n == NULL)	2025-08-07 11:41:37 -07:00
dsl_deadlist.c	Fix missed assertion update in physical rewrite patch	2025-08-13 15:56:25 -04:00
dsl_deleg.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
dsl_destroy.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dsl_dir.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
dsl_pool.c	Prefer VERIFY0P(n) over VERIFY(n == NULL)	2025-08-07 11:41:37 -07:00
dsl_prop.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
dsl_scan.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
dsl_synctask.c	dmu_tx_assign: make all VERIFY0 calls use DMU_TX_SUSPEND	2025-05-28 10:28:59 -07:00
dsl_userhold.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
edonr_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
fm.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
gzip.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
hkdf.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
lz4_zfs.c	SPDX: license tags: BSD-2-Clause	2025-03-13 17:56:46 -07:00
lz4.c	SPDX: license tags: BSD-2-Clause	2025-03-13 17:56:46 -07:00
lzjb.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
metaslab.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
mmp.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
multilist.c	Allow vmem_alloc backed multilists	2025-08-12 13:36:03 -07:00
objlist.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
pathname.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
range_tree.c	range_tree: drop duplicate zfs_ prefix from rs_set_fill_raw	2025-09-29 16:50:53 -07:00
refcount.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
rrwlock.c	Prefer VERIFY0P(n) over VERIFY(n == NULL)	2025-08-07 11:41:37 -07:00
sa.c	Prefer VERIFY0(n) over VERIFY3U(n, ==, 0)	2025-08-07 11:41:25 -07:00
sha2_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
skein_zfs.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_checkpoint.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
spa_config.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
spa_errlog.c	Allow physical rewrite without logical	2025-08-06 10:36:07 -07:00
spa_history.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
spa_log_spacemap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
spa_misc.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
spa_stats.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
spa.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
space_map.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
space_reftree.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
THIRDPARTYLICENSE.cityhash	OpenZFS 8484 - Implement aggregate sum and use for arc counters	2018-06-06 09:35:59 -07:00
THIRDPARTYLICENSE.cityhash.descrip	OpenZFS 8484 - Implement aggregate sum and use for arc counters	2018-06-06 09:35:59 -07:00
txg.c	txg_wait_synced_flags: add TXG_WAIT_SUSPEND flag to not wait if pool suspended	2025-05-28 10:27:46 -07:00
uberblock.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
unique.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_draid_rand.c	SPDX: license tags: LicenseRef-OpenZFS-ThirdParty-PublicDomain	2025-03-13 17:57:31 -07:00
vdev_draid.c	Detect a slow raidz child during reads	2025-09-10 15:31:30 -07:00
vdev_file.c	Enable zhack to work properly with 4k sector size disks	2025-09-10 15:01:32 -07:00
vdev_indirect_births.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
vdev_indirect_mapping.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
vdev_indirect.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
vdev_initialize.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
vdev_label.c	Cleanup ZIO_FLAG_IO_RETRY vs TRYHARD usage	2025-11-12 13:05:31 -08:00
vdev_mirror.c	Allow physical rewrite without logical	2025-08-06 10:36:07 -07:00
vdev_missing.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_queue.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
vdev_raidz_math_aarch64_neon_common.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_aarch64_neon.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_aarch64_neonx2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx512bw.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_avx512f.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_impl.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_powerpc_altivec_common.h	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_powerpc_altivec.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_scalar.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_sse2.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math_ssse3.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
vdev_raidz_math.c	tunables: don't assert initialisation in impl getters	2025-05-28 16:50:22 -07:00
vdev_raidz.c	Fix typo in vdev_raidz.c	2025-11-12 13:06:19 -08:00
vdev_rebuild.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
vdev_removal.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
vdev_root.c	Implement allocation size ranges and use for gang leaves (#17111)	2025-05-02 15:32:18 -07:00
vdev_trim.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
vdev.c	Add knob to disable slow io notifications	2025-11-12 13:07:14 -08:00
zap_leaf.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zap_micro.c	ZAP: Remove dmu_object_info_from_dnode() call	2025-11-12 13:07:00 -08:00
zap.c	Prefer VERIFY0(n) over VERIFY3U(n, ==, 0)	2025-08-07 11:41:25 -07:00
zcp_get.c	zcp: get_prop: fix encryptionroot and encryption	2025-05-27 20:04:37 -04:00
zcp_global.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_iter.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_set.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zcp_synctask.c	zcp_synctask: add zfs.sync.clone()	2025-06-10 14:53:10 -07:00
zcp.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
zfeature.c	Synchronize the update of feature refcount	2025-09-09 17:03:27 -07:00
zfs_byteswap.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_chksum.c	chksum: run 256K benchmark on demand, preserve chksum_stat_data	2025-12-10 10:21:29 -08:00
zfs_crrd.c	Fix time database update calculations	2025-09-15 12:43:34 -07:00
zfs_debug_common.c	nvlist: Add nvlist_snprintf() and zfs_dbgmsg_nvlist()	2025-04-18 09:22:16 -04:00
zfs_fm.c	Add knob to disable slow io notifications	2025-11-12 13:07:14 -08:00
zfs_fuid.c	Pass flags to more DMU write/hold functions	2025-11-12 13:04:58 -08:00
zfs_impl.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_ioctl.c	Add BRT support to zpool prefetch command	2025-11-12 13:07:09 -08:00
zfs_log.c	ZIL: pass commit errors back to ITX callbacks	2025-08-08 16:43:20 -07:00
zfs_onexit.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_quota.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
zfs_ratelimit.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zfs_replay.c	dmu_tx: rename dmu_tx_assign() flags from TXG_* to DMU_TX_* (#17143)	2025-03-18 16:04:22 -07:00
zfs_rlock.c	Prefer VERIFY0(n) over VERIFY3U(n, ==, 0)	2025-08-07 11:41:25 -07:00
zfs_sa.c	ZIL: allow zil_commit() to fail with error	2025-08-08 16:43:09 -07:00
zfs_vnops.c	ZIL: allow zil_commit() to fail with error	2025-08-08 16:43:09 -07:00
zfs_znode.c	Add default user/group/project quota properties	2025-04-03 10:35:22 -07:00
zil.c	ZIL: Make allocations more flexible	2025-08-14 08:50:17 -07:00
zio_checksum.c	Prefer VERIFY0(n) over VERIFY(n == 0)	2025-08-07 11:40:59 -07:00
zio_compress.c	Removed unused zio_decompress_fail_fraction variable	2025-08-06 17:10:03 -07:00
zio_inject.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00
zio.c	DDT: Reduce global DDT lock scope during writes	2025-12-10 10:21:29 -08:00
zle.c	SPDX: license tags: CDDL-1.0	2025-03-13 17:56:27 -07:00
zrlock.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
zthr.c	Prefer VERIFY0P(n) over VERIFY3P(n, ==, NULL)	2025-08-07 11:41:42 -07:00
zvol.c	spa_misc: add an API for spa_namespace_lock	2025-11-12 13:06:54 -08:00