mirror_zfs/module/zfs
Alexander Motin dc91a6a660 Several B-tree optimizations
- Introduce first element offset within a leaf.  It allows to reduce
by ~50% average memmove() size when adding/removing elements.  If the
added/removed element is in the first half of the leaf, we may shift
elements before it and adjust the bth_first instead of moving more
elements after it.
 - Use memcpy() instead of memmove() when we know there is no overlap.
 - Switch from uint64_t to uint32_t.  It does not limit anything,
but 32-bit arches should appreciate it greatly in hot paths.
 - Store leaf capacity in struct btree to avoid 64-bit divisions.
 - Adjust zfs_btree_insert_into_leaf() to always result in balanced
leaves after splitting, no matter where the new element was inserted.
Not that we care about it much, but it should also allow B-trees with
as little as two elements per leaf instead of 4 previously.

When scrubbing pool of 12 SSDs, storing 1.5TB of 4KB zvol blocks this
reduces amount of time spent in memmove() inside the scan thread from
13.7% to 5.7% and total scrub time by ~15 seconds out of 9 minutes.
It should also reduce spacemaps load time, but I haven't measured it.

Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #13582
2022-07-26 10:10:37 -07:00
..
abd.c Avoid small buffer copying on write 2022-07-26 10:10:37 -07:00
aggsum.c More aggsum optimizations 2021-06-09 13:05:34 -07:00
arc.c Avoid small buffer copying on write 2022-07-26 10:10:37 -07:00
blkptr.c Add zstd support to zfs 2020-08-20 10:30:06 -07:00
bplist.c
bpobj.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
bptree.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
bqueue.c
btree.c Several B-tree optimizations 2022-07-26 10:10:37 -07:00
dataset_kstats.c Introduce write-mostly sums 2021-06-09 13:05:34 -07:00
dbuf_stats.c Reduce dbuf_find() lock contention 2022-05-06 12:02:45 -07:00
dbuf.c More speculative prefetcher improvements 2022-07-26 10:10:37 -07:00
ddt_zap.c Refactor dnode dirty context from dbuf_dirty 2020-02-26 16:09:17 -08:00
ddt.c Tinker with slop space accounting with dedup 2021-09-14 12:38:05 -07:00
dmu_diff.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dmu_object.c Introduce CPU_SEQID_UNSTABLE 2020-11-02 11:51:12 -08:00
dmu_objset.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dmu_recv.c Receive checks should allow unencrypted child datasets 2022-02-16 17:58:55 -08:00
dmu_redact.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dmu_send.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dmu_traverse.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dmu_tx.c Annotated dprintf as printf-like 2021-06-24 13:12:36 -07:00
dmu_zfetch.c More speculative prefetcher improvements 2022-07-26 10:10:37 -07:00
dmu.c Default to zfs_dmu_offset_next_sync=1 2022-04-01 09:59:47 -07:00
dnode_sync.c Report dnodes with faulty bonuslen 2022-02-16 17:58:55 -08:00
dnode.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dsl_bookmark.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dsl_crypt.c Introduce a flag to skip comparing the local mac when raw sending 2022-02-04 16:14:56 -08:00
dsl_dataset.c Remove unneeded "extern inline" function declarations 2022-02-16 17:58:56 -08:00
dsl_deadlist.c Livelist logic should handle dedup blkptrs 2021-06-09 13:05:34 -07:00
dsl_deleg.c
dsl_destroy.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dsl_dir.c Fix ENOSPC when unlinking multiple files from full pool 2022-03-08 11:46:03 -08:00
dsl_pool.c Fix ENOSPC when unlinking multiple files from full pool 2022-03-08 11:46:03 -08:00
dsl_prop.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dsl_scan.c Several sorted scrub optimizations 2022-07-26 10:10:37 -07:00
dsl_synctask.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
dsl_userhold.c Replace sprintf()->snprintf() and strcpy()->strlcpy() 2020-06-07 11:42:12 -07:00
edonr_zfs.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
fm.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
gzip.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
hkdf.c
lz4.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
lzjb.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
Makefile.in Distributed Spare (dRAID) Feature 2020-11-13 13:51:51 -08:00
metaslab.c Improve mg_aliquot math 2022-07-26 10:10:37 -07:00
mmp.c Optimize small random numbers generation 2021-09-14 12:10:17 -07:00
multilist.c Optimize small random numbers generation 2021-09-14 12:10:17 -07:00
objlist.c
pathname.c Replace ZFS on Linux references with OpenZFS 2020-10-08 20:10:13 -07:00
range_tree.c Several sorted scrub optimizations 2022-07-26 10:10:37 -07:00
refcount.c Export minimal zfs_refcount interfaces 2022-04-06 10:29:00 -07:00
rrwlock.c Rename refcount.h to zfs_refcount.h 2020-07-29 16:35:33 -07:00
sa.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
sha256.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
skein_zfs.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
spa_boot.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
spa_checkpoint.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
spa_config.c Cleaning up uio headers 2021-02-20 20:16:50 -08:00
spa_errlog.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
spa_history.c Annotated dprintf as printf-like 2021-06-24 13:12:36 -07:00
spa_log_spacemap.c Improve log spacemap load time 2022-07-26 10:10:37 -07:00
spa_misc.c Remove refcount from spa_config_*() 2022-07-26 10:10:37 -07:00
spa_stats.c Remove pool io kstats 2021-06-10 10:50:16 -07:00
spa.c Improve log spacemap load time 2022-07-26 10:10:37 -07:00
space_map.c Optimize small random numbers generation 2021-09-14 12:10:17 -07:00
space_reftree.c
THIRDPARTYLICENSE.cityhash
THIRDPARTYLICENSE.cityhash.descrip
txg.c Annotated dprintf as printf-like 2021-06-24 13:12:36 -07:00
uberblock.c
unique.c
vdev_cache.c
vdev_draid_rand.c Distributed Spare (dRAID) Feature 2020-11-13 13:51:51 -08:00
vdev_draid.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
vdev_indirect_births.c
vdev_indirect_mapping.c
vdev_indirect.c module/zfs: vdev_indirect: vdev_indirect_repair: remove unused variable 2022-05-02 15:42:58 -07:00
vdev_initialize.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
vdev_label.c Use fallthrough macro 2021-11-02 09:50:30 -07:00
vdev_mirror.c Scrub mirror children without BPs 2022-07-14 10:21:29 -07:00
vdev_missing.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
vdev_queue.c Avoid vq_lock drop in vdev_queue_aggregate() 2021-09-14 14:31:22 -07:00
vdev_raidz_math_aarch64_neon_common.h FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_aarch64_neon.c
vdev_raidz_math_aarch64_neonx2.c
vdev_raidz_math_avx2.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_avx512bw.c Refactor ccompile.h to not include system headers 2020-07-25 20:09:50 -07:00
vdev_raidz_math_avx512f.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_impl.h Distributed Spare (dRAID) Feature 2020-11-13 13:51:51 -08:00
vdev_raidz_math_powerpc_altivec_common.h FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_powerpc_altivec.c Prefix zfs internal endian checks with _ZFS 2020-07-28 13:02:49 -07:00
vdev_raidz_math_scalar.c Use fallthrough macro 2021-11-02 09:50:30 -07:00
vdev_raidz_math_sse2.c FreeBSD: fix the build with Clang 11 2020-08-17 15:40:17 -07:00
vdev_raidz_math_ssse3.c Refactor ccompile.h to not include system headers 2020-07-25 20:09:50 -07:00
vdev_raidz_math.c Initialize parity blocks before RAID-Z reconstruction benchmarking 2021-09-14 14:32:16 -07:00
vdev_raidz.c Scrub mirror children without BPs 2022-07-14 10:21:29 -07:00
vdev_rebuild.c Use dsl_scan_setup_check() to setup a scrub 2021-04-14 13:19:49 -07:00
vdev_removal.c Improve log spacemap load time 2022-07-26 10:10:37 -07:00
vdev_root.c Distributed Spare (dRAID) Feature 2020-11-13 13:51:51 -08:00
vdev_trim.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
vdev.c Improve log spacemap load time 2022-07-26 10:10:37 -07:00
zap_leaf.c Remove unneeded "extern inline" function declarations 2022-02-16 17:58:56 -08:00
zap_micro.c Remove unneeded "extern inline" function declarations 2022-02-16 17:58:56 -08:00
zap.c Remove unneeded "extern inline" function declarations 2022-02-16 17:58:56 -08:00
zcp_get.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
zcp_global.c
zcp_iter.c
zcp_set.c Support setting user properties in a channel program 2020-02-14 13:41:42 -08:00
zcp_synctask.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
zcp.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
zfeature.c Throw const on some strings 2020-10-02 17:44:10 -07:00
zfs_byteswap.c Mark functions as static 2020-06-18 12:20:38 -07:00
zfs_fm.c fm: remove unused variables 2022-05-02 15:42:58 -07:00
zfs_fuid.c Fix regression in POSIX mode behavior 2021-03-19 22:50:46 -07:00
zfs_ioctl.c file reference counts can get corrupted 2021-09-14 12:37:38 -07:00
zfs_log.c Initialize all fields in zfs_log_xvattr() 2021-09-14 12:42:21 -07:00
zfs_onexit.c file reference counts can get corrupted 2021-09-14 12:37:38 -07:00
zfs_quota.c File incorrectly zeroed when receiving incremental stream that toggles -L 2020-06-09 10:41:01 -07:00
zfs_ratelimit.c
zfs_replay.c Use fallthrough macro 2021-11-02 09:50:30 -07:00
zfs_rlock.c Add a "try" operation for range locks 2020-07-06 11:53:31 -07:00
zfs_sa.c Extending FreeBSD UIO Struct 2021-01-20 21:27:30 -08:00
zfs_vnops.c Fix clearing set-uid and set-gid bits on a file when replying a write 2022-02-16 17:58:55 -08:00
zil.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
zio_checksum.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
zio_compress.c module: zfs: fix unused, remove argsused 2022-02-16 17:58:56 -08:00
zio_inject.c Optimize small random numbers generation 2021-09-14 12:10:17 -07:00
zio.c Enable encrypted raw sending to pools with greater ashift 2022-02-23 16:47:37 -08:00
zle.c Add include files for prototypes 2020-06-18 12:21:25 -07:00
zrlock.c Remove dead code 2020-06-18 12:21:18 -07:00
zthr.c Avoid memory allocations in the ARC eviction thread 2022-02-03 15:30:52 -08:00
zvol.c zvol: remove unused variable 2022-05-02 15:42:58 -07:00