c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2026-06-10 07:56:39 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
f375b23c026aec00cc9527470084191b5071d9b2
mirror_zfs/include/sys
T
History
Rich Ercolani f375b23c02 Tiered early abort, zstd edition 
It turns out that "do LZ4 and zstd-1 both fail" is a great heuristic
for "don't even bother trying higher zstd tiers".

By way of illustration:
$ cat /incompress | mbuffer | zfs recv -o compression=zstd-12 evenfaster/lowcomp_1M_zstd12_normal
summary: 39.8 GiByte in  3min 40.2sec - average of  185 MiB/s
$ echo 3 | sudo tee /sys/module/zzstd/parameters/zstd_lz4_pass
3
$ cat /incompress | mbuffer -m 4G | zfs recv -o compression=zstd-12 evenfaster/lowcomp_1M_zstd12_patched
summary: 39.8 GiByte in 48.6sec - average of  839 MiB/s
$ sudo zfs list -p -o name,used,lused,ratio evenfaster/lowcomp_1M_zstd12_normal evenfaster/lowcomp_1M_zstd12_patched
NAME                                         USED        LUSED  RATIO
evenfaster/lowcomp_1M_zstd12_normal   39549931520  42721221632   1.08
evenfaster/lowcomp_1M_zstd12_patched  39626399744  42721217536   1.07
$ python3 -c "print(39626399744 - 39549931520)"
76468224
$

I'll take 76 MB out of 42 GB for > 4x speedup.

Reviewed-by: Allan Jude <allan@klarasystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: George Melikov <mail@gmelikov.ru>
Reviewed-by: Kjeld Schouten <kjeld@schouten-lebbing.nl>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Closes #13244
2022-05-24 09:43:22 -07:00
..
crypto
autoconf: single-step includes
2022-05-10 10:18:51 -07:00
fm
autoconf: single-step includes
2022-05-10 10:18:51 -07:00
fs
linux: libzfs: simplify module-loaded check
2022-05-18 12:51:42 -07:00
lua
autoconf: single-step includes
2022-05-10 10:18:51 -07:00
sysevent
autoconf: single-step includes
2022-05-10 10:18:51 -07:00
zstd
Tiered early abort, zstd edition
2022-05-24 09:43:22 -07:00
abd_impl.h
Avoid small buffer copying on write
2021-07-27 16:05:47 -07:00
abd.h
Avoid small buffer copying on write
2021-07-27 16:05:47 -07:00
aggsum.h
More aggsum optimizations
2021-06-07 09:02:47 -07:00
arc_impl.h
Replace *CTASSERT() with _Static_assert()
2022-01-26 11:38:52 -08:00
arc.h
include: sys/arc.h: shim out arc_referenced()
2021-12-23 09:36:26 -08:00
avl_impl.h
Remove avl_size field from struct avl_tree
2021-07-01 09:32:31 -06:00
avl.h
lib{efi,avl,share,tpool,zfs_core,zfsbootenv,zutil}: -fvisibility=hidden
2021-06-09 17:04:32 -07:00
bitops.h
Remove NOTE(CONSTCOND) and note.h
2021-07-26 12:07:53 -07:00
blkptr.h
OpenZFS 8067 - zdb should be able to dump literal embedded block pointer
2017-07-07 11:28:01 -07:00
bplist.h
Fast Clone Deletion
2019-07-26 10:54:14 -07:00
bpobj.h
Fast Clone Deletion
2019-07-26 10:54:14 -07:00
bptree.h
Illumos 4914 - zfs on-disk bookmark structure should be named *_phys_t
2014-08-06 14:48:41 -07:00
bqueue.h
Implement Redacted Send/Receive
2019-06-19 09:48:12 -07:00
btree.h
Fix typos
2020-06-09 21:24:09 -07:00
dataset_kstats.h
Introduce write-mostly sums
2021-05-27 14:27:29 -06:00
dbuf.h
Reduce dbuf_find() lock contention
2022-05-04 11:17:29 -07:00
ddt.h
Appease GCC sprintf warnings found on Fedora 32/GCC 10.0.1
2020-08-24 10:32:59 -07:00
dmu_impl.h
Remove UIO_ZEROCOPY functions structures
2020-10-30 10:00:33 -07:00
dmu_objset.h
Introduce a tunable to exclude special class buffers from L2ARC
2021-11-11 12:52:16 -08:00
dmu_recv.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
dmu_redact.h
Suppress cppcheck invalidSyntax warninigs
2021-03-05 17:56:35 -08:00
dmu_send.h
Removed duplicated includes
2021-03-22 12:34:58 -07:00
dmu_traverse.h
Implement Redacted Send/Receive
2019-06-19 09:48:12 -07:00
dmu_tx.h
Add Module Parameter Regarding Log Size Limit
2021-07-20 09:40:24 -06:00
dmu_zfetch.h
Split dmu_zfetch() speculation and execution parts
2021-03-19 22:56:11 -07:00
dmu.h
Improve log spacemap load time
2022-04-26 10:44:21 -07:00
dnode.h
Fix lseek(SEEK_DATA/SEEK_HOLE) mmap consistency
2021-11-07 14:27:44 -07:00
dsl_bookmark.h
Rename refcount.h to zfs_refcount.h
2020-07-29 16:35:33 -07:00
dsl_crypt.h
dmu_objset_from_ds must be called with dp_config_rwlock held
2020-03-12 10:55:02 -07:00
dsl_dataset.h
Improve zpool status output, list all affected datasets
2022-04-25 17:25:42 -07:00
dsl_deadlist.h
Add fast path for zfs_ioc_space_snaps() handling of empty_bpobj
2019-08-20 11:34:52 -07:00
dsl_deleg.h
Remove code for zfs remap
2019-06-24 16:44:01 -07:00
dsl_destroy.h
Fast Clone Deletion
2019-07-26 10:54:14 -07:00
dsl_dir.h
Remove NOTE(CONSTCOND) and note.h
2021-07-26 12:07:53 -07:00
dsl_pool.h
Fix ENOSPC when unlinking multiple files from full pool
2022-03-08 09:16:35 -08:00
dsl_prop.h
Support inheriting properties in channel programs
2020-01-22 17:03:17 -08:00
dsl_scan.h
Use dsl_scan_setup_check() to setup a scrub
2021-04-08 14:33:15 -07:00
dsl_synctask.h
Add upper bound for slop space calculation
2021-02-24 09:52:43 -08:00
dsl_userhold.h
Illumos #3740
2013-11-04 11:17:48 -08:00
edonr.h
OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R
2016-10-03 14:51:15 -07:00
efi_partition.h
Integrate carcass of libspl/i/s/vtoc.h into i/s/efi_partition.h
2022-03-15 15:13:54 -07:00
frame.h
Linux 5.10 compat: frame.h renamed objtool.h
2020-11-02 22:01:10 +00:00
hkdf.h
Encryption patch follow-up
2017-10-11 16:54:48 -04:00
metaslab_impl.h
Improve log spacemap load time
2022-04-26 10:44:21 -07:00
metaslab.h
Improve log spacemap load time
2022-04-26 10:44:21 -07:00
mmp.h
Add zfs_multihost_interval tunable handler for FreeBSD
2020-06-23 13:32:42 -07:00
mntent.h
Add FreeBSD required defines to mntent.h
2019-11-30 15:49:09 -08:00
mod.h
linux: module: weld all but spl.ko into zfs.ko
2022-04-20 13:28:24 -07:00
multilist.h
Re-embed multilist_t storage
2021-06-10 10:42:31 -06:00
nvpair_impl.h
Add const to nvlist functions to properly expose their real behavior
2021-12-06 18:19:13 -07:00
nvpair.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
objlist.h
Implement Redacted Send/Receive
2019-06-19 09:48:12 -07:00
pathname.h
Replace ZFS on Linux references with OpenZFS
2020-10-08 20:10:13 -07:00
qat.h
include: qat.h: mark unused macro arguments as used
2021-12-23 09:36:21 -08:00
range_tree.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
rrwlock.h
Rename refcount.h to zfs_refcount.h
2020-07-29 16:35:33 -07:00
sa_impl.h
Rename refcount.h to zfs_refcount.h
2020-07-29 16:35:33 -07:00
sa.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
skein.h
OpenZFS 4185 - add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R
2016-10-03 14:51:15 -07:00
spa_boot.h
Support custom build directories and move includes
2010-09-08 12:38:56 -07:00
spa_checkpoint.h
Serialize ZTHR operations to eliminate races
2019-01-13 10:09:46 -08:00
spa_checksum.h
Implementation of AVX2 optimized Fletcher-4
2016-06-02 14:30:51 -07:00
spa_impl.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
spa_log_spacemap.h
Improve log spacemap load time
2022-04-26 10:44:21 -07:00
spa.h
Expose zpool guids through kstats
2022-05-18 10:25:33 -07:00
space_map.h
Extend zdb to print inconsistencies in livelists and metaslabs
2020-07-14 17:51:05 -07:00
space_reftree.h
Reduce loaded range tree memory usage
2019-10-09 10:36:03 -07:00
sysevent.h
OpenZFS 6939 - add sysevents to zfs core for commands
2017-07-12 21:28:13 -07:00
txg_impl.h
Fix typos in include/
2019-08-30 09:53:15 -07:00
txg.h
Optimize txg_kick() process (#12274)
2021-07-01 09:20:27 -06:00
u8_textprep_data.h
Support custom build directories and move includes
2010-09-08 12:38:56 -07:00
u8_textprep.h
Throw const on some strings
2020-10-02 17:44:10 -07:00
uberblock_impl.h
MMP interval and fail_intervals in uberblock
2019-03-21 12:47:57 -07:00
uberblock.h
Multi-modifier protection (MMP)
2017-07-13 13:54:00 -04:00
uio_impl.h
Cleaning up uio headers
2021-02-20 20:16:50 -08:00
unique.h
Illumos #3742
2013-11-04 10:55:25 -08:00
uuid.h
Support custom build directories and move includes
2010-09-08 12:38:56 -07:00
vdev_disk.h
Make struct vdev_disk_t be platform private
2020-06-16 11:43:33 -07:00
vdev_draid.h
Verify dRAID empty sectors
2022-01-04 16:46:32 -08:00
vdev_file.h
Add zfs_file_* interface, remove vnodes
2019-11-21 09:32:57 -08:00
vdev_impl.h
Replace *CTASSERT() with _Static_assert()
2022-01-26 11:38:52 -08:00
vdev_indirect_births.h
OpenZFS 7614, 9064 - zfs device evacuation/removal
2018-04-14 12:16:17 -07:00
vdev_indirect_mapping.h
OpenZFS 7614, 9064 - zfs device evacuation/removal
2018-04-14 12:16:17 -07:00
vdev_initialize.h
Add TRIM support
2019-03-29 09:13:20 -07:00
vdev_raidz_impl.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
vdev_raidz.h
Verify dRAID empty sectors
2022-01-04 16:46:32 -08:00
vdev_rebuild.h
Fix various typos
2021-04-02 18:52:15 -07:00
vdev_removal.h
panic in removal_remap test on 4K devices
2019-06-13 13:12:39 -07:00
vdev_trim.h
Trim L2ARC
2020-06-09 10:15:08 -07:00
vdev.h
Vdev Properties Feature
2021-11-30 07:46:25 -07:00
xvattr.h
Forbid b{copy,zero,cmp}(). Don't include <strings.h> for <string.h>
2022-03-15 15:13:48 -07:00
zap_impl.h
OpenZFS 7793 - ztest fails assertion in dmu_tx_willuse_space
2017-03-07 09:51:59 -08:00
zap_leaf.h
Fix ENOSPC in "Handle zap_add() failures in ..."
2018-04-18 14:19:50 -07:00
zap.h
fat zap should prefetch when iterating
2019-06-12 13:13:09 -07:00
zcp_global.h
OpenZFS 7431 - ZFS Channel Programs
2018-02-08 15:28:18 -08:00
zcp_iter.h
OpenZFS 7431 - ZFS Channel Programs
2018-02-08 15:28:18 -08:00
zcp_prop.h
OpenZFS 7431 - ZFS Channel Programs
2018-02-08 15:28:18 -08:00
zcp_set.h
Support setting user properties in a channel program
2020-02-14 13:41:42 -08:00
zcp.h
filesystem_limit/snapshot_limit is incorrectly enforced against root
2020-07-11 17:18:02 -07:00
zfeature.h
Revert "zhack: Add 'feature disable' command"
2016-05-17 11:52:07 -07:00
zfs_acl.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
zfs_bootenv.h
zfs label bootenv should store data as nvlist
2020-09-15 15:42:27 -07:00
zfs_context.h
Remove hw_serial, ddi_strtoul()
2022-05-13 10:15:31 -07:00
zfs_debug.h
Annotated dprintf as printf-like
2021-06-22 21:53:45 -07:00
zfs_delay.h
Update build system and packaging
2018-05-29 16:00:33 -07:00
zfs_file.h
file reference counts can get corrupted
2021-07-10 19:00:37 -06:00
zfs_fuid.h
Replace sprintf()->snprintf() and strcpy()->strlcpy()
2020-06-07 11:42:12 -07:00
zfs_ioctl_impl.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
zfs_ioctl.h
file reference counts can get corrupted
2021-07-10 19:00:37 -06:00
zfs_onexit.h
file reference counts can get corrupted
2021-07-10 19:00:37 -06:00
zfs_project.h
Minor diff reduction with ZoF in include/sys
2019-11-27 11:11:03 -08:00
zfs_quota.h
File incorrectly zeroed when receiving incremental stream that toggles -L
2020-06-09 10:41:01 -07:00
zfs_racct.h
Restore FreeBSD resource usage accounting
2021-02-19 22:34:33 -08:00
zfs_ratelimit.h
Change checksum & IO delay ratelimit values
2018-03-04 17:34:51 -08:00
zfs_refcount.h
Correct refcount_add in dmu_zfetch
2021-10-08 11:10:34 -07:00
zfs_rlock.h
Add a "try" operation for range locks
2020-07-06 11:53:31 -07:00
zfs_sa.h
log xattr=sa create/remove/update to ZIL
2022-02-22 13:06:43 -08:00
zfs_stat.h
Support custom build directories and move includes
2010-09-08 12:38:56 -07:00
zfs_sysfs.h
zcommon: pre-iterate over sysfs instead of statting every feature
2021-12-16 16:43:10 -08:00
zfs_vfsops.h
Add 'zfs rename -u' to rename without remounting
2020-09-01 16:14:16 -07:00
zfs_vnops.h
Extending FreeBSD UIO Struct
2021-01-20 21:27:30 -08:00
zfs_znode.h
Speed up WB_SYNC_NONE when a WB_SYNC_ALL occurs simultaneously
2022-05-03 13:23:26 -07:00
zil_impl.h
make zil max block size tunable
2019-06-10 11:48:42 -07:00
zil.h
log xattr=sa create/remove/update to ZIL
2022-02-22 13:06:43 -08:00
zio_checksum.h
libzfs: convert to -fvisibility=hidden
2021-06-03 13:17:55 -07:00
zio_compress.h
Add zstd support to zfs
2020-08-20 10:30:06 -07:00
zio_crypt.h
module/*.ko: prune .data, global .rodata
2022-01-14 15:37:55 -08:00
zio_impl.h
Add zstd support to zfs
2020-08-20 10:30:06 -07:00
zio_priority.h
Add device rebuild feature
2020-07-03 11:05:50 -07:00
zio.h
Improve zpool status output, list all affected datasets
2022-04-25 17:25:42 -07:00
zrlock.h
OpenZFS 6328 - Fix cstyle errors in zfs codebase
2017-01-12 09:42:11 -08:00
zthr.h
Avoid memory allocations in the ARC eviction thread
2022-01-21 10:28:13 -08:00
zvol_impl.h
zvol: make calls to platform ops static
2022-02-07 10:24:38 -08:00
zvol.h
Remove unused function zvol_set_volblocksize()
2021-10-26 17:07:53 -07:00