mirror_zfs/tests/zfs-tests/tests/functional
Michael Niewöhner 10b3c7f5e4 Add zstd support to zfs
This PR adds two new compression types, based on ZStandard:

- zstd: A basic ZStandard compression algorithm Available compression.
  Levels for zstd are zstd-1 through zstd-19, where the compression
  increases with every level, but speed decreases.

- zstd-fast: A faster version of the ZStandard compression algorithm
  zstd-fast is basically a "negative" level of zstd. The compression
  decreases with every level, but speed increases.

  Available compression levels for zstd-fast:
   - zstd-fast-1 through zstd-fast-10
   - zstd-fast-20 through zstd-fast-100 (in increments of 10)
   - zstd-fast-500 and zstd-fast-1000

For more information check the man page.

Implementation details:

Rather than treat each level of zstd as a different algorithm (as was
done historically with gzip), the block pointer `enum zio_compress`
value is simply zstd for all levels, including zstd-fast, since they all
use the same decompression function.

The compress= property (a 64bit unsigned integer) uses the lower 7 bits
to store the compression algorithm (matching the number of bits used in
a block pointer, as the 8th bit was borrowed for embedded block
pointers).  The upper bits are used to store the compression level.

It is necessary to be able to determine what compression level was used
when later reading a block back, so the concept used in LZ4, where the
first 32bits of the on-disk value are the size of the compressed data
(since the allocation is rounded up to the nearest ashift), was
extended, and we store the version of ZSTD and the level as well as the
compressed size. This value is returned when decompressing a block, so
that if the block needs to be recompressed (L2ARC, nop-write, etc), that
the same parameters will be used to result in the matching checksum.

All of the internal ZFS code ( `arc_buf_hdr_t`, `objset_t`,
`zio_prop_t`, etc.) uses the separated _compress and _complevel
variables.  Only the properties ZAP contains the combined/bit-shifted
value. The combined value is split when the compression_changed_cb()
callback is called, and sets both objset members (os_compress and
os_complevel).

The userspace tools all use the combined/bit-shifted value.

Additional notes:

zdb can now also decode the ZSTD compression header (flag -Z) and
inspect the size, version and compression level saved in that header.
For each record, if it is ZSTD compressed, the parameters of the decoded
compression header get printed.

ZSTD is included with all current tests and new tests are added
as-needed.

Per-dataset feature flags now get activated when the property is set.
If a compression algorithm requires a feature flag, zfs activates the
feature when the property is set, rather than waiting for the first
block to be born.  This is currently only used by zstd but can be
extended as needed.

Portions-Sponsored-By: The FreeBSD Foundation
Co-authored-by: Allan Jude <allanjude@freebsd.org>
Co-authored-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Co-authored-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Co-authored-by: Michael Niewöhner <foss@mniewoehner.de>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allanjude@freebsd.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Sebastian Gottschall <s.gottschall@dd-wrt.com>
Signed-off-by: Kjeld Schouten-Lebbing <kjeld@schouten-lebbing.nl>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #6247
Closes #9024
Closes #10277
Closes #10278
2020-08-20 10:30:06 -07:00
..
acl Fix typos in tests/ 2019-09-02 18:14:53 -07:00
alloc_class ZTS: minor improvements to alloc_class_009_pos functional test 2020-07-30 09:11:05 -07:00
arc ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
atime ZTS: Misc fixes for FreeBSD 2020-02-25 16:23:27 -08:00
bootfs Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
btree ZTS: Fix ksh path in btree tests 2020-01-15 16:23:29 -08:00
cache ZTS: Fix and change testcase cache_010_neg 2020-04-13 10:50:41 -07:00
cachefile Fix typos in tests/ 2019-09-02 18:14:53 -07:00
casenorm ZTS: Adapt casenorm tests for FreeBSD 2020-02-26 08:41:30 -08:00
channel_program Fix lua stack overflow on recursive call to gsub() 2020-07-27 16:11:47 -07:00
chattr Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
checksum Use abs_top_builddir when referencing libraries 2020-07-10 14:26:32 -07:00
clean_mirror ZTS: clean_mirror and scrub_mirror cleanup 2018-07-09 12:46:14 -07:00
cli_root Add zstd support to zfs 2020-08-20 10:30:06 -07:00
cli_user ZTS: Misc fixes for FreeBSD 2020-02-25 16:23:27 -08:00
compression Add zstd support to zfs 2020-08-20 10:30:06 -07:00
cp_files Fix ENOSPC in "Handle zap_add() failures in ..." 2018-04-18 14:19:50 -07:00
ctime Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
deadman ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
delegate Clarify comments in config/*.m4, vdev_geom.c, zfs_allow_*.ksh 2020-06-22 09:46:37 -07:00
devices ZTS: devices_001_pos and devices_002_neg 2019-12-27 12:11:27 -08:00
events Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
exec Enable remaining tests 2017-05-22 12:34:32 -04:00
fallocate linux: add basic fallocate(mode=0/2) compatibility 2020-06-18 11:22:11 -07:00
fault ZED: Do not offline a missing device if no spare is available 2020-08-18 22:13:17 -07:00
features ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
grow ZTS: Adopt OpenZFS test analysis script 2018-06-20 14:03:13 -07:00
history Add zstd support to zfs 2020-08-20 10:30:06 -07:00
hkdf Use abs_top_builddir when referencing libraries 2020-07-10 14:26:32 -07:00
inheritance Fixes for make dist 2020-06-26 14:20:02 -07:00
inuse ZTS: Eliminate partitioning from zpool_create etc 2020-02-20 08:10:13 -08:00
io Direct IO support 2018-08-27 10:04:21 -07:00
large_files ZTS: Another round of changes for FreeBSD 2020-03-06 09:31:32 -08:00
largest_pool Fix typos 2020-06-09 21:24:09 -07:00
libzfs Use abs_top_builddir when referencing libraries 2020-07-10 14:26:32 -07:00
limits Linux 4.10 compat: has_capability() 2020-07-19 09:56:21 -07:00
link_count ZTS: Add zts-report exceptions for FreeBSD 2020-03-04 15:09:40 -08:00
log_spacemap ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
migration ZTS: Fix a few defaults 2020-01-31 08:51:23 -08:00
mmap Fix typos 2020-06-09 21:24:09 -07:00
mmp ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
mount ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
mv_files ZTS: Provide an alternative to shuf for FreeBSD 2020-01-09 09:31:17 -08:00
nestedfs OpenZFS 7290 - ZFS test suite needs to control what utilities it can run 2017-04-06 09:25:36 -07:00
no_space ZTS: Eliminate partitioning from zpool_create etc 2020-02-20 08:10:13 -08:00
nopwrite Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
online_offline Add subcommand to wait for background zfs activity to complete 2019-09-13 18:09:06 -07:00
pam pam: implement a zfs_key pam module 2020-06-24 18:45:44 -07:00
persist_l2arc Fix typos 2020-06-09 21:24:09 -07:00
pool_checkpoint ZTS: Misc test fixes for FreeBSD 2020-02-13 13:52:34 -08:00
pool_names Fix some ZFS Test Suite issues 2017-09-25 10:32:34 -07:00
poolversion Fix some ZFS Test Suite issues 2017-09-25 10:32:34 -07:00
privilege ZTS: Move privilege tests to sunos.run 2020-02-21 08:52:44 -08:00
procfs ZTS: Fix pool_state cleanup 2019-12-28 08:43:23 -08:00
projectquota Fix typos in tests/ 2019-09-02 18:08:56 -07:00
pyzfs Centralize variable substitution 2020-07-14 17:33:44 -07:00
quota ZTS: Wait for free space between quota tests 2020-03-26 10:48:19 -07:00
raidz Fix typos in tests/ 2019-09-02 18:08:56 -07:00
redacted_send Avoid core dump on invalid redaction bookmark 2020-03-18 12:54:12 -07:00
redundancy ZTS: Eliminate functions named 'random' 2020-01-08 09:08:30 -08:00
refquota ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
refreserv ZTS: refreserv_005_pos.ksh 2020-05-08 13:50:02 -07:00
removal ZTS: Wait for free space between quota tests 2020-03-26 10:48:19 -07:00
rename_dirs Enable remaining tests 2017-05-22 12:34:32 -04:00
replacement Add device rebuild feature 2020-07-03 11:05:50 -07:00
reservation Fix typos in tests/ 2019-08-30 16:52:00 -07:00
rootpool Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
rsend Add support to decode a resume token 2020-07-23 17:44:03 -07:00
scrub_mirror ZTS: clean_mirror and scrub_mirror cleanup 2018-07-09 12:46:14 -07:00
slog ZTS: Fix faulty slog_replay_fs_001 test 2020-02-20 08:11:51 -08:00
snapshot ZTS: Catalog tunable names for tests in tunables.cfg 2020-01-14 14:57:28 -08:00
snapused Fix "file is executable, but no shebang" warnings 2018-04-06 16:34:21 -07:00
sparse ZTS: Adopt OpenZFS test analysis script 2018-06-20 14:03:13 -07:00
suid Implement secpolicy_vnode_setid_retain() 2019-07-26 13:52:30 -07:00
threadsappend ZTS: Adopt OpenZFS test analysis script 2018-06-20 14:03:13 -07:00
tmpfile Mark functions as static 2020-06-18 12:20:38 -07:00
trim Trim L2ARC 2020-06-09 10:15:08 -07:00
truncate Update ZTS to work on FreeBSD 2019-12-18 12:29:43 -08:00
upgrade ZTS: Fix upgrade_readonly_pool 2019-10-03 09:39:13 -07:00
user_namespace Fix "file is executable, but no shebang" warnings 2018-04-06 16:34:21 -07:00
userquota ZTS: Fix userquota_006_pos on FreeBSD 2020-02-20 08:14:24 -08:00
vdev_zaps ZTS: Fix vdev_zaps_005_pos on CentOS 6 2019-08-22 08:53:44 -07:00
write_dirs Fix typos 2020-06-09 21:24:09 -07:00
xattr ZTS: Misc fixes for FreeBSD 2020-02-24 10:17:55 -08:00
zvol ZTS: Skip udev actions in zvol_misc when not Linux 2020-03-31 10:35:14 -07:00
Makefile.am Add device rebuild feature 2020-07-03 11:05:50 -07:00