mirror_zfs/tests/zfs-tests/include/properties.shlib
Tino Reichardt 985c33b132
Introduce BLAKE3 checksums as an OpenZFS feature
This commit adds BLAKE3 checksums to OpenZFS, it has similar
performance to Edon-R, but without the caveats around the latter.

Homepage of BLAKE3: https://github.com/BLAKE3-team/BLAKE3
Wikipedia: https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE3

Short description of Wikipedia:

  BLAKE3 is a cryptographic hash function based on Bao and BLAKE2,
  created by Jack O'Connor, Jean-Philippe Aumasson, Samuel Neves, and
  Zooko Wilcox-O'Hearn. It was announced on January 9, 2020, at Real
  World Crypto. BLAKE3 is a single algorithm with many desirable
  features (parallelism, XOF, KDF, PRF and MAC), in contrast to BLAKE
  and BLAKE2, which are algorithm families with multiple variants.
  BLAKE3 has a binary tree structure, so it supports a practically
  unlimited degree of parallelism (both SIMD and multithreading) given
  enough input. The official Rust and C implementations are
  dual-licensed as public domain (CC0) and the Apache License.

Along with adding the BLAKE3 hash into the OpenZFS infrastructure a
new benchmarking file called chksum_bench was introduced.  When read
it reports the speed of the available checksum functions.

On Linux: cat /proc/spl/kstat/zfs/chksum_bench
On FreeBSD: sysctl kstat.zfs.misc.chksum_bench

This is an example output of an i3-1005G1 test system with Debian 11:

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1196    1602    1761    1749    1762    1759    1751
skein-generic      546     591     608     615     619     612     616
sha256-generic     240     300     316     314     304     285     276
sha512-generic     353     441     467     476     472     467     426
blake3-generic     308     313     313     313     312     313     312
blake3-sse2        402    1289    1423    1446    1432    1458    1413
blake3-sse41       427    1470    1625    1704    1679    1607    1629
blake3-avx2        428    1920    3095    3343    3356    3318    3204
blake3-avx512      473    2687    4905    5836    5844    5643    5374

Output on Debian 5.10.0-10-amd64 system: (Ryzen 7 5800X)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1840    2458    2665    2719    2711    2723    2693
skein-generic      870     966     996     992    1003    1005    1009
sha256-generic     415     442     453     455     457     457     457
sha512-generic     608     690     711     718     719     720     721
blake3-generic     301     313     311     309     309     310     310
blake3-sse2        343    1865    2124    2188    2180    2181    2186
blake3-sse41       364    2091    2396    2509    2463    2482    2488
blake3-avx2        365    2590    4399    4971    4915    4802    4764

Output on Debian 5.10.0-9-powerpc64le system: (POWER 9)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1213    1703    1889    1918    1957    1902    1907
skein-generic      434     492     520     522     511     525     525
sha256-generic     167     183     187     188     188     187     188
sha512-generic     186     216     222     221     225     224     224
blake3-generic     153     152     154     153     151     153     153
blake3-sse2        391    1170    1366    1406    1428    1426    1414
blake3-sse41       352    1049    1212    1174    1262    1258    1259

Output on Debian 5.10.0-11-arm64 system: (Pi400)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic      487     603     629     639     643     641     641
skein-generic      271     299     303     308     309     309     307
sha256-generic     117     127     128     130     130     129     130
sha512-generic     145     165     170     172     173     174     175
blake3-generic      81      29      71      89      89      89      89
blake3-sse2        112     323     368     379     380     371     374
blake3-sse41       101     315     357     368     369     364     360

Structurally, the new code is mainly split into these parts:
- 1x cross platform generic c variant: blake3_generic.c
- 4x assembly for X86-64 (SSE2, SSE4.1, AVX2, AVX512)
- 2x assembly for ARMv8 (NEON converted from SSE2)
- 2x assembly for PPC64-LE (POWER8 converted from SSE2)
- one file for switching between the implementations

Note the PPC64 assembly requires the VSX instruction set and the
kfpu_begin() / kfpu_end() calls on PowerPC were updated accordingly.

Reviewed-by: Felix Dörre <felix@dogcraft.de>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Co-authored-by: Rich Ercolani <rincebrain@gmail.com>
Closes #10058
Closes #12918
2022-06-08 15:55:57 -07:00

130 lines
3.5 KiB
Plaintext

#
# This file and its contents are supplied under the terms of the
# Common Development and Distribution License ("CDDL"), version 1.0.
# You may only use this file in accordance with the terms of version
# 1.0 of the CDDL.
#
# A full copy of the text of the CDDL should have accompanied this
# source. A copy of the CDDL is also available via the Internet at
# http://www.illumos.org/license/CDDL.
#
#
# Copyright (c) 2012, 2016, Delphix. All rights reserved.
#
. $STF_SUITE/include/libtest.shlib
typeset -a compress_prop_vals=('off' 'lzjb' 'lz4' 'gzip' 'zle' 'zstd')
typeset -a checksum_prop_vals=('on' 'off' 'fletcher2' 'fletcher4' 'sha256'
'noparity' 'sha512' 'skein' 'blake3')
if ! is_freebsd; then
checksum_prop_vals+=('edonr')
fi
typeset -a recsize_prop_vals=('512' '1024' '2048' '4096' '8192' '16384'
'32768' '65536' '131072' '262144' '524288' '1048576')
typeset -a canmount_prop_vals=('on' 'off' 'noauto')
typeset -a copies_prop_vals=('1' '2' '3')
typeset -a logbias_prop_vals=('latency' 'throughput')
typeset -a primarycache_prop_vals=('all' 'none' 'metadata')
typeset -a redundant_metadata_prop_vals=('all' 'most')
typeset -a secondarycache_prop_vals=('all' 'none' 'metadata')
typeset -a snapdir_prop_vals=('hidden' 'visible')
typeset -a sync_prop_vals=('standard' 'always' 'disabled')
typeset -a fs_props=('compress' 'checksum' 'recsize'
'canmount' 'copies' 'logbias' 'primarycache' 'redundant_metadata'
'secondarycache' 'snapdir' 'sync')
typeset -a vol_props=('compress' 'checksum' 'copies' 'logbias' 'primarycache'
'secondarycache' 'redundant_metadata' 'sync')
#
# Given the 'prop' passed in, return 'num_vals' elements of the corresponding
# values array to the user, excluding any elements below 'first.' This allows
# us to exclude 'off' and 'on' which can be either unwanted, or a duplicate of
# another property respectively.
#
function get_rand_prop_vals
{
typeset prop=$1
typeset -i num_vals=$2
typeset -i first=$3
[[ -z $prop || -z $num_vals || -z $first ]] && \
log_fail "get_rand_prop_vals: bad arguments"
typeset retstr=""
typeset prop_vals_var=${prop}_prop_vals
typeset -a prop_vals=($(eval echo \${${prop_vals_var}[@]}))
[[ -z $prop_vals ]] && \
log_fail "get_rand_prop_vals: bad prop $prop"
typeset -i last=$((${#prop_vals[@]} - 1))
typeset -i i
for i in $(range_shuffle $first $last | head -n $num_vals); do
retstr="${prop_vals[$i]} $retstr"
done
echo $retstr
}
#
# Functions to toggle on/off properties
#
typeset -a binary_props=('atime' 'devices' 'exec' 'readonly' 'setuid' 'xattr')
if is_freebsd; then
binary_props+=('jailed')
else
binary_props+=('zoned')
fi
# Newer Linuxes dropped non-blocking mandatory locks
if ! is_linux || [ $(linux_version) -lt $(linux_version "4.4") ]; then
binary_props+=('nbmand')
fi
function toggle_prop
{
typeset ds=$1
typeset prop=$2
typeset val=$(get_prop $prop $ds)
typeset newval='off'
[[ $val = $newval ]] && newval='on'
log_must zfs set $prop=$newval $ds
}
function toggle_binary_props
{
typeset ds=$1
typeset prop
for prop in "${binary_props[@]}"; do
toggle_prop $ds $prop
done
}
function randomize_ds_props
{
typeset ds=$1
typeset prop proplist val
if ds_is_volume $ds; then
toggle_prop $ds readonly
proplist="${vol_props[@]}"
elif ds_is_filesystem $ds; then
toggle_binary_props $ds
proplist="${fs_props[@]}"
else
log_fail "$ds is neither a volume nor a file system"
fi
for prop in $proplist; do
typeset val=$(get_rand_prop_vals $prop 1 0)
log_must zfs set $prop=$val $ds
done
}