c/mirror_zfs
1
0
Fork 0
mirror of https://git.proxmox.com/git/mirror_zfs.git synced 2024-11-17 01:51:00 +03:00
Code Issues Packages Projects Releases Wiki Activity

Introduce BLAKE3 checksums as an OpenZFS feature

Browse Source 
This commit adds BLAKE3 checksums to OpenZFS, it has similar
performance to Edon-R, but without the caveats around the latter.

Homepage of BLAKE3: https://github.com/BLAKE3-team/BLAKE3
Wikipedia: https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE3

Short description of Wikipedia:

  BLAKE3 is a cryptographic hash function based on Bao and BLAKE2,
  created by Jack O'Connor, Jean-Philippe Aumasson, Samuel Neves, and
  Zooko Wilcox-O'Hearn. It was announced on January 9, 2020, at Real
  World Crypto. BLAKE3 is a single algorithm with many desirable
  features (parallelism, XOF, KDF, PRF and MAC), in contrast to BLAKE
  and BLAKE2, which are algorithm families with multiple variants.
  BLAKE3 has a binary tree structure, so it supports a practically
  unlimited degree of parallelism (both SIMD and multithreading) given
  enough input. The official Rust and C implementations are
  dual-licensed as public domain (CC0) and the Apache License.

Along with adding the BLAKE3 hash into the OpenZFS infrastructure a
new benchmarking file called chksum_bench was introduced.  When read
it reports the speed of the available checksum functions.

On Linux: cat /proc/spl/kstat/zfs/chksum_bench
On FreeBSD: sysctl kstat.zfs.misc.chksum_bench

This is an example output of an i3-1005G1 test system with Debian 11:

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1196    1602    1761    1749    1762    1759    1751
skein-generic      546     591     608     615     619     612     616
sha256-generic     240     300     316     314     304     285     276
sha512-generic     353     441     467     476     472     467     426
blake3-generic     308     313     313     313     312     313     312
blake3-sse2        402    1289    1423    1446    1432    1458    1413
blake3-sse41       427    1470    1625    1704    1679    1607    1629
blake3-avx2        428    1920    3095    3343    3356    3318    3204
blake3-avx512      473    2687    4905    5836    5844    5643    5374

Output on Debian 5.10.0-10-amd64 system: (Ryzen 7 5800X)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1840    2458    2665    2719    2711    2723    2693
skein-generic      870     966     996     992    1003    1005    1009
sha256-generic     415     442     453     455     457     457     457
sha512-generic     608     690     711     718     719     720     721
blake3-generic     301     313     311     309     309     310     310
blake3-sse2        343    1865    2124    2188    2180    2181    2186
blake3-sse41       364    2091    2396    2509    2463    2482    2488
blake3-avx2        365    2590    4399    4971    4915    4802    4764

Output on Debian 5.10.0-9-powerpc64le system: (POWER 9)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic     1213    1703    1889    1918    1957    1902    1907
skein-generic      434     492     520     522     511     525     525
sha256-generic     167     183     187     188     188     187     188
sha512-generic     186     216     222     221     225     224     224
blake3-generic     153     152     154     153     151     153     153
blake3-sse2        391    1170    1366    1406    1428    1426    1414
blake3-sse41       352    1049    1212    1174    1262    1258    1259

Output on Debian 5.10.0-11-arm64 system: (Pi400)

implementation      1k      4k     16k     64k    256k      1m      4m
edonr-generic      487     603     629     639     643     641     641
skein-generic      271     299     303     308     309     309     307
sha256-generic     117     127     128     130     130     129     130
sha512-generic     145     165     170     172     173     174     175
blake3-generic      81      29      71      89      89      89      89
blake3-sse2        112     323     368     379     380     371     374
blake3-sse41       101     315     357     368     369     364     360

Structurally, the new code is mainly split into these parts:
- 1x cross platform generic c variant: blake3_generic.c
- 4x assembly for X86-64 (SSE2, SSE4.1, AVX2, AVX512)
- 2x assembly for ARMv8 (NEON converted from SSE2)
- 2x assembly for PPC64-LE (POWER8 converted from SSE2)
- one file for switching between the implementations

Note the PPC64 assembly requires the VSX instruction set and the
kfpu_begin() / kfpu_end() calls on PowerPC were updated accordingly.

Reviewed-by: Felix Dörre <felix@dogcraft.de>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tino Reichardt <milky-zfs@mcmilk.de>
Co-authored-by: Rich Ercolani <rincebrain@gmail.com>
Closes #10058
Closes #12918
This commit is contained in:
Tino Reichardt 2022-06-09 00:55:57 +02:00 committed by GitHub
parent b9d98453f9
commit 985c33b132
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
53 changed files with 22804 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
AUTHORS
View File

@ -285,6 +285,7 @@ CONTRIBUTORS:
Tim Connors <tconnors@rather.puzzling.org>
Tim Crawford <tcrawford@datto.com>
Tim Haley <Tim.Haley@Sun.COM>
Tino Reichardt <milky-zfs@mcmilk.de>
Tobin Harding <me@tobin.cc>
Tom Caputi <tcaputi@datto.com>
Tom Matthews <tom@axiom-partners.com>

89
cmd/ztest.c
View File

@ -121,6 +121,7 @@
#include <sys/zfeature.h>
#include <sys/dsl_userhold.h>
#include <sys/abd.h>
#include <sys/blake3.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
@ -417,6 +418,7 @@ ztest_func_t ztest_device_removal;
ztest_func_t ztest_spa_checkpoint_create_discard;
ztest_func_t ztest_initialize;
ztest_func_t ztest_trim;
ztest_func_t ztest_blake3;
ztest_func_t ztest_fletcher;
ztest_func_t ztest_fletcher_incr;
ztest_func_t ztest_verify_dnode_bt;
@ -470,6 +472,7 @@ ztest_info_t ztest_info[] = {
ZTI_INIT(ztest_spa_checkpoint_create_discard, 1, &zopt_rarely),
ZTI_INIT(ztest_initialize, 1, &zopt_sometimes),
ZTI_INIT(ztest_trim, 1, &zopt_sometimes),
ZTI_INIT(ztest_blake3, 1, &zopt_rarely),
ZTI_INIT(ztest_fletcher, 1, &zopt_rarely),
ZTI_INIT(ztest_fletcher_incr, 1, &zopt_rarely),
ZTI_INIT(ztest_verify_dnode_bt, 1, &zopt_sometimes),
@ -6373,6 +6376,92 @@ ztest_reguid(ztest_ds_t *zd, uint64_t id)
VERIFY3U(load, ==, spa_load_guid(spa));
}
void
ztest_blake3(ztest_ds_t *zd, uint64_t id)
{
(void) zd, (void) id;
hrtime_t end = gethrtime() + NANOSEC;
zio_cksum_salt_t salt;
void *salt_ptr = &salt.zcs_bytes;
struct abd *abd_data, *abd_meta;
void *buf, *templ;
int i, *ptr;
uint32_t size;
BLAKE3_CTX ctx;
size = ztest_random_blocksize();
buf = umem_alloc(size, UMEM_NOFAIL);
abd_data = abd_alloc(size, B_FALSE);
abd_meta = abd_alloc(size, B_TRUE);
for (i = 0, ptr = buf; i < size / sizeof (*ptr); i++, ptr++)
*ptr = ztest_random(UINT_MAX);
memset(salt_ptr, 'A', 32);
abd_copy_from_buf_off(abd_data, buf, 0, size);
abd_copy_from_buf_off(abd_meta, buf, 0, size);
while (gethrtime() <= end) {
int run_count = 100;
zio_cksum_t zc_ref1, zc_ref2;
zio_cksum_t zc_res1, zc_res2;
void *ref1 = &zc_ref1;
void *ref2 = &zc_ref2;
void *res1 = &zc_res1;
void *res2 = &zc_res2;
/* BLAKE3_KEY_LEN = 32 */
VERIFY0(blake3_set_impl_name("generic"));
templ = abd_checksum_blake3_tmpl_init(&salt);
Blake3_InitKeyed(&ctx, salt_ptr);
Blake3_Update(&ctx, buf, size);
Blake3_Final(&ctx, ref1);
zc_ref2 = zc_ref1;
ZIO_CHECKSUM_BSWAP(&zc_ref2);
abd_checksum_blake3_tmpl_free(templ);
VERIFY0(blake3_set_impl_name("cycle"));
while (run_count-- > 0) {
/* Test current implementation */
Blake3_InitKeyed(&ctx, salt_ptr);
Blake3_Update(&ctx, buf, size);
Blake3_Final(&ctx, res1);
zc_res2 = zc_res1;
ZIO_CHECKSUM_BSWAP(&zc_res2);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
/* Test ABD - data */
templ = abd_checksum_blake3_tmpl_init(&salt);
abd_checksum_blake3_native(abd_data, size,
templ, &zc_res1);
abd_checksum_blake3_byteswap(abd_data, size,
templ, &zc_res2);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
/* Test ABD - metadata */
abd_checksum_blake3_native(abd_meta, size,
templ, &zc_res1);
abd_checksum_blake3_byteswap(abd_meta, size,
templ, &zc_res2);
abd_checksum_blake3_tmpl_free(templ);
VERIFY0(memcmp(ref1, res1, 32));
VERIFY0(memcmp(ref2, res2, 32));
}
}
abd_free(abd_data);
abd_free(abd_meta);
umem_free(buf, size);
}
void
ztest_fletcher(ztest_ds_t *zd, uint64_t id)
{

2
config/always-arch.m4
View File

@ -30,6 +30,8 @@ AC_DEFUN([ZFS_AC_CONFIG_ALWAYS_ARCH], [
;;
esac
AM_CONDITIONAL([TARGET_CPU_AARCH64], test $TARGET_CPU = aarch64)
AM_CONDITIONAL([TARGET_CPU_X86_64], test $TARGET_CPU = x86_64)
AM_CONDITIONAL([TARGET_CPU_POWERPC], test $TARGET_CPU = powerpc)
AM_CONDITIONAL([TARGET_CPU_SPARC64], test $TARGET_CPU = sparc64)
])

2
include/Makefile.am
View File

@ -23,6 +23,7 @@ COMMON_H = \
sys/avl.h \
sys/avl_impl.h \
sys/bitops.h \
sys/blake3.h \
sys/blkptr.h \
sys/bplist.h \
sys/bpobj.h \
@ -117,6 +118,7 @@ COMMON_H = \
sys/zfeature.h \
sys/zfs_acl.h \
sys/zfs_bootenv.h \
sys/zfs_chksum.h \
sys/zfs_context.h \
sys/zfs_debug.h \
sys/zfs_delay.h \

2
include/os/freebsd/spl/sys/ccompile.h
View File

@ -74,10 +74,12 @@ extern "C" {
#ifndef LOCORE
#ifndef HAVE_RPC_TYPES
#ifndef _KERNEL
typedef int bool_t;
typedef int enum_t;
#endif
#endif
#endif
#ifndef __cplusplus
#define __init

34
include/os/linux/kernel/linux/simd_powerpc.h
View File

@ -58,24 +58,44 @@
#include <linux/version.h>
#define kfpu_allowed() 1
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 5, 0)
#define kfpu_end() \
{ \
disable_kernel_vsx(); \
disable_kernel_altivec(); \
preempt_enable(); \
}
#define kfpu_begin() \
{ \
preempt_disable(); \
enable_kernel_altivec(); \
}
#if LINUX_VERSION_CODE >= KERNEL_VERSION(4, 5, 0)
#define kfpu_end() \
{ \
disable_kernel_altivec(); \
preempt_enable(); \
enable_kernel_vsx(); \
}
#else
/* seems that before 4.5 no-one bothered disabling ... */
/* seems that before 4.5 no-one bothered */
#define kfpu_begin()
#define kfpu_end() preempt_enable()
#endif
#define kfpu_init() 0
#define kfpu_fini() ((void) 0)
static inline boolean_t
zfs_vsx_available(void)
{
boolean_t res;
#if defined(__powerpc64__)
u64 msr;
#else
u32 msr;
#endif
kfpu_begin();
__asm volatile("mfmsr %0" : "=r"(msr));
res = (msr & 0x800000) != 0;
kfpu_end();
return (res);
}
/*
* Check if AltiVec instruction set is available
*/

120
include/sys/blake3.h Normal file
View File

@ -0,0 +1,120 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef BLAKE3_H
#define BLAKE3_H
#ifdef _KERNEL
#include <sys/types.h>
#else
#include <stdint.h>
#include <stdlib.h>
#endif
#ifdef __cplusplus
extern "C" {
#endif
#define BLAKE3_KEY_LEN 32
#define BLAKE3_OUT_LEN 32
#define BLAKE3_MAX_DEPTH 54
#define BLAKE3_BLOCK_LEN 64
#define BLAKE3_CHUNK_LEN 1024
/*
* This struct is a private implementation detail.
* It has to be here because it's part of BLAKE3_CTX below.
*/
typedef struct {
uint32_t cv[8];
uint64_t chunk_counter;
uint8_t buf[BLAKE3_BLOCK_LEN];
uint8_t buf_len;
uint8_t blocks_compressed;
uint8_t flags;
} blake3_chunk_state_t;
typedef struct {
uint32_t key[8];
blake3_chunk_state_t chunk;
uint8_t cv_stack_len;
/*
* The stack size is MAX_DEPTH + 1 because we do lazy merging. For
* example, with 7 chunks, we have 3 entries in the stack. Adding an
* 8th chunk requires a 4th entry, rather than merging everything down
* to 1, because we don't know whether more input is coming. This is
* different from how the reference implementation does things.
*/
uint8_t cv_stack[(BLAKE3_MAX_DEPTH + 1) * BLAKE3_OUT_LEN];
/* const blake3_impl_ops_t *ops */
const void *ops;
} BLAKE3_CTX;
/* init the context for hash operation */
void Blake3_Init(BLAKE3_CTX *ctx);
/* init the context for a MAC and/or tree hash operation */
void Blake3_InitKeyed(BLAKE3_CTX *ctx, const uint8_t key[BLAKE3_KEY_LEN]);
/* process the input bytes */
void Blake3_Update(BLAKE3_CTX *ctx, const void *input, size_t input_len);
/* finalize the hash computation and output the result */
void Blake3_Final(const BLAKE3_CTX *ctx, uint8_t *out);
/* finalize the hash computation and output the result */
void Blake3_FinalSeek(const BLAKE3_CTX *ctx, uint64_t seek, uint8_t *out,
size_t out_len);
/* return number of supported implementations */
extern int blake3_get_impl_count(void);
/* return id of selected implementation */
extern int blake3_get_impl_id(void);
/* return name of selected implementation */
extern const char *blake3_get_impl_name(void);
/* setup id as fastest implementation */
extern void blake3_set_impl_fastest(uint32_t id);
/* set implementation by id */
extern void blake3_set_impl_id(uint32_t id);
/* set implementation by name */
extern int blake3_set_impl_name(const char *name);
/* set startup implementation */
extern void blake3_setup_impl(void);
#ifdef __cplusplus
}
#endif
#endif /* BLAKE3_H */

48
include/sys/zfs_chksum.h Normal file
View File

@ -0,0 +1,48 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef _ZFS_CHKSUM_H
#define _ZFS_CHKSUM_H
#ifdef _KERNEL
#include <sys/types.h>
#else
#include <stdint.h>
#include <stdlib.h>
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* Benchmark the chksums of ZFS when the module is loading */
void chksum_init(void);
void chksum_fini(void);
#ifdef __cplusplus
}
#endif
#endif /* _ZFS_CHKSUM_H */

3
include/sys/zfs_ioctl.h
View File

@ -124,6 +124,7 @@ typedef enum drr_headertype {
* default use of "zfs send" won't encounter the bug mentioned above.
*/
#define DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS (1 << 27)
#define DMU_BACKUP_FEATURE_BLAKE3 (1 << 28)
/*
* Mask of all supported backup features
@ -134,7 +135,7 @@ typedef enum drr_headertype {
DMU_BACKUP_FEATURE_COMPRESSED | DMU_BACKUP_FEATURE_LARGE_DNODE | \
DMU_BACKUP_FEATURE_RAW | DMU_BACKUP_FEATURE_HOLDS | \
DMU_BACKUP_FEATURE_REDACTED | DMU_BACKUP_FEATURE_SWITCH_TO_LARGE_BLOCKS | \
DMU_BACKUP_FEATURE_ZSTD)
DMU_BACKUP_FEATURE_ZSTD | DMU_BACKUP_FEATURE_BLAKE3)
/* Are all features in the given flag word currently supported? */
#define DMU_STREAM_SUPPORTED(x) (!((x) & ~DMU_BACKUP_FEATURE_MASK))

1
include/sys/zio.h
View File

@ -89,6 +89,7 @@ enum zio_checksum {
ZIO_CHECKSUM_SHA512,
ZIO_CHECKSUM_SKEIN,
ZIO_CHECKSUM_EDONR,
ZIO_CHECKSUM_BLAKE3,
ZIO_CHECKSUM_FUNCTIONS
};

12
include/sys/zio_checksum.h
View File

@ -21,7 +21,8 @@
/*
* Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2014, 2016 by Delphix. All rights reserved.
* Copyright Saso Kiselkov 2013, All rights reserved.
* Copyright (c) 2013 Saso Kiselkov, All rights reserved.
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef _SYS_ZIO_CHECKSUM_H
@ -107,6 +108,8 @@ _SYS_ZIO_CHECKSUM_H zio_checksum_info_t
/*
* Checksum routines.
*/
/* SHA2 */
extern zio_checksum_t abd_checksum_SHA256;
extern zio_checksum_t abd_checksum_SHA512_native;
extern zio_checksum_t abd_checksum_SHA512_byteswap;
@ -123,6 +126,13 @@ extern zio_checksum_t abd_checksum_edonr_byteswap;
extern zio_checksum_tmpl_init_t abd_checksum_edonr_tmpl_init;
extern zio_checksum_tmpl_free_t abd_checksum_edonr_tmpl_free;
/* BLAKE3 */
extern zio_checksum_t abd_checksum_blake3_native;
extern zio_checksum_t abd_checksum_blake3_byteswap;
extern zio_checksum_tmpl_init_t abd_checksum_blake3_tmpl_init;
extern zio_checksum_tmpl_free_t abd_checksum_blake3_tmpl_free;
/* Fletcher 4 */
_SYS_ZIO_CHECKSUM_H zio_abd_checksum_func_t fletcher_4_abd_ops;
extern zio_checksum_t abd_fletcher_4_native;
extern zio_checksum_t abd_fletcher_4_byteswap;

1
include/zfeature_common.h
View File

@ -77,6 +77,7 @@ typedef enum spa_feature {
SPA_FEATURE_DRAID,
SPA_FEATURE_ZILSAXATTR,
SPA_FEATURE_HEAD_ERRLOG,
SPA_FEATURE_BLAKE3,
SPA_FEATURES
} spa_feature_t;

25
lib/libicp/Makefile.am
View File

@ -13,6 +13,10 @@ nodist_libicp_la_SOURCES = \
module/icp/algs/aes/aes_impl_x86-64.c \
module/icp/algs/aes/aes_impl.c \
module/icp/algs/aes/aes_modes.c \
module/icp/algs/blake3/blake3.c \
module/icp/algs/blake3/blake3_generic.c \
module/icp/algs/blake3/blake3_impl.c \
module/icp/algs/blake3/blake3_x86-64.c \
module/icp/algs/edonr/edonr.c \
module/icp/algs/modes/modes.c \
module/icp/algs/modes/cbc.c \
@ -36,15 +40,30 @@ nodist_libicp_la_SOURCES = \
module/icp/core/kcf_mech_tabs.c \
module/icp/core/kcf_prov_tabs.c
if TARGET_CPU_AARCH64
nodist_libicp_la_SOURCES += \
module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S \
module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S
endif
if TARGET_CPU_POWERPC
nodist_libicp_la_SOURCES += \
module/icp/asm-ppc64/blake3/b3_ppc64le_sse2.S \
module/icp/asm-ppc64/blake3/b3_ppc64le_sse41.S
endif
if TARGET_CPU_X86_64
nodist_libicp_la_SOURCES += \
module/icp/asm-x86_64/aes/aeskey.c
nodist_libicp_la_SOURCES += \
module/icp/asm-x86_64/aes/aeskey.c \
module/icp/asm-x86_64/aes/aes_amd64.S \
module/icp/asm-x86_64/aes/aes_aesni.S \
module/icp/asm-x86_64/modes/gcm_pclmulqdq.S \
module/icp/asm-x86_64/modes/aesni-gcm-x86_64.S \
module/icp/asm-x86_64/modes/ghash-x86_64.S \
module/icp/asm-x86_64/sha2/sha256_impl.S \
module/icp/asm-x86_64/sha2/sha512_impl.S
module/icp/asm-x86_64/sha2/sha512_impl.S \
module/icp/asm-x86_64/blake3/blake3_avx2.S \
module/icp/asm-x86_64/blake3/blake3_avx512.S \
module/icp/asm-x86_64/blake3/blake3_sse2.S \
module/icp/asm-x86_64/blake3/blake3_sse41.S
endif

18
lib/libspl/include/sys/simd.h
View File

@ -491,6 +491,24 @@ zfs_altivec_available(void)
#endif
return (has_altivec);
}
static inline boolean_t
zfs_vsx_available(void)
{
boolean_t has_vsx = B_FALSE;
#if defined(__ALTIVEC__) && !defined(__FreeBSD__)
sighandler_t savesig;
savesig = signal(SIGILL, sigillhandler);
if (setjmp(env)) {
signal(SIGILL, savesig);
has_vsx = B_FALSE;
} else {
__asm__ __volatile__("xssubsp 0,0,0\n");
signal(SIGILL, savesig);
has_vsx = B_TRUE;
}
#endif
return (has_vsx);
}
#else
#define kfpu_allowed() 0

9
lib/libzfs/libzfs.abi
View File

@ -583,7 +583,7 @@
<elf-symbol name='fletcher_4_superscalar_ops' size='64' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='libzfs_config_ops' size='16' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='sa_protocol_names' size='16' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='spa_feature_table' size='2016' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='spa_feature_table' size='2072' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfeature_checks_disable' size='4' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfs_deleg_perm_tab' size='512' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
<elf-symbol name='zfs_history_event_names' size='328' type='object-type' binding='global-binding' visibility='default-visibility' is-defined='yes'/>
@ -4770,8 +4770,8 @@
</function-decl>
</abi-instr>
<abi-instr address-size='64' path='module/zcommon/zfeature_common.c' language='LANG_C99'>
<array-type-def dimensions='1' type-id='83f29ca2' size-in-bits='16128' id='9d5e9e2e'>
<subrange length='36' type-id='7359adad' id='ae666bde'/>
<array-type-def dimensions='1' type-id='83f29ca2' size-in-bits='16576' id='9d5e9e2e'>
<subrange length='37' type-id='7359adad' id='ae666bde'/>
</array-type-def>
<enum-decl name='spa_feature' id='33ecb627'>
<underlying-type type-id='9cac1fee'/>
@ -4812,7 +4812,8 @@
<enumerator name='SPA_FEATURE_DRAID' value='33'/>
<enumerator name='SPA_FEATURE_ZILSAXATTR' value='34'/>
<enumerator name='SPA_FEATURE_HEAD_ERRLOG' value='35'/>
<enumerator name='SPA_FEATURES' value='36'/>
<enumerator name='SPA_FEATURE_BLAKE3' value='36'/>
<enumerator name='SPA_FEATURES' value='37'/>
</enum-decl>
<typedef-decl name='spa_feature_t' type-id='33ecb627' id='d6618c78'/>
<enum-decl name='zfeature_flags' id='6db816a4'>

2
lib/libzpool/Makefile.am
View File

@ -67,6 +67,7 @@ nodist_libzpool_la_SOURCES = \
module/zfs/abd.c \
module/zfs/aggsum.c \
module/zfs/arc.c \
module/zfs/blake3_zfs.c \
module/zfs/blkptr.c \
module/zfs/bplist.c \
module/zfs/bpobj.c \
@ -171,6 +172,7 @@ nodist_libzpool_la_SOURCES = \
module/zfs/zcp_synctask.c \
module/zfs/zfeature.c \
module/zfs/zfs_byteswap.c \
module/zfs/zfs_chksum.c \
module/zfs/zfs_fm.c \
module/zfs/zfs_fuid.c \
module/zfs/zfs_ratelimit.c \

7
man/man7/zfsprops.7
View File

@ -743,7 +743,7 @@ This property is not inherited.
.It Xo
.Sy checksum Ns = Ns Sy on Ns | Ns Sy off Ns | Ns Sy fletcher2 Ns | Ns
.Sy fletcher4 Ns | Ns Sy sha256 Ns | Ns Sy noparity Ns | Ns
.Sy sha512 Ns | Ns Sy skein Ns | Ns Sy edonr
.Sy sha512 Ns | Ns Sy skein Ns | Ns Sy edonr Ns | Ns Sy blake3
.Xc
Controls the checksum used to verify data integrity.
The default value is
@ -768,8 +768,9 @@ a recommended practice.
The
.Sy sha512 ,
.Sy skein ,
.Sy edonr ,
and
.Sy edonr
.Sy blake3
checksum algorithms require enabling the appropriate features on the pool.
.Pp
Please see
@ -984,7 +985,7 @@ mount options.
.It Xo
.Sy dedup Ns = Ns Sy off Ns | Ns Sy on Ns | Ns Sy verify Ns | Ns
.Sy sha256 Ns Oo , Ns Sy verify Oc Ns | Ns Sy sha512 Ns Oo , Ns Sy verify Oc Ns | Ns Sy skein Ns Oo , Ns Sy verify Oc Ns | Ns
.Sy edonr , Ns Sy verify
.Sy edonr , Ns Sy verify Ns | Ns Sy blake3 Ns Oo , Ns Sy verify Oc Ns
.Xc
Configures deduplication for a dataset.
The default value is

8
man/man7/zpool-features.7
View File

@ -326,6 +326,12 @@ while
.Sy freeing
is non-zero.
.
.feature org.openzfs blake3 no extensible_dataset
This feature enables the use of the BLAKE3 hash algorithm for checksum and dedup.
BLAKE3 is a secure hash algorithm focused on high performance.
.Pp
.checksum-spiel blake3
.
.feature com.delphix bookmarks yes extensible_dataset
This feature enables use of the
.Nm zfs Cm bookmark
@ -436,6 +442,8 @@ in ZFS, which means that the checksum is pre-seeded with a secret
to be checksummed.
Thus the produced checksums are unique to a given pool,
preventing hash collision attacks on systems with dedup.
.Pp
.checksum-spiel edonr
.
.feature com.delphix embedded_data no
This feature improves the performance and compression ratio of

32
module/Kbuild.in
View File

@ -75,6 +75,10 @@ ICP_OBJS := \
algs/aes/aes_impl.o \
algs/aes/aes_impl_generic.o \
algs/aes/aes_modes.o \
algs/blake3/blake3.o \
algs/blake3/blake3_generic.o \
algs/blake3/blake3_impl.o \
algs/blake3/blake3_x86-64.o \
algs/edonr/edonr.o \
algs/modes/cbc.o \
algs/modes/ccm.o \
@ -105,23 +109,44 @@ ICP_OBJS_X86_64 := \
asm-x86_64/aes/aes_aesni.o \
asm-x86_64/aes/aes_amd64.o \
asm-x86_64/aes/aeskey.o \
asm-x86_64/blake3/blake3_avx2.o \
asm-x86_64/blake3/blake3_avx512.o \
asm-x86_64/blake3/blake3_sse2.o \
asm-x86_64/blake3/blake3_sse41.o \
asm-x86_64/modes/aesni-gcm-x86_64.o \
asm-x86_64/modes/gcm_pclmulqdq.o \
asm-x86_64/modes/ghash-x86_64.o \
asm-x86_64/sha2/sha256_impl.o \
asm-x86_64/sha2/sha512_impl.o
ICP_OBJS_X86 := \
algs/aes/aes_impl_aesni.o \
algs/aes/aes_impl_x86-64.o \
algs/modes/gcm_pclmulqdq.o
ICP_OBJS_ARM64 := \
asm-aarch64/blake3/b3_aarch64_sse2.o \
asm-aarch64/blake3/b3_aarch64_sse41.o
ICP_OBJS_PPC_PPC64 := \
asm-ppc64/blake3/b3_ppc64le_sse2.o \
asm-ppc64/blake3/b3_ppc64le_sse41.o
zfs-objs += $(addprefix icp/,$(ICP_OBJS))
zfs-$(CONFIG_X86) += $(addprefix icp/,$(ICP_OBJS_X86))
zfs-$(CONFIG_X86_64) += $(addprefix icp/,$(ICP_OBJS_X86_64))
zfs-$(CONFIG_ARM64) += $(addprefix icp/,$(ICP_OBJS_ARM64))
zfs-$(CONFIG_PPC) += $(addprefix icp/,$(ICP_OBJS_PPC_PPC64))
zfs-$(CONFIG_PPC64) += $(addprefix icp/,$(ICP_OBJS_PPC_PPC64))
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : asflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : ccflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64) \
$(ICP_OBJS_ARM64) $(ICP_OBJS_PPC_PPC64)) : asflags-y += -I$(icp_include)
$(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64) \
$(ICP_OBJS_ARM64) $(ICP_OBJS_PPC_PPC64)) : ccflags-y += -I$(icp_include)
# Suppress objtool "can't find jump dest instruction at" warnings. They
# are caused by the constants which are defined in the text section of the
@ -129,6 +154,7 @@ $(addprefix $(obj)/icp/,$(ICP_OBJS) $(ICP_OBJS_X86) $(ICP_OBJS_X86_64)) : ccflag
# utility tries to interpret them as opcodes and obviously fails doing so.
OBJECT_FILES_NON_STANDARD_aesni-gcm-x86_64.o := y
OBJECT_FILES_NON_STANDARD_ghash-x86_64.o := y
# Suppress objtool "unsupported stack pointer realignment" warnings. We are
# not using a DRAP register while aligning the stack to a 64 byte boundary.
# See #6950 for the reasoning.
@ -261,6 +287,7 @@ ZFS_OBJS := \
abd.o \
aggsum.o \
arc.o \
blake3_zfs.o \
blkptr.o \
bplist.o \
bpobj.o \
@ -358,6 +385,7 @@ ZFS_OBJS := \
zcp_synctask.o \
zfeature.o \
zfs_byteswap.o \
zfs_chksum.o \
zfs_fm.o \
zfs_fuid.o \
zfs_ioctl.o \

32
module/Makefile.bsd
View File

@ -10,6 +10,10 @@ INCDIR=${.CURDIR:H}/include
KMOD= openzfs
.PATH: ${SRCDIR}/avl \
${SRCDIR}/icp/algs/blake3 \
${SRCDIR}/icp/asm-aarch64/blake3 \
${SRCDIR}/icp/asm-ppc64/blake3 \
${SRCDIR}/icp/asm-x86_64/blake3 \
${SRCDIR}/lua \
${SRCDIR}/nvpair \
${SRCDIR}/icp/algs/edonr \
@ -31,6 +35,7 @@ CFLAGS+= -I${INCDIR}/os/freebsd
CFLAGS+= -I${INCDIR}/os/freebsd/spl
CFLAGS+= -I${INCDIR}/os/freebsd/zfs
CFLAGS+= -I${SRCDIR}/zstd/include
CFLAGS+= -I${SRCDIR}/icp/include
CFLAGS+= -include ${INCDIR}/os/freebsd/spl/sys/ccompile.h
CFLAGS+= -D__KERNEL__ -DFREEBSD_NAMECACHE -DBUILDING_ZFS -D__BSD_VISIBLE=1 \
@ -38,7 +43,8 @@ CFLAGS+= -D__KERNEL__ -DFREEBSD_NAMECACHE -DBUILDING_ZFS -D__BSD_VISIBLE=1 \
-D_SYS_VMEM_H_ -DKDTRACE_HOOKS -DSMP -DCOMPAT_FREEBSD11
.if ${MACHINE_ARCH} == "amd64"
CFLAGS+= -DHAVE_AVX2 -DHAVE_AVX -D__x86_64 -DHAVE_SSE2 -DHAVE_AVX512F -DHAVE_SSSE3
CFLAGS+= -D__x86_64 -DHAVE_SSE2 -DHAVE_SSSE3 -DHAVE_SSE4_1 -DHAVE_SSE4_2 \
-DHAVE_AVX -DHAVE_AVX2 -DHAVE_AVX512F -DHAVE_AVX512VL
.endif
.if defined(WITH_DEBUG) && ${WITH_DEBUG} == "true"
@ -79,6 +85,26 @@ SRCS+= avl.c
# icp
SRCS+= edonr.c
#icp/algs/blake3
SRCS+= blake3.c \
blake3_generic.c \
blake3_impl.c \
blake3_x86-64.c
#icp/asm-aarch64/blake3
SRCS+= b3_aarch64_sse2.S \
b3_aarch64_sse41.S
#icp/asm-ppc64/blake3
SRCS+= b3_ppc64le_sse2.S \
b3_ppc64le_sse41.S
#icp/asm-x86_64/blake3
SRCS+= blake3_avx2.S \
blake3_avx512.S \
blake3_sse2.S \
blake3_sse41.S
#lua
SRCS+= lapi.c \
lauxlib.c \
@ -189,6 +215,7 @@ SRCS+= zfeature_common.c \
SRCS+= abd.c \
aggsum.c \
arc.c \
blake3_zfs.c \
blkptr.c \
bplist.c \
bpobj.c \
@ -291,6 +318,7 @@ SRCS+= abd.c \
zcp_synctask.c \
zfeature.c \
zfs_byteswap.c \
zfs_chksum.c \
zfs_file_os.c \
zfs_fm.c \
zfs_fuid.c \
@ -337,8 +365,6 @@ SRCS+= zfs_zstd.c \
zstd_decompress.c \
zstd_decompress_block.c
beforeinstall:
.if ${MK_DEBUG_FILES} != "no"
mtree -eu \

732
module/icp/algs/blake3/blake3.c Normal file
View File

@ -0,0 +1,732 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/blake3.h>
#include "blake3_impl.h"
/*
* We need 1056 byte stack for blake3_compress_subtree_wide()
* - we define this pragma to make gcc happy
*/
#if defined(__GNUC__)
#pragma GCC diagnostic ignored "-Wframe-larger-than="
#endif
/* internal used */
typedef struct {
uint32_t input_cv[8];
uint64_t counter;
uint8_t block[BLAKE3_BLOCK_LEN];
uint8_t block_len;
uint8_t flags;
} output_t;
/* internal flags */
enum blake3_flags {
CHUNK_START = 1 << 0,
CHUNK_END = 1 << 1,
PARENT = 1 << 2,
ROOT = 1 << 3,
KEYED_HASH = 1 << 4,
DERIVE_KEY_CONTEXT = 1 << 5,
DERIVE_KEY_MATERIAL = 1 << 6,
};
/* internal start */
static void chunk_state_init(blake3_chunk_state_t *ctx,
const uint32_t key[8], uint8_t flags)
{
memcpy(ctx->cv, key, BLAKE3_KEY_LEN);
ctx->chunk_counter = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
ctx->buf_len = 0;
ctx->blocks_compressed = 0;
ctx->flags = flags;
}
static void chunk_state_reset(blake3_chunk_state_t *ctx,
const uint32_t key[8], uint64_t chunk_counter)
{
memcpy(ctx->cv, key, BLAKE3_KEY_LEN);
ctx->chunk_counter = chunk_counter;
ctx->blocks_compressed = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
ctx->buf_len = 0;
}
static size_t chunk_state_len(const blake3_chunk_state_t *ctx)
{
return (BLAKE3_BLOCK_LEN * (size_t)ctx->blocks_compressed) +
((size_t)ctx->buf_len);
}
static size_t chunk_state_fill_buf(blake3_chunk_state_t *ctx,
const uint8_t *input, size_t input_len)
{
size_t take = BLAKE3_BLOCK_LEN - ((size_t)ctx->buf_len);
if (take > input_len) {
take = input_len;
}
uint8_t *dest = ctx->buf + ((size_t)ctx->buf_len);
memcpy(dest, input, take);
ctx->buf_len += (uint8_t)take;
return (take);
}
static uint8_t chunk_state_maybe_start_flag(const blake3_chunk_state_t *ctx)
{
if (ctx->blocks_compressed == 0) {
return (CHUNK_START);
} else {
return (0);
}
}
static output_t make_output(const uint32_t input_cv[8],
const uint8_t *block, uint8_t block_len,
uint64_t counter, uint8_t flags)
{
output_t ret;
memcpy(ret.input_cv, input_cv, 32);
memcpy(ret.block, block, BLAKE3_BLOCK_LEN);
ret.block_len = block_len;
ret.counter = counter;
ret.flags = flags;
return (ret);
}
/*
* Chaining values within a given chunk (specifically the compress_in_place
* interface) are represented as words. This avoids unnecessary bytes<->words
* conversion overhead in the portable implementation. However, the hash_many
* interface handles both user input and parent node blocks, so it accepts
* bytes. For that reason, chaining values in the CV stack are represented as
* bytes.
*/
static void output_chaining_value(const blake3_impl_ops_t *ops,
const output_t *ctx, uint8_t cv[32])
{
uint32_t cv_words[8];
memcpy(cv_words, ctx->input_cv, 32);
ops->compress_in_place(cv_words, ctx->block, ctx->block_len,
ctx->counter, ctx->flags);
store_cv_words(cv, cv_words);
}
static void output_root_bytes(const blake3_impl_ops_t *ops, const output_t *ctx,
uint64_t seek, uint8_t *out, size_t out_len)
{
uint64_t output_block_counter = seek / 64;
size_t offset_within_block = seek % 64;
uint8_t wide_buf[64];
while (out_len > 0) {
ops->compress_xof(ctx->input_cv, ctx->block, ctx->block_len,
output_block_counter, ctx->flags | ROOT, wide_buf);
size_t available_bytes = 64 - offset_within_block;
size_t memcpy_len;
if (out_len > available_bytes) {
memcpy_len = available_bytes;
} else {
memcpy_len = out_len;
}
memcpy(out, wide_buf + offset_within_block, memcpy_len);
out += memcpy_len;
out_len -= memcpy_len;
output_block_counter += 1;
offset_within_block = 0;
}
}
static void chunk_state_update(const blake3_impl_ops_t *ops,
blake3_chunk_state_t *ctx, const uint8_t *input, size_t input_len)
{
if (ctx->buf_len > 0) {
size_t take = chunk_state_fill_buf(ctx, input, input_len);
input += take;
input_len -= take;
if (input_len > 0) {
ops->compress_in_place(ctx->cv, ctx->buf,
BLAKE3_BLOCK_LEN, ctx->chunk_counter,
ctx->flags|chunk_state_maybe_start_flag(ctx));
ctx->blocks_compressed += 1;
ctx->buf_len = 0;
memset(ctx->buf, 0, BLAKE3_BLOCK_LEN);
}
}
while (input_len > BLAKE3_BLOCK_LEN) {
ops->compress_in_place(ctx->cv, input, BLAKE3_BLOCK_LEN,
ctx->chunk_counter,
ctx->flags|chunk_state_maybe_start_flag(ctx));
ctx->blocks_compressed += 1;
input += BLAKE3_BLOCK_LEN;
input_len -= BLAKE3_BLOCK_LEN;
}
size_t take = chunk_state_fill_buf(ctx, input, input_len);
input += take;
input_len -= take;
}
static output_t chunk_state_output(const blake3_chunk_state_t *ctx)
{
uint8_t block_flags =
ctx->flags | chunk_state_maybe_start_flag(ctx) | CHUNK_END;
return (make_output(ctx->cv, ctx->buf, ctx->buf_len, ctx->chunk_counter,
block_flags));
}
static output_t parent_output(const uint8_t block[BLAKE3_BLOCK_LEN],
const uint32_t key[8], uint8_t flags)
{
return (make_output(key, block, BLAKE3_BLOCK_LEN, 0, flags | PARENT));
}
/*
* Given some input larger than one chunk, return the number of bytes that
* should go in the left subtree. This is the largest power-of-2 number of
* chunks that leaves at least 1 byte for the right subtree.
*/
static size_t left_len(size_t content_len)
{
/*
* Subtract 1 to reserve at least one byte for the right side.
* content_len
* should always be greater than BLAKE3_CHUNK_LEN.
*/
size_t full_chunks = (content_len - 1) / BLAKE3_CHUNK_LEN;
return (round_down_to_power_of_2(full_chunks) * BLAKE3_CHUNK_LEN);
}
/*
* Use SIMD parallelism to hash up to MAX_SIMD_DEGREE chunks at the same time
* on a single thread. Write out the chunk chaining values and return the
* number of chunks hashed. These chunks are never the root and never empty;
* those cases use a different codepath.
*/
static size_t compress_chunks_parallel(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t *out)
{
const uint8_t *chunks_array[MAX_SIMD_DEGREE];
size_t input_position = 0;
size_t chunks_array_len = 0;
while (input_len - input_position >= BLAKE3_CHUNK_LEN) {
chunks_array[chunks_array_len] = &input[input_position];
input_position += BLAKE3_CHUNK_LEN;
chunks_array_len += 1;
}
ops->hash_many(chunks_array, chunks_array_len, BLAKE3_CHUNK_LEN /
BLAKE3_BLOCK_LEN, key, chunk_counter, B_TRUE, flags, CHUNK_START,
CHUNK_END, out);
/*
* Hash the remaining partial chunk, if there is one. Note that the
* empty chunk (meaning the empty message) is a different codepath.
*/
if (input_len > input_position) {
uint64_t counter = chunk_counter + (uint64_t)chunks_array_len;
blake3_chunk_state_t chunk_state;
chunk_state_init(&chunk_state, key, flags);
chunk_state.chunk_counter = counter;
chunk_state_update(ops, &chunk_state, &input[input_position],
input_len - input_position);
output_t output = chunk_state_output(&chunk_state);
output_chaining_value(ops, &output, &out[chunks_array_len *
BLAKE3_OUT_LEN]);
return (chunks_array_len + 1);
} else {
return (chunks_array_len);
}
}
/*
* Use SIMD parallelism to hash up to MAX_SIMD_DEGREE parents at the same time
* on a single thread. Write out the parent chaining values and return the
* number of parents hashed. (If there's an odd input chaining value left over,
* return it as an additional output.) These parents are never the root and
* never empty; those cases use a different codepath.
*/
static size_t compress_parents_parallel(const blake3_impl_ops_t *ops,
const uint8_t *child_chaining_values, size_t num_chaining_values,
const uint32_t key[8], uint8_t flags, uint8_t *out)
{
const uint8_t *parents_array[MAX_SIMD_DEGREE_OR_2];
size_t parents_array_len = 0;
while (num_chaining_values - (2 * parents_array_len) >= 2) {
parents_array[parents_array_len] = &child_chaining_values[2 *
parents_array_len * BLAKE3_OUT_LEN];
parents_array_len += 1;
}
ops->hash_many(parents_array, parents_array_len, 1, key, 0, B_FALSE,
flags | PARENT, 0, 0, out);
/* If there's an odd child left over, it becomes an output. */
if (num_chaining_values > 2 * parents_array_len) {
memcpy(&out[parents_array_len * BLAKE3_OUT_LEN],
&child_chaining_values[2 * parents_array_len *
BLAKE3_OUT_LEN], BLAKE3_OUT_LEN);
return (parents_array_len + 1);
} else {
return (parents_array_len);
}
}
/*
* The wide helper function returns (writes out) an array of chaining values
* and returns the length of that array. The number of chaining values returned
* is the dyanmically detected SIMD degree, at most MAX_SIMD_DEGREE. Or fewer,
* if the input is shorter than that many chunks. The reason for maintaining a
* wide array of chaining values going back up the tree, is to allow the
* implementation to hash as many parents in parallel as possible.
*
* As a special case when the SIMD degree is 1, this function will still return
* at least 2 outputs. This guarantees that this function doesn't perform the
* root compression. (If it did, it would use the wrong flags, and also we
* wouldn't be able to implement exendable ouput.) Note that this function is
* not used when the whole input is only 1 chunk long; that's a different
* codepath.
*
* Why not just have the caller split the input on the first update(), instead
* of implementing this special rule? Because we don't want to limit SIMD or
* multi-threading parallelism for that update().
*/
static size_t blake3_compress_subtree_wide(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t *out)
{
/*
* Note that the single chunk case does *not* bump the SIMD degree up
* to 2 when it is 1. If this implementation adds multi-threading in
* the future, this gives us the option of multi-threading even the
* 2-chunk case, which can help performance on smaller platforms.
*/
if (input_len <= (size_t)(ops->degree * BLAKE3_CHUNK_LEN)) {
return (compress_chunks_parallel(ops, input, input_len, key,
chunk_counter, flags, out));
}
/*
* With more than simd_degree chunks, we need to recurse. Start by
* dividing the input into left and right subtrees. (Note that this is
* only optimal as long as the SIMD degree is a power of 2. If we ever
* get a SIMD degree of 3 or something, we'll need a more complicated
* strategy.)
*/
size_t left_input_len = left_len(input_len);
size_t right_input_len = input_len - left_input_len;
const uint8_t *right_input = &input[left_input_len];
uint64_t right_chunk_counter = chunk_counter +
(uint64_t)(left_input_len / BLAKE3_CHUNK_LEN);
/*
* Make space for the child outputs. Here we use MAX_SIMD_DEGREE_OR_2
* to account for the special case of returning 2 outputs when the
* SIMD degree is 1.
*/
uint8_t cv_array[2 * MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN];
size_t degree = ops->degree;
if (left_input_len > BLAKE3_CHUNK_LEN && degree == 1) {
/*
* The special case: We always use a degree of at least two,
* to make sure there are two outputs. Except, as noted above,
* at the chunk level, where we allow degree=1. (Note that the
* 1-chunk-input case is a different codepath.)
*/
degree = 2;
}
uint8_t *right_cvs = &cv_array[degree * BLAKE3_OUT_LEN];
/*
* Recurse! If this implementation adds multi-threading support in the
* future, this is where it will go.
*/
size_t left_n = blake3_compress_subtree_wide(ops, input, left_input_len,
key, chunk_counter, flags, cv_array);
size_t right_n = blake3_compress_subtree_wide(ops, right_input,
right_input_len, key, right_chunk_counter, flags, right_cvs);
/*
* The special case again. If simd_degree=1, then we'll have left_n=1
* and right_n=1. Rather than compressing them into a single output,
* return them directly, to make sure we always have at least two
* outputs.
*/
if (left_n == 1) {
memcpy(out, cv_array, 2 * BLAKE3_OUT_LEN);
return (2);
}
/* Otherwise, do one layer of parent node compression. */
size_t num_chaining_values = left_n + right_n;
return compress_parents_parallel(ops, cv_array,
num_chaining_values, key, flags, out);
}
/*
* Hash a subtree with compress_subtree_wide(), and then condense the resulting
* list of chaining values down to a single parent node. Don't compress that
* last parent node, however. Instead, return its message bytes (the
* concatenated chaining values of its children). This is necessary when the
* first call to update() supplies a complete subtree, because the topmost
* parent node of that subtree could end up being the root. It's also necessary
* for extended output in the general case.
*
* As with compress_subtree_wide(), this function is not used on inputs of 1
* chunk or less. That's a different codepath.
*/
static void compress_subtree_to_parent_node(const blake3_impl_ops_t *ops,
const uint8_t *input, size_t input_len, const uint32_t key[8],
uint64_t chunk_counter, uint8_t flags, uint8_t out[2 * BLAKE3_OUT_LEN])
{
uint8_t cv_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN];
size_t num_cvs = blake3_compress_subtree_wide(ops, input, input_len,
key, chunk_counter, flags, cv_array);
/*
* If MAX_SIMD_DEGREE is greater than 2 and there's enough input,
* compress_subtree_wide() returns more than 2 chaining values. Condense
* them into 2 by forming parent nodes repeatedly.
*/
uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
while (num_cvs > 2) {
num_cvs = compress_parents_parallel(ops, cv_array, num_cvs, key,
flags, out_array);
memcpy(cv_array, out_array, num_cvs * BLAKE3_OUT_LEN);
}
memcpy(out, cv_array, 2 * BLAKE3_OUT_LEN);
}
static void hasher_init_base(BLAKE3_CTX *ctx, const uint32_t key[8],
uint8_t flags)
{
memcpy(ctx->key, key, BLAKE3_KEY_LEN);
chunk_state_init(&ctx->chunk, key, flags);
ctx->cv_stack_len = 0;
ctx->ops = blake3_impl_get_ops();
}
/*
* As described in hasher_push_cv() below, we do "lazy merging", delaying
* merges until right before the next CV is about to be added. This is
* different from the reference implementation. Another difference is that we
* aren't always merging 1 chunk at a time. Instead, each CV might represent
* any power-of-two number of chunks, as long as the smaller-above-larger
* stack order is maintained. Instead of the "count the trailing 0-bits"
* algorithm described in the spec, we use a "count the total number of
* 1-bits" variant that doesn't require us to retain the subtree size of the
* CV on top of the stack. The principle is the same: each CV that should
* remain in the stack is represented by a 1-bit in the total number of chunks
* (or bytes) so far.
*/
static void hasher_merge_cv_stack(BLAKE3_CTX *ctx, uint64_t total_len)
{
size_t post_merge_stack_len = (size_t)popcnt(total_len);
while (ctx->cv_stack_len > post_merge_stack_len) {
uint8_t *parent_node =
&ctx->cv_stack[(ctx->cv_stack_len - 2) * BLAKE3_OUT_LEN];
output_t output =
parent_output(parent_node, ctx->key, ctx->chunk.flags);
output_chaining_value(ctx->ops, &output, parent_node);
ctx->cv_stack_len -= 1;
}
}
/*
* In reference_impl.rs, we merge the new CV with existing CVs from the stack
* before pushing it. We can do that because we know more input is coming, so
* we know none of the merges are root.
*
* This setting is different. We want to feed as much input as possible to
* compress_subtree_wide(), without setting aside anything for the chunk_state.
* If the user gives us 64 KiB, we want to parallelize over all 64 KiB at once
* as a single subtree, if at all possible.
*
* This leads to two problems:
* 1) This 64 KiB input might be the only call that ever gets made to update.
* In this case, the root node of the 64 KiB subtree would be the root node
* of the whole tree, and it would need to be ROOT finalized. We can't
* compress it until we know.
* 2) This 64 KiB input might complete a larger tree, whose root node is
* similarly going to be the the root of the whole tree. For example, maybe
* we have 196 KiB (that is, 128 + 64) hashed so far. We can't compress the
* node at the root of the 256 KiB subtree until we know how to finalize it.
*
* The second problem is solved with "lazy merging". That is, when we're about
* to add a CV to the stack, we don't merge it with anything first, as the
* reference impl does. Instead we do merges using the *previous* CV that was
* added, which is sitting on top of the stack, and we put the new CV
* (unmerged) on top of the stack afterwards. This guarantees that we never
* merge the root node until finalize().
*
* Solving the first problem requires an additional tool,
* compress_subtree_to_parent_node(). That function always returns the top
* *two* chaining values of the subtree it's compressing. We then do lazy
* merging with each of them separately, so that the second CV will always
* remain unmerged. (That also helps us support extendable output when we're
* hashing an input all-at-once.)
*/
static void hasher_push_cv(BLAKE3_CTX *ctx, uint8_t new_cv[BLAKE3_OUT_LEN],
uint64_t chunk_counter)
{
hasher_merge_cv_stack(ctx, chunk_counter);
memcpy(&ctx->cv_stack[ctx->cv_stack_len * BLAKE3_OUT_LEN], new_cv,
BLAKE3_OUT_LEN);
ctx->cv_stack_len += 1;
}
void
Blake3_Init(BLAKE3_CTX *ctx)
{
hasher_init_base(ctx, BLAKE3_IV, 0);
}
void
Blake3_InitKeyed(BLAKE3_CTX *ctx, const uint8_t key[BLAKE3_KEY_LEN])
{
uint32_t key_words[8];
load_key_words(key, key_words);
hasher_init_base(ctx, key_words, KEYED_HASH);
}
static void
Blake3_Update2(BLAKE3_CTX *ctx, const void *input, size_t input_len)
{
/*
* Explicitly checking for zero avoids causing UB by passing a null
* pointer to memcpy. This comes up in practice with things like:
* std::vector<uint8_t> v;
* blake3_hasher_update(&hasher, v.data(), v.size());
*/
if (input_len == 0) {
return;
}
const uint8_t *input_bytes = (const uint8_t *)input;
/*
* If we have some partial chunk bytes in the internal chunk_state, we
* need to finish that chunk first.
*/
if (chunk_state_len(&ctx->chunk) > 0) {
size_t take = BLAKE3_CHUNK_LEN - chunk_state_len(&ctx->chunk);
if (take > input_len) {
take = input_len;
}
chunk_state_update(ctx->ops, &ctx->chunk, input_bytes, take);
input_bytes += take;
input_len -= take;
/*
* If we've filled the current chunk and there's more coming,
* finalize this chunk and proceed. In this case we know it's
* not the root.
*/
if (input_len > 0) {
output_t output = chunk_state_output(&ctx->chunk);
uint8_t chunk_cv[32];
output_chaining_value(ctx->ops, &output, chunk_cv);
hasher_push_cv(ctx, chunk_cv, ctx->chunk.chunk_counter);
chunk_state_reset(&ctx->chunk, ctx->key,
ctx->chunk.chunk_counter + 1);
} else {
return;
}
}
/*
* Now the chunk_state is clear, and we have more input. If there's
* more than a single chunk (so, definitely not the root chunk), hash
* the largest whole subtree we can, with the full benefits of SIMD
* (and maybe in the future, multi-threading) parallelism. Two
* restrictions:
* - The subtree has to be a power-of-2 number of chunks. Only
* subtrees along the right edge can be incomplete, and we don't know
* where the right edge is going to be until we get to finalize().
* - The subtree must evenly divide the total number of chunks up
* until this point (if total is not 0). If the current incomplete
* subtree is only waiting for 1 more chunk, we can't hash a subtree
* of 4 chunks. We have to complete the current subtree first.
* Because we might need to break up the input to form powers of 2, or
* to evenly divide what we already have, this part runs in a loop.
*/
while (input_len > BLAKE3_CHUNK_LEN) {
size_t subtree_len = round_down_to_power_of_2(input_len);
uint64_t count_so_far =
ctx->chunk.chunk_counter * BLAKE3_CHUNK_LEN;
/*
* Shrink the subtree_len until it evenly divides the count so
* far. We know that subtree_len itself is a power of 2, so we
* can use a bitmasking trick instead of an actual remainder
* operation. (Note that if the caller consistently passes
* power-of-2 inputs of the same size, as is hopefully
* typical, this loop condition will always fail, and
* subtree_len will always be the full length of the input.)
*
* An aside: We don't have to shrink subtree_len quite this
* much. For example, if count_so_far is 1, we could pass 2
* chunks to compress_subtree_to_parent_node. Since we'll get
* 2 CVs back, we'll still get the right answer in the end,
* and we might get to use 2-way SIMD parallelism. The problem
* with this optimization, is that it gets us stuck always
* hashing 2 chunks. The total number of chunks will remain
* odd, and we'll never graduate to higher degrees of
* parallelism. See
* https://github.com/BLAKE3-team/BLAKE3/issues/69.
*/
while ((((uint64_t)(subtree_len - 1)) & count_so_far) != 0) {
subtree_len /= 2;
}
/*
* The shrunken subtree_len might now be 1 chunk long. If so,
* hash that one chunk by itself. Otherwise, compress the
* subtree into a pair of CVs.
*/
uint64_t subtree_chunks = subtree_len / BLAKE3_CHUNK_LEN;
if (subtree_len <= BLAKE3_CHUNK_LEN) {
blake3_chunk_state_t chunk_state;
chunk_state_init(&chunk_state, ctx->key,
ctx->chunk.flags);
chunk_state.chunk_counter = ctx->chunk.chunk_counter;
chunk_state_update(ctx->ops, &chunk_state, input_bytes,
subtree_len);
output_t output = chunk_state_output(&chunk_state);
uint8_t cv[BLAKE3_OUT_LEN];
output_chaining_value(ctx->ops, &output, cv);
hasher_push_cv(ctx, cv, chunk_state.chunk_counter);
} else {
/*
* This is the high-performance happy path, though
* getting here depends on the caller giving us a long
* enough input.
*/
uint8_t cv_pair[2 * BLAKE3_OUT_LEN];
compress_subtree_to_parent_node(ctx->ops, input_bytes,
subtree_len, ctx->key, ctx-> chunk.chunk_counter,
ctx->chunk.flags, cv_pair);
hasher_push_cv(ctx, cv_pair, ctx->chunk.chunk_counter);
hasher_push_cv(ctx, &cv_pair[BLAKE3_OUT_LEN],
ctx->chunk.chunk_counter + (subtree_chunks / 2));
}
ctx->chunk.chunk_counter += subtree_chunks;
input_bytes += subtree_len;
input_len -= subtree_len;
}
/*
* If there's any remaining input less than a full chunk, add it to
* the chunk state. In that case, also do a final merge loop to make
* sure the subtree stack doesn't contain any unmerged pairs. The
* remaining input means we know these merges are non-root. This merge
* loop isn't strictly necessary here, because hasher_push_chunk_cv
* already does its own merge loop, but it simplifies
* blake3_hasher_finalize below.
*/
if (input_len > 0) {
chunk_state_update(ctx->ops, &ctx->chunk, input_bytes,
input_len);
hasher_merge_cv_stack(ctx, ctx->chunk.chunk_counter);
}
}
void
Blake3_Update(BLAKE3_CTX *ctx, const void *input, size_t todo)
{
size_t done = 0;
const uint8_t *data = input;
const size_t block_max = 1024 * 64;
/* max feed buffer to leave the stack size small */
while (todo != 0) {
size_t block = (todo >= block_max) ? block_max : todo;
Blake3_Update2(ctx, data + done, block);
done += block;
todo -= block;
}
}
void
Blake3_Final(const BLAKE3_CTX *ctx, uint8_t *out)
{
Blake3_FinalSeek(ctx, 0, out, BLAKE3_OUT_LEN);
}
void
Blake3_FinalSeek(const BLAKE3_CTX *ctx, uint64_t seek, uint8_t *out,
size_t out_len)
{
/*
* Explicitly checking for zero avoids causing UB by passing a null
* pointer to memcpy. This comes up in practice with things like:
* std::vector<uint8_t> v;
* blake3_hasher_finalize(&hasher, v.data(), v.size());
*/
if (out_len == 0) {
return;
}
/* If the subtree stack is empty, then the current chunk is the root. */
if (ctx->cv_stack_len == 0) {
output_t output = chunk_state_output(&ctx->chunk);
output_root_bytes(ctx->ops, &output, seek, out, out_len);
return;
}
/*
* If there are any bytes in the chunk state, finalize that chunk and
* do a roll-up merge between that chunk hash and every subtree in the
* stack. In this case, the extra merge loop at the end of
* blake3_hasher_update guarantees that none of the subtrees in the
* stack need to be merged with each other first. Otherwise, if there
* are no bytes in the chunk state, then the top of the stack is a
* chunk hash, and we start the merge from that.
*/
output_t output;
size_t cvs_remaining;
if (chunk_state_len(&ctx->chunk) > 0) {
cvs_remaining = ctx->cv_stack_len;
output = chunk_state_output(&ctx->chunk);
} else {
/* There are always at least 2 CVs in the stack in this case. */
cvs_remaining = ctx->cv_stack_len - 2;
output = parent_output(&ctx->cv_stack[cvs_remaining * 32],
ctx->key, ctx->chunk.flags);
}
while (cvs_remaining > 0) {
cvs_remaining -= 1;
uint8_t parent_block[BLAKE3_BLOCK_LEN];
memcpy(parent_block, &ctx->cv_stack[cvs_remaining * 32], 32);
output_chaining_value(ctx->ops, &output, &parent_block[32]);
output = parent_output(parent_block, ctx->key,
ctx->chunk.flags);
}
output_root_bytes(ctx->ops, &output, seek, out, out_len);
}

202
module/icp/algs/blake3/blake3_generic.c Normal file
View File

@ -0,0 +1,202 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include "blake3_impl.h"
#define rotr32(x, n) (((x) >> (n)) | ((x) << (32 - (n))))
static inline void g(uint32_t *state, size_t a, size_t b, size_t c, size_t d,
uint32_t x, uint32_t y)
{
state[a] = state[a] + state[b] + x;
state[d] = rotr32(state[d] ^ state[a], 16);
state[c] = state[c] + state[d];
state[b] = rotr32(state[b] ^ state[c], 12);
state[a] = state[a] + state[b] + y;
state[d] = rotr32(state[d] ^ state[a], 8);
state[c] = state[c] + state[d];
state[b] = rotr32(state[b] ^ state[c], 7);
}
static inline void round_fn(uint32_t state[16], const uint32_t *msg,
size_t round)
{
/* Select the message schedule based on the round. */
const uint8_t *schedule = BLAKE3_MSG_SCHEDULE[round];
/* Mix the columns. */
g(state, 0, 4, 8, 12, msg[schedule[0]], msg[schedule[1]]);
g(state, 1, 5, 9, 13, msg[schedule[2]], msg[schedule[3]]);
g(state, 2, 6, 10, 14, msg[schedule[4]], msg[schedule[5]]);
g(state, 3, 7, 11, 15, msg[schedule[6]], msg[schedule[7]]);
/* Mix the rows. */
g(state, 0, 5, 10, 15, msg[schedule[8]], msg[schedule[9]]);
g(state, 1, 6, 11, 12, msg[schedule[10]], msg[schedule[11]]);
g(state, 2, 7, 8, 13, msg[schedule[12]], msg[schedule[13]]);
g(state, 3, 4, 9, 14, msg[schedule[14]], msg[schedule[15]]);
}
static inline void compress_pre(uint32_t state[16], const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags)
{
uint32_t block_words[16];
block_words[0] = load32(block + 4 * 0);
block_words[1] = load32(block + 4 * 1);
block_words[2] = load32(block + 4 * 2);
block_words[3] = load32(block + 4 * 3);
block_words[4] = load32(block + 4 * 4);
block_words[5] = load32(block + 4 * 5);
block_words[6] = load32(block + 4 * 6);
block_words[7] = load32(block + 4 * 7);
block_words[8] = load32(block + 4 * 8);
block_words[9] = load32(block + 4 * 9);
block_words[10] = load32(block + 4 * 10);
block_words[11] = load32(block + 4 * 11);
block_words[12] = load32(block + 4 * 12);
block_words[13] = load32(block + 4 * 13);
block_words[14] = load32(block + 4 * 14);
block_words[15] = load32(block + 4 * 15);
state[0] = cv[0];
state[1] = cv[1];
state[2] = cv[2];
state[3] = cv[3];
state[4] = cv[4];
state[5] = cv[5];
state[6] = cv[6];
state[7] = cv[7];
state[8] = BLAKE3_IV[0];
state[9] = BLAKE3_IV[1];
state[10] = BLAKE3_IV[2];
state[11] = BLAKE3_IV[3];
state[12] = counter_low(counter);
state[13] = counter_high(counter);
state[14] = (uint32_t)block_len;
state[15] = (uint32_t)flags;
round_fn(state, &block_words[0], 0);
round_fn(state, &block_words[0], 1);
round_fn(state, &block_words[0], 2);
round_fn(state, &block_words[0], 3);
round_fn(state, &block_words[0], 4);
round_fn(state, &block_words[0], 5);
round_fn(state, &block_words[0], 6);
}
static inline void blake3_compress_in_place_generic(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags)
{
uint32_t state[16];
compress_pre(state, cv, block, block_len, counter, flags);
cv[0] = state[0] ^ state[8];
cv[1] = state[1] ^ state[9];
cv[2] = state[2] ^ state[10];
cv[3] = state[3] ^ state[11];
cv[4] = state[4] ^ state[12];
cv[5] = state[5] ^ state[13];
cv[6] = state[6] ^ state[14];
cv[7] = state[7] ^ state[15];
}
static inline void hash_one_generic(const uint8_t *input, size_t blocks,
const uint32_t key[8], uint64_t counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t out[BLAKE3_OUT_LEN])
{
uint32_t cv[8];
memcpy(cv, key, BLAKE3_KEY_LEN);
uint8_t block_flags = flags | flags_start;
while (blocks > 0) {
if (blocks == 1) {
block_flags |= flags_end;
}
blake3_compress_in_place_generic(cv, input, BLAKE3_BLOCK_LEN,
counter, block_flags);
input = &input[BLAKE3_BLOCK_LEN];
blocks -= 1;
block_flags = flags;
}
store_cv_words(out, cv);
}
static inline void blake3_compress_xof_generic(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64])
{
uint32_t state[16];
compress_pre(state, cv, block, block_len, counter, flags);
store32(&out[0 * 4], state[0] ^ state[8]);
store32(&out[1 * 4], state[1] ^ state[9]);
store32(&out[2 * 4], state[2] ^ state[10]);
store32(&out[3 * 4], state[3] ^ state[11]);
store32(&out[4 * 4], state[4] ^ state[12]);
store32(&out[5 * 4], state[5] ^ state[13]);
store32(&out[6 * 4], state[6] ^ state[14]);
store32(&out[7 * 4], state[7] ^ state[15]);
store32(&out[8 * 4], state[8] ^ cv[0]);
store32(&out[9 * 4], state[9] ^ cv[1]);
store32(&out[10 * 4], state[10] ^ cv[2]);
store32(&out[11 * 4], state[11] ^ cv[3]);
store32(&out[12 * 4], state[12] ^ cv[4]);
store32(&out[13 * 4], state[13] ^ cv[5]);
store32(&out[14 * 4], state[14] ^ cv[6]);
store32(&out[15 * 4], state[15] ^ cv[7]);
}
static inline void blake3_hash_many_generic(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8], uint64_t counter,
boolean_t increment_counter, uint8_t flags, uint8_t flags_start,
uint8_t flags_end, uint8_t *out)
{
while (num_inputs > 0) {
hash_one_generic(inputs[0], blocks, key, counter, flags,
flags_start, flags_end, out);
if (increment_counter) {
counter += 1;
}
inputs += 1;
num_inputs -= 1;
out = &out[BLAKE3_OUT_LEN];
}
}
static inline boolean_t blake3_is_generic_supported(void)
{
return (B_TRUE);
}
const blake3_impl_ops_t blake3_generic_impl = {
.compress_in_place = blake3_compress_in_place_generic,
.compress_xof = blake3_compress_xof_generic,
.hash_many = blake3_hash_many_generic,
.is_supported = blake3_is_generic_supported,
.degree = 4,
.name = "generic"
};

256
module/icp/algs/blake3/blake3_impl.c Normal file
View File

@ -0,0 +1,256 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/zio_checksum.h>
#include "blake3_impl.h"
static const blake3_impl_ops_t *const blake3_impls[] = {
&blake3_generic_impl,
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
&blake3_sse2_impl,
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE4_1)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
&blake3_sse41_impl,
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
&blake3_avx2_impl,
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
&blake3_avx512_impl,
#endif
};
/* this pointer holds current ops for implementation */
static const blake3_impl_ops_t *blake3_selected_impl = &blake3_generic_impl;
/* special implementation selections */
#define IMPL_FASTEST (UINT32_MAX)
#define IMPL_CYCLE (UINT32_MAX-1)
#define IMPL_USER (UINT32_MAX-2)
#define IMPL_PARAM (UINT32_MAX-3)
#define IMPL_READ(i) (*(volatile uint32_t *) &(i))
static uint32_t icp_blake3_impl = IMPL_FASTEST;
#define BLAKE3_IMPL_NAME_MAX 16
/* id of fastest implementation */
static uint32_t blake3_fastest_id = 0;
/* currently used id */
static uint32_t blake3_current_id = 0;
/* id of module parameter (-1 == unused) */
static int blake3_param_id = -1;
/* return number of supported implementations */
int
blake3_get_impl_count(void)
{
static int impls = 0;
int i;
if (impls)
return (impls);
for (i = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
impls++;
}
return (impls);
}
/* return id of selected implementation */
int
blake3_get_impl_id(void)
{
return (blake3_current_id);
}
/* return name of selected implementation */
const char *
blake3_get_impl_name(void)
{
return (blake3_selected_impl->name);
}
/* setup id as fastest implementation */
void
blake3_set_impl_fastest(uint32_t id)
{
blake3_fastest_id = id;
}
/* set implementation by id */
void
blake3_set_impl_id(uint32_t id)
{
int i, cid;
/* select fastest */
if (id == IMPL_FASTEST)
id = blake3_fastest_id;
/* select next or first */
if (id == IMPL_CYCLE)
id = (++blake3_current_id) % blake3_get_impl_count();
/* 0..N for the real impl */
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
if (cid == id) {
blake3_current_id = cid;
blake3_selected_impl = blake3_impls[i];
return;
}
cid++;
}
}
/* set implementation by name */
int
blake3_set_impl_name(const char *name)
{
int i, cid;
if (strcmp(name, "fastest") == 0) {
atomic_swap_32(&icp_blake3_impl, IMPL_FASTEST);
blake3_set_impl_id(IMPL_FASTEST);
return (0);
} else if (strcmp(name, "cycle") == 0) {
atomic_swap_32(&icp_blake3_impl, IMPL_CYCLE);
blake3_set_impl_id(IMPL_CYCLE);
return (0);
}
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
if (strcmp(name, blake3_impls[i]->name) == 0) {
if (icp_blake3_impl == IMPL_PARAM) {
blake3_param_id = cid;
return (0);
}
blake3_selected_impl = blake3_impls[i];
blake3_current_id = cid;
return (0);
}
cid++;
}
return (-EINVAL);
}
/* setup implementation */
void
blake3_setup_impl(void)
{
switch (IMPL_READ(icp_blake3_impl)) {
case IMPL_PARAM:
blake3_set_impl_id(blake3_param_id);
atomic_swap_32(&icp_blake3_impl, IMPL_USER);
break;
case IMPL_FASTEST:
blake3_set_impl_id(IMPL_FASTEST);
break;
case IMPL_CYCLE:
blake3_set_impl_id(IMPL_CYCLE);
break;
default:
blake3_set_impl_id(blake3_current_id);
break;
}
}
/* return selected implementation */
const blake3_impl_ops_t *
blake3_impl_get_ops(void)
{
/* each call to ops will cycle */
if (icp_blake3_impl == IMPL_CYCLE)
blake3_set_impl_id(IMPL_CYCLE);
return (blake3_selected_impl);
}
#if defined(_KERNEL) && defined(__linux__)
static int
icp_blake3_impl_set(const char *name, zfs_kernel_param_t *kp)
{
char req_name[BLAKE3_IMPL_NAME_MAX];
size_t i;
/* sanitize input */
i = strnlen(name, BLAKE3_IMPL_NAME_MAX);
if (i == 0 || i >= BLAKE3_IMPL_NAME_MAX)
return (-EINVAL);
strlcpy(req_name, name, BLAKE3_IMPL_NAME_MAX);
while (i > 0 && isspace(req_name[i-1]))
i--;
req_name[i] = '\0';
atomic_swap_32(&icp_blake3_impl, IMPL_PARAM);
return (blake3_set_impl_name(req_name));
}
static int
icp_blake3_impl_get(char *buffer, zfs_kernel_param_t *kp)
{
int i, cid, cnt = 0;
char *fmt;
/* cycling */
fmt = (icp_blake3_impl == IMPL_CYCLE) ? "[cycle] " : "cycle ";
cnt += sprintf(buffer + cnt, fmt);
/* fastest one */
fmt = (icp_blake3_impl == IMPL_FASTEST) ? "[fastest] " : "fastest ";
cnt += sprintf(buffer + cnt, fmt);
/* user selected */
for (i = 0, cid = 0; i < ARRAY_SIZE(blake3_impls); i++) {
if (!blake3_impls[i]->is_supported()) continue;
fmt = (icp_blake3_impl == IMPL_USER &&
cid == blake3_current_id) ? "[%s] " : "%s ";
cnt += sprintf(buffer + cnt, fmt, blake3_impls[i]->name);
cid++;
}
buffer[cnt] = 0;
return (cnt);
}
module_param_call(icp_blake3_impl, icp_blake3_impl_set, icp_blake3_impl_get,
NULL, 0644);
MODULE_PARM_DESC(icp_blake3_impl, "Select BLAKE3 implementation.");
#endif

213
module/icp/algs/blake3/blake3_impl.h Normal file
View File

@ -0,0 +1,213 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Based on BLAKE3 v1.3.1, https://github.com/BLAKE3-team/BLAKE3
* Copyright (c) 2019-2020 Samuel Neves and Jack O'Connor
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#ifndef BLAKE3_IMPL_H
#define BLAKE3_IMPL_H
#ifdef __cplusplus
extern "C" {
#endif
#include <sys/types.h>
#include <sys/blake3.h>
#include <sys/simd.h>
/*
* Methods used to define BLAKE3 assembler implementations
*/
typedef void (*blake3_compress_in_place_f)(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN],
uint8_t block_len, uint64_t counter,
uint8_t flags);
typedef void (*blake3_compress_xof_f)(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
typedef void (*blake3_hash_many_f)(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
typedef boolean_t (*blake3_is_supported_f)(void);
typedef struct blake3_impl_ops {
blake3_compress_in_place_f compress_in_place;
blake3_compress_xof_f compress_xof;
blake3_hash_many_f hash_many;
blake3_is_supported_f is_supported;
int degree;
const char *name;
} blake3_impl_ops_t;
/* Return selected BLAKE3 implementation ops */
extern const blake3_impl_ops_t *blake3_impl_get_ops(void);
extern const blake3_impl_ops_t blake3_generic_impl;
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern const blake3_impl_ops_t blake3_sse2_impl;
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE4_1)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern const blake3_impl_ops_t blake3_sse41_impl;
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
extern const blake3_impl_ops_t blake3_avx2_impl;
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
extern const blake3_impl_ops_t blake3_avx512_impl;
#endif
#if defined(__x86_64)
#define MAX_SIMD_DEGREE 16
#else
#define MAX_SIMD_DEGREE 4
#endif
#define MAX_SIMD_DEGREE_OR_2 (MAX_SIMD_DEGREE > 2 ? MAX_SIMD_DEGREE : 2)
static const uint32_t BLAKE3_IV[8] = {
0x6A09E667UL, 0xBB67AE85UL, 0x3C6EF372UL, 0xA54FF53AUL,
0x510E527FUL, 0x9B05688CUL, 0x1F83D9ABUL, 0x5BE0CD19UL};
static const uint8_t BLAKE3_MSG_SCHEDULE[7][16] = {
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15},
{2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8},
{3, 4, 10, 12, 13, 2, 7, 14, 6, 5, 9, 0, 11, 15, 8, 1},
{10, 7, 12, 9, 14, 3, 13, 15, 4, 0, 11, 2, 5, 8, 1, 6},
{12, 13, 9, 11, 15, 10, 14, 8, 7, 2, 5, 3, 0, 1, 6, 4},
{9, 14, 11, 5, 8, 12, 15, 1, 13, 3, 0, 10, 2, 6, 4, 7},
{11, 15, 5, 0, 1, 9, 8, 6, 14, 10, 2, 12, 3, 4, 7, 13},
};
/* Find index of the highest set bit */
static inline unsigned int highest_one(uint64_t x) {
#if defined(__GNUC__) || defined(__clang__)
return (63 ^ __builtin_clzll(x));
#elif defined(_MSC_VER) && defined(IS_X86_64)
unsigned long index;
_BitScanReverse64(&index, x);
return (index);
#elif defined(_MSC_VER) && defined(IS_X86_32)
if (x >> 32) {
unsigned long index;
_BitScanReverse(&index, x >> 32);
return (32 + index);
} else {
unsigned long index;
_BitScanReverse(&index, x);
return (index);
}
#else
unsigned int c = 0;
if (x & 0xffffffff00000000ULL) { x >>= 32; c += 32; }
if (x & 0x00000000ffff0000ULL) { x >>= 16; c += 16; }
if (x & 0x000000000000ff00ULL) { x >>= 8; c += 8; }
if (x & 0x00000000000000f0ULL) { x >>= 4; c += 4; }
if (x & 0x000000000000000cULL) { x >>= 2; c += 2; }
if (x & 0x0000000000000002ULL) { c += 1; }
return (c);
#endif
}
/* Count the number of 1 bits. */
static inline unsigned int popcnt(uint64_t x) {
unsigned int count = 0;
while (x != 0) {
count += 1;
x &= x - 1;
}
return (count);
}
/*
* Largest power of two less than or equal to x.
* As a special case, returns 1 when x is 0.
*/
static inline uint64_t round_down_to_power_of_2(uint64_t x) {
return (1ULL << highest_one(x | 1));
}
static inline uint32_t counter_low(uint64_t counter) {
return ((uint32_t)counter);
}
static inline uint32_t counter_high(uint64_t counter) {
return ((uint32_t)(counter >> 32));
}
static inline uint32_t load32(const void *src) {
const uint8_t *p = (const uint8_t *)src;
return ((uint32_t)(p[0]) << 0) | ((uint32_t)(p[1]) << 8) |
((uint32_t)(p[2]) << 16) | ((uint32_t)(p[3]) << 24);
}
static inline void load_key_words(const uint8_t key[BLAKE3_KEY_LEN],
uint32_t key_words[8]) {
key_words[0] = load32(&key[0 * 4]);
key_words[1] = load32(&key[1 * 4]);
key_words[2] = load32(&key[2 * 4]);
key_words[3] = load32(&key[3 * 4]);
key_words[4] = load32(&key[4 * 4]);
key_words[5] = load32(&key[5 * 4]);
key_words[6] = load32(&key[6 * 4]);
key_words[7] = load32(&key[7 * 4]);
}
static inline void store32(void *dst, uint32_t w) {
uint8_t *p = (uint8_t *)dst;
p[0] = (uint8_t)(w >> 0);
p[1] = (uint8_t)(w >> 8);
p[2] = (uint8_t)(w >> 16);
p[3] = (uint8_t)(w >> 24);
}
static inline void store_cv_words(uint8_t bytes_out[32], uint32_t cv_words[8]) {
store32(&bytes_out[0 * 4], cv_words[0]);
store32(&bytes_out[1 * 4], cv_words[1]);
store32(&bytes_out[2 * 4], cv_words[2]);
store32(&bytes_out[3 * 4], cv_words[3]);
store32(&bytes_out[4 * 4], cv_words[4]);
store32(&bytes_out[5 * 4], cv_words[5]);
store32(&bytes_out[6 * 4], cv_words[6]);
store32(&bytes_out[7 * 4], cv_words[7]);
}
#ifdef __cplusplus
}
#endif
#endif /* BLAKE3_IMPL_H */

248
module/icp/algs/blake3/blake3_x86-64.c Normal file
View File

@ -0,0 +1,248 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021-2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include "blake3_impl.h"
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern void zfs_blake3_compress_in_place_sse2(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_sse2(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_sse2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_sse2(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_sse2(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_sse2(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_sse2(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_sse2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_sse2(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_sse2_supported(void)
{
#if defined(__x86_64)
return (kfpu_allowed() && zfs_sse2_available());
#elif defined(__PPC64__)
return (kfpu_allowed() && zfs_vsx_available());
#else
return (kfpu_allowed());
#endif
}
const blake3_impl_ops_t blake3_sse2_impl = {
.compress_in_place = blake3_compress_in_place_sse2,
.compress_xof = blake3_compress_xof_sse2,
.hash_many = blake3_hash_many_sse2,
.is_supported = blake3_is_sse2_supported,
.degree = 4,
.name = "sse2"
};
#endif
#if defined(__aarch64__) || \
(defined(__x86_64) && defined(HAVE_SSE2)) || \
(defined(__PPC64__) && defined(__LITTLE_ENDIAN__))
extern void zfs_blake3_compress_in_place_sse41(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_sse41(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_sse41(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_sse41(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_sse41(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_sse41(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_sse41(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_sse41(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_sse41(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_sse41_supported(void)
{
#if defined(__x86_64)
return (kfpu_allowed() && zfs_sse4_1_available());
#elif defined(__PPC64__)
return (kfpu_allowed() && zfs_vsx_available());
#else
return (kfpu_allowed());
#endif
}
const blake3_impl_ops_t blake3_sse41_impl = {
.compress_in_place = blake3_compress_in_place_sse41,
.compress_xof = blake3_compress_xof_sse41,
.hash_many = blake3_hash_many_sse41,
.is_supported = blake3_is_sse41_supported,
.degree = 4,
.name = "sse41"
};
#endif
#if defined(__x86_64) && defined(HAVE_SSE4_1) && defined(HAVE_AVX2)
extern void zfs_blake3_hash_many_avx2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_hash_many_avx2(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_avx2(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_avx2_supported(void)
{
return (kfpu_allowed() && zfs_sse4_1_available() &&
zfs_avx2_available());
}
const blake3_impl_ops_t blake3_avx2_impl = {
.compress_in_place = blake3_compress_in_place_sse41,
.compress_xof = blake3_compress_xof_sse41,
.hash_many = blake3_hash_many_avx2,
.is_supported = blake3_is_avx2_supported,
.degree = 8,
.name = "avx2"
};
#endif
#if defined(__x86_64) && defined(HAVE_AVX512F) && defined(HAVE_AVX512VL)
extern void zfs_blake3_compress_in_place_avx512(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags);
extern void zfs_blake3_compress_xof_avx512(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]);
extern void zfs_blake3_hash_many_avx512(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out);
static void blake3_compress_in_place_avx512(uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags) {
kfpu_begin();
zfs_blake3_compress_in_place_avx512(cv, block, block_len, counter,
flags);
kfpu_end();
}
static void blake3_compress_xof_avx512(const uint32_t cv[8],
const uint8_t block[BLAKE3_BLOCK_LEN], uint8_t block_len,
uint64_t counter, uint8_t flags, uint8_t out[64]) {
kfpu_begin();
zfs_blake3_compress_xof_avx512(cv, block, block_len, counter, flags,
out);
kfpu_end();
}
static void blake3_hash_many_avx512(const uint8_t * const *inputs,
size_t num_inputs, size_t blocks, const uint32_t key[8],
uint64_t counter, boolean_t increment_counter, uint8_t flags,
uint8_t flags_start, uint8_t flags_end, uint8_t *out) {
kfpu_begin();
zfs_blake3_hash_many_avx512(inputs, num_inputs, blocks, key, counter,
increment_counter, flags, flags_start, flags_end, out);
kfpu_end();
}
static boolean_t blake3_is_avx512_supported(void)
{
return (kfpu_allowed() && zfs_avx512f_available() &&
zfs_avx512vl_available());
}
const blake3_impl_ops_t blake3_avx512_impl = {
.compress_in_place = blake3_compress_in_place_avx512,
.compress_xof = blake3_compress_xof_avx512,
.hash_many = blake3_hash_many_avx512,
.is_supported = blake3_is_avx512_supported,
.degree = 16,
.name = "avx512"
};
#endif

2450
module/icp/asm-aarch64/blake3/b3_aarch64_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2463
module/icp/asm-aarch64/blake3/b3_aarch64_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

2823
module/icp/asm-ppc64/blake3/b3_ppc64le_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

3064
module/icp/asm-ppc64/blake3/b3_ppc64le_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

1845
module/icp/asm-x86_64/blake3/blake3_avx2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2618
module/icp/asm-x86_64/blake3/blake3_avx512.S Normal file
View File

File diff suppressed because it is too large Load Diff

2323
module/icp/asm-x86_64/blake3/blake3_sse2.S Normal file
View File

File diff suppressed because it is too large Load Diff

2058
module/icp/asm-x86_64/blake3/blake3_sse41.S Normal file
View File

File diff suppressed because it is too large Load Diff

13
module/zcommon/zfeature_common.c
View File

@ -696,7 +696,6 @@ zpool_feature_init(void)
ZFEATURE_FLAG_MOS, ZFEATURE_TYPE_BOOLEAN, NULL, sfeatures);
{
static const spa_feature_t zilsaxattr_deps[] = {
SPA_FEATURE_EXTENSIBLE_DATASET,
SPA_FEATURE_NONE
@ -714,6 +713,18 @@ zpool_feature_init(void)
ZFEATURE_FLAG_ACTIVATE_ON_ENABLE, ZFEATURE_TYPE_BOOLEAN, NULL,
sfeatures);
{
static const spa_feature_t blake3_deps[] = {
SPA_FEATURE_EXTENSIBLE_DATASET,
SPA_FEATURE_NONE
};
zfeature_register(SPA_FEATURE_BLAKE3,
"org.openzfs:blake3", "blake3",
"BLAKE3 hash algorithm.",
ZFEATURE_FLAG_PER_DATASET, ZFEATURE_TYPE_BOOLEAN,
blake3_deps, sfeatures);
}
zfs_mod_list_supported_free(sfeatures);
}

8
module/zcommon/zfs_prop.c
View File

@ -84,6 +84,7 @@ zfs_prop_init(void)
{ "sha512", ZIO_CHECKSUM_SHA512 },
{ "skein", ZIO_CHECKSUM_SKEIN },
{ "edonr", ZIO_CHECKSUM_EDONR },
{ "blake3", ZIO_CHECKSUM_BLAKE3 },
{ NULL }
};
@ -102,6 +103,9 @@ zfs_prop_init(void)
ZIO_CHECKSUM_SKEIN | ZIO_CHECKSUM_VERIFY },
{ "edonr,verify",
ZIO_CHECKSUM_EDONR | ZIO_CHECKSUM_VERIFY },
{ "blake3", ZIO_CHECKSUM_BLAKE3 },
{ "blake3,verify",
ZIO_CHECKSUM_BLAKE3 | ZIO_CHECKSUM_VERIFY },
{ NULL }
};
@ -394,12 +398,12 @@ zfs_prop_init(void)
ZIO_CHECKSUM_DEFAULT, PROP_INHERIT, ZFS_TYPE_FILESYSTEM |
ZFS_TYPE_VOLUME,
"on | off | fletcher2 | fletcher4 | sha256 | sha512 | skein"
" | edonr",
" | edonr | blake3",
"CHECKSUM", checksum_table, sfeatures);
zprop_register_index(ZFS_PROP_DEDUP, "dedup", ZIO_CHECKSUM_OFF,
PROP_INHERIT, ZFS_TYPE_FILESYSTEM | ZFS_TYPE_VOLUME,
"on | off | verify | sha256[,verify] | sha512[,verify] | "
"skein[,verify] | edonr,verify",
"skein[,verify] | edonr,verify | blake3[,verify]",
"DEDUP", dedup_table, sfeatures);
zprop_register_index(ZFS_PROP_COMPRESSION, "compression",
ZIO_COMPRESS_DEFAULT, PROP_INHERIT,

113
module/zfs/blake3_zfs.c Normal file
View File

@ -0,0 +1,113 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://opensource.org/licenses/CDDL-1.0.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright 2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/zfs_context.h>
#include <sys/zio_checksum.h>
#include <sys/blake3.h>
#include <sys/abd.h>
static int
blake3_incremental(void *buf, size_t size, void *arg)
{
BLAKE3_CTX *ctx = arg;
Blake3_Update(ctx, buf, size);
return (0);
}
/*
* Computes a native 256-bit BLAKE3 MAC checksum. Please note that this
* function requires the presence of a ctx_template that should be allocated
* using abd_checksum_blake3_tmpl_init.
*/
void
abd_checksum_blake3_native(abd_t *abd, uint64_t size, const void *ctx_template,
zio_cksum_t *zcp)
{
BLAKE3_CTX *ctx;
ctx = kmem_alloc(sizeof (*ctx), KM_NOSLEEP);
ASSERT(ctx != 0);
ASSERT(ctx_template != 0);
memcpy(ctx, ctx_template, sizeof (*ctx));
(void) abd_iterate_func(abd, 0, size, blake3_incremental, ctx);
Blake3_Final(ctx, (uint8_t *)zcp);
memset(ctx, 0, sizeof (*ctx));
kmem_free(ctx, sizeof (*ctx));
}
/*
* Byteswapped version of abd_checksum_blake3_native. This just invokes
* the native checksum function and byteswaps the resulting checksum (since
* BLAKE3 is internally endian-insensitive).
*/
void
abd_checksum_blake3_byteswap(abd_t *abd, uint64_t size,
const void *ctx_template, zio_cksum_t *zcp)
{
zio_cksum_t tmp;
ASSERT(ctx_template != 0);
abd_checksum_blake3_native(abd, size, ctx_template, &tmp);
zcp->zc_word[0] = BSWAP_64(tmp.zc_word[0]);
zcp->zc_word[1] = BSWAP_64(tmp.zc_word[1]);
zcp->zc_word[2] = BSWAP_64(tmp.zc_word[2]);
zcp->zc_word[3] = BSWAP_64(tmp.zc_word[3]);
}
/*
* Allocates a BLAKE3 MAC template suitable for using in BLAKE3 MAC checksum
* computations and returns a pointer to it.
*/
void *
abd_checksum_blake3_tmpl_init(const zio_cksum_salt_t *salt)
{
BLAKE3_CTX *ctx;
ASSERT(sizeof (salt->zcs_bytes) == 32);
/* init reference object */
ctx = kmem_zalloc(sizeof (*ctx), KM_SLEEP);
Blake3_InitKeyed(ctx, salt->zcs_bytes);
return (ctx);
}
/*
* Frees a BLAKE3 context template previously allocated using
* zio_checksum_blake3_tmpl_init.
*/
void
abd_checksum_blake3_tmpl_free(void *ctx_template)
{
BLAKE3_CTX *ctx = ctx_template;
memset(ctx, 0, sizeof (*ctx));
kmem_free(ctx, sizeof (*ctx));
}

3
module/zfs/spa_misc.c
View File

@ -30,6 +30,7 @@
*/
#include <sys/zfs_context.h>
#include <sys/zfs_chksum.h>
#include <sys/spa_impl.h>
#include <sys/zio.h>
#include <sys/zio_checksum.h>
@ -2417,6 +2418,7 @@ spa_init(spa_mode_t mode)
vdev_raidz_math_init();
vdev_file_init();
zfs_prop_init();
chksum_init();
zpool_prop_init();
zpool_feature_init();
spa_config_load();
@ -2438,6 +2440,7 @@ spa_fini(void)
vdev_cache_stat_fini();
vdev_mirror_stat_fini();
vdev_raidz_math_fini();
chksum_fini();
zil_fini();
dmu_fini();
zio_fini();

316
module/zfs/zfs_chksum.c Normal file
View File

@ -0,0 +1,316 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2021 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <sys/types.h>
#include <sys/spa.h>
#include <sys/zio_checksum.h>
#include <sys/zfs_context.h>
#include <sys/zfs_chksum.h>
#include <sys/blake3.h>
static kstat_t *chksum_kstat = NULL;
typedef struct {
const char *name;
const char *impl;
uint64_t bs1k;
uint64_t bs4k;
uint64_t bs16k;
uint64_t bs64k;
uint64_t bs256k;
uint64_t bs1m;
uint64_t bs4m;
zio_cksum_salt_t salt;
zio_checksum_t *(func);
zio_checksum_tmpl_init_t *(init);
zio_checksum_tmpl_free_t *(free);
} chksum_stat_t;
static int chksum_stat_cnt = 0;
static chksum_stat_t *chksum_stat_data = 0;
/*
* i3-1005G1 test output:
*
* implementation 1k 4k 16k 64k 256k 1m 4m
* fletcher-4 5421 15001 26468 32555 34720 32801 18847
* edonr-generic 1196 1602 1761 1749 1762 1759 1751
* skein-generic 546 591 608 615 619 612 616
* sha256-generic 246 270 274 274 277 275 276
* sha256-avx 262 296 304 307 307 307 306
* sha256-sha-ni 769 1072 1172 1220 1219 1232 1228
* sha256-openssl 240 300 316 314 304 285 276
* sha512-generic 333 374 385 392 391 393 392
* sha512-openssl 353 441 467 476 472 467 426
* sha512-avx 362 444 473 475 479 476 478
* sha512-avx2 394 500 530 538 543 545 542
* blake3-generic 308 313 313 313 312 313 312
* blake3-sse2 402 1289 1423 1446 1432 1458 1413
* blake3-sse41 427 1470 1625 1704 1679 1607 1629
* blake3-avx2 428 1920 3095 3343 3356 3318 3204
* blake3-avx512 473 2687 4905 5836 5844 5643 5374
*/
static int
chksum_stat_kstat_headers(char *buf, size_t size)
{
ssize_t off = 0;
off += snprintf(buf + off, size, "%-23s", "implementation");
off += snprintf(buf + off, size - off, "%8s", "1k");
off += snprintf(buf + off, size - off, "%8s", "4k");
off += snprintf(buf + off, size - off, "%8s", "16k");
off += snprintf(buf + off, size - off, "%8s", "64k");
off += snprintf(buf + off, size - off, "%8s", "256k");
off += snprintf(buf + off, size - off, "%8s", "1m");
(void) snprintf(buf + off, size - off, "%8s\n", "4m");
return (0);
}
static int
chksum_stat_kstat_data(char *buf, size_t size, void *data)
{
chksum_stat_t *cs;
ssize_t off = 0;
char b[24];
cs = (chksum_stat_t *)data;
snprintf(b, 23, "%s-%s", cs->name, cs->impl);
off += snprintf(buf + off, size - off, "%-23s", b);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs1k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs4k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs16k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs64k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs256k);
off += snprintf(buf + off, size - off, "%8llu",
(u_longlong_t)cs->bs1m);
(void) snprintf(buf + off, size - off, "%8llu\n",
(u_longlong_t)cs->bs4m);
return (0);
}
static void *
chksum_stat_kstat_addr(kstat_t *ksp, loff_t n)
{
if (n < chksum_stat_cnt)
ksp->ks_private = (void *)(chksum_stat_data + n);
else
ksp->ks_private = NULL;
return (ksp->ks_private);
}
static void
chksum_run(chksum_stat_t *cs, abd_t *abd, void *ctx, int round,
uint64_t *result)
{
hrtime_t start;
uint64_t run_bw, run_time_ns, run_count = 0, size = 0;
uint32_t l, loops = 0;
zio_cksum_t zcp;
switch (round) {
case 1: /* 1k */
size = 1<<10; loops = 128; break;
case 2: /* 2k */
size = 1<<12; loops = 64; break;
case 3: /* 4k */
size = 1<<14; loops = 32; break;
case 4: /* 16k */
size = 1<<16; loops = 16; break;
case 5: /* 256k */
size = 1<<18; loops = 8; break;
case 6: /* 1m */
size = 1<<20; loops = 4; break;
case 7: /* 4m */
size = 1<<22; loops = 1; break;
}
kpreempt_disable();
start = gethrtime();
do {
for (l = 0; l < loops; l++, run_count++)
cs->func(abd, size, ctx, &zcp);
run_time_ns = gethrtime() - start;
} while (run_time_ns < MSEC2NSEC(1));
kpreempt_enable();
run_bw = size * run_count * NANOSEC;
run_bw /= run_time_ns; /* B/s */
*result = run_bw/1024/1024; /* MiB/s */
}
static void
chksum_benchit(chksum_stat_t *cs)
{
abd_t *abd;
void *ctx = 0;
void *salt = &cs->salt.zcs_bytes;
/* allocate test memory via default abd interface */
abd = abd_alloc_linear(1<<22, B_FALSE);
memset(salt, 0, sizeof (cs->salt.zcs_bytes));
if (cs->init) {
ctx = cs->init(&cs->salt);
}
chksum_run(cs, abd, ctx, 1, &cs->bs1k);
chksum_run(cs, abd, ctx, 2, &cs->bs4k);
chksum_run(cs, abd, ctx, 3, &cs->bs16k);
chksum_run(cs, abd, ctx, 4, &cs->bs64k);
chksum_run(cs, abd, ctx, 5, &cs->bs256k);
chksum_run(cs, abd, ctx, 6, &cs->bs1m);
chksum_run(cs, abd, ctx, 7, &cs->bs4m);
/* free up temp memory */
if (cs->free) {
cs->free(ctx);
}
abd_free(abd);
}
/*
* Initialize and benchmark all supported implementations.
*/
static void
chksum_benchmark(void)
{
#ifndef _KERNEL
/* we need the benchmark only for the kernel module */
return;
#endif
chksum_stat_t *cs;
int cbid = 0, id;
uint64_t max = 0;
/* space for the benchmark times */
chksum_stat_cnt = 4;
chksum_stat_cnt += blake3_get_impl_count();
chksum_stat_data = (chksum_stat_t *)kmem_zalloc(
sizeof (chksum_stat_t) * chksum_stat_cnt, KM_SLEEP);
/* edonr */
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_edonr_tmpl_init;
cs->func = abd_checksum_edonr_native;
cs->free = abd_checksum_edonr_tmpl_free;
cs->name = "edonr";
cs->impl = "generic";
chksum_benchit(cs);
/* skein */
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_skein_tmpl_init;
cs->func = abd_checksum_skein_native;
cs->free = abd_checksum_skein_tmpl_free;
cs->name = "skein";
cs->impl = "generic";
chksum_benchit(cs);
/* sha256 */
cs = &chksum_stat_data[cbid++];
cs->init = 0;
cs->func = abd_checksum_SHA256;
cs->free = 0;
cs->name = "sha256";
cs->impl = "generic";
chksum_benchit(cs);
/* sha512 */
cs = &chksum_stat_data[cbid++];
cs->init = 0;
cs->func = abd_checksum_SHA512_native;
cs->free = 0;
cs->name = "sha512";
cs->impl = "generic";
chksum_benchit(cs);
/* blake3 */
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
cs = &chksum_stat_data[cbid++];
cs->init = abd_checksum_blake3_tmpl_init;
cs->func = abd_checksum_blake3_native;
cs->free = abd_checksum_blake3_tmpl_free;
cs->name = "blake3";
cs->impl = blake3_get_impl_name();
chksum_benchit(cs);
if (cs->bs256k > max) {
max = cs->bs256k;
blake3_set_impl_fastest(id);
}
}
}
void
chksum_init(void)
{
/* Benchmark supported implementations */
chksum_benchmark();
/* Install kstats for all implementations */
chksum_kstat = kstat_create("zfs", 0, "chksum_bench", "misc",
KSTAT_TYPE_RAW, 0, KSTAT_FLAG_VIRTUAL);
if (chksum_kstat != NULL) {
chksum_kstat->ks_data = NULL;
chksum_kstat->ks_ndata = UINT32_MAX;
kstat_set_raw_ops(chksum_kstat,
chksum_stat_kstat_headers,
chksum_stat_kstat_data,
chksum_stat_kstat_addr);
kstat_install(chksum_kstat);
}
/* setup implementations */
blake3_setup_impl();
}
void
chksum_fini(void)
{
if (chksum_kstat != NULL) {
kstat_delete(chksum_kstat);
chksum_kstat = NULL;
}
if (chksum_stat_cnt) {
kmem_free(chksum_stat_data,
sizeof (chksum_stat_t) * chksum_stat_cnt);
chksum_stat_cnt = 0;
chksum_stat_data = 0;
}
}

6
module/zfs/zio_checksum.c
View File

@ -195,6 +195,10 @@ zio_checksum_info_t zio_checksum_table[ZIO_CHECKSUM_FUNCTIONS] = {
abd_checksum_edonr_tmpl_init, abd_checksum_edonr_tmpl_free,
ZCHECKSUM_FLAG_METADATA | ZCHECKSUM_FLAG_SALTED |
ZCHECKSUM_FLAG_NOPWRITE, "edonr"},
{{abd_checksum_blake3_native, abd_checksum_blake3_byteswap},
abd_checksum_blake3_tmpl_init, abd_checksum_blake3_tmpl_free,
ZCHECKSUM_FLAG_METADATA | ZCHECKSUM_FLAG_DEDUP |
ZCHECKSUM_FLAG_SALTED | ZCHECKSUM_FLAG_NOPWRITE, "blake3"},
};
/*
@ -207,6 +211,8 @@ zio_checksum_to_feature(enum zio_checksum cksum)
VERIFY((cksum & ~ZIO_CHECKSUM_MASK) == 0);
switch (cksum) {
case ZIO_CHECKSUM_BLAKE3:
return (SPA_FEATURE_BLAKE3);
case ZIO_CHECKSUM_SHA512:
return (SPA_FEATURE_SHA512);
case ZIO_CHECKSUM_SKEIN:

4
tests/runfiles/common.run
View File

@ -113,8 +113,8 @@ tests = ['tst.destroy_fs', 'tst.destroy_snap', 'tst.get_count_and_limit',
tags = ['functional', 'channel_program', 'synctask_core']
[tests/functional/checksum]
tests = ['run_edonr_test', 'run_sha2_test', 'run_skein_test', 'filetest_001_pos',
'filetest_002_pos']
tests = ['run_edonr_test', 'run_sha2_test', 'run_skein_test', 'run_blake3_test',
'filetest_001_pos', 'filetest_002_pos']
tags = ['functional', 'checksum']
[tests/functional/clean_mirror]

1
tests/zfs-tests/cmd/.gitignore vendored
View File

@ -42,6 +42,7 @@
/ereports
/zfs_diff-socket
/dosmode_readonly_write
/blake3_test
/edonr_test
/skein_test
/sha2_test

6
tests/zfs-tests/cmd/Makefile.am
View File

@ -98,15 +98,19 @@ scripts_zfs_tests_bin_PROGRAMS += %D%/ereports
libzfs.la
scripts_zfs_tests_bin_PROGRAMS += %D%/edonr_test %D%/skein_test %D%/sha2_test
scripts_zfs_tests_bin_PROGRAMS += %D%/edonr_test %D%/skein_test \
%D%/sha2_test %D%/blake3_test
%C%_skein_test_SOURCES = %D%/checksum/skein_test.c
%C%_sha2_test_SOURCES = %D%/checksum/sha2_test.c
%C%_edonr_test_SOURCES = %D%/checksum/edonr_test.c
%C%_blake3_test_SOURCES = %D%/checksum/blake3_test.c
%C%_skein_test_LDADD = \
libicp.la \
libspl.la \
libspl_assert.la
%C%_sha2_test_LDADD = $(%C%_skein_test_LDADD)
%C%_edonr_test_LDADD = $(%C%_skein_test_LDADD)
%C%_blake3_test_LDADD = $(%C%_skein_test_LDADD)
if BUILD_LINUX

575
tests/zfs-tests/cmd/checksum/blake3_test.c Normal file
View File

@ -0,0 +1,575 @@
/*
* CDDL HEADER START
*
* The contents of this file are subject to the terms of the
* Common Development and Distribution License (the "License").
* You may not use this file except in compliance with the License.
*
* You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
* or http://www.opensolaris.org/os/licensing.
* See the License for the specific language governing permissions
* and limitations under the License.
*
* When distributing Covered Code, include this CDDL HEADER in each
* file and include the License file at usr/src/OPENSOLARIS.LICENSE.
* If applicable, add the following below this CDDL HEADER, with the
* fields enclosed by brackets "[]" replaced with your own identifying
* information: Portions Copyright [yyyy] [name of copyright owner]
*
* CDDL HEADER END
*/
/*
* Copyright (c) 2022 Tino Reichardt <milky-zfs@mcmilk.de>
*/
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <sys/time.h>
#include <sys/blake3.h>
/*
* set it to a define for debugging
*/
#undef BLAKE3_DEBUG
/*
* C version of:
* https://github.com/BLAKE3-team/BLAKE3/tree/master/test_vectors
*/
typedef struct {
/* input length for this entry */
const int input_len;
/* hash value */
const char *hash;
/* salted hash value */
const char *shash;
} blake3_test_t;
/* BLAKE3 is variable here */
#define TEST_DIGEST_LEN 262
/*
* key for the keyed hashing
*/
static const char *salt = "whats the Elvish word for friend";
static blake3_test_t TestArray[] = {
{
0,
"af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262e0"
"0f03e7b69af26b7faaf09fcd333050338ddfe085b8cc869ca98b206c08243a26f5"
"487789e8f660afe6c99ef9e0c52b92e7393024a80459cf91f476f9ffdbda7001c2"
"2e159b402631f277ca96f2defdf1078282314e763699a31c5363165421cce14d",
"92b2b75604ed3c761f9d6f62392c8a9227ad0ea3f09573e783f1498a4ed60d26b1"
"8171a2f22a4b94822c701f107153dba24918c4bae4d2945c20ece13387627d3b73"
"cbf97b797d5e59948c7ef788f54372df45e45e4293c7dc18c1d41144a9758be589"
"60856be1eabbe22c2653190de560ca3b2ac4aa692a9210694254c371e851bc8f",
},
{
1,
"2d3adedff11b61f14c886e35afa036736dcd87a74d27b5c1510225d0f592e213c3"
"a6cb8bf623e20cdb535f8d1a5ffb86342d9c0b64aca3bce1d31f60adfa137b358a"
"d4d79f97b47c3d5e79f179df87a3b9776ef8325f8329886ba42f07fb138bb502f4"
"081cbcec3195c5871e6c23e2cc97d3c69a613eba131e5f1351f3f1da786545e5",
"6d7878dfff2f485635d39013278ae14f1454b8c0a3a2d34bc1ab38228a80c95b65"
"68c0490609413006fbd428eb3fd14e7756d90f73a4725fad147f7bf70fd61c4e0c"
"f7074885e92b0e3f125978b4154986d4fb202a3f331a3fb6cf349a3a70e49990f9"
"8fe4289761c8602c4e6ab1138d31d3b62218078b2f3ba9a88e1d08d0dd4cea11",
},
{
2,
"7b7015bb92cf0b318037702a6cdd81dee41224f734684c2c122cd6359cb1ee63d8"
"386b22e2ddc05836b7c1bb693d92af006deb5ffbc4c70fb44d0195d0c6f252faac"
"61659ef86523aa16517f87cb5f1340e723756ab65efb2f91964e14391de2a43226"
"3a6faf1d146937b35a33621c12d00be8223a7f1919cec0acd12097ff3ab00ab1",
"5392ddae0e0a69d5f40160462cbd9bd889375082ff224ac9c758802b7a6fd20a9f"
"fbf7efd13e989a6c246f96d3a96b9d279f2c4e63fb0bdff633957acf50ee1a5f65"
"8be144bab0f6f16500dee4aa5967fc2c586d85a04caddec90fffb7633f46a60786"
"024353b9e5cebe277fcd9514217fee2267dcda8f7b31697b7c54fab6a939bf8f",
},
{
3,
"e1be4d7a8ab5560aa4199eea339849ba8e293d55ca0a81006726d184519e647f5b"
"49b82f805a538c68915c1ae8035c900fd1d4b13902920fd05e1450822f36de9454"
"b7e9996de4900c8e723512883f93f4345f8a58bfe64ee38d3ad71ab027765d25cd"
"d0e448328a8e7a683b9a6af8b0af94fa09010d9186890b096a08471e4230a134",
"39e67b76b5a007d4921969779fe666da67b5213b096084ab674742f0d5ec62b9b9"
"142d0fab08e1b161efdbb28d18afc64d8f72160c958e53a950cdecf91c1a1bbab1"
"a9c0f01def762a77e2e8545d4dec241e98a89b6db2e9a5b070fc110caae2622690"
"bd7b76c02ab60750a3ea75426a6bb8803c370ffe465f07fb57def95df772c39f",
},
{
4,
"f30f5ab28fe047904037f77b6da4fea1e27241c5d132638d8bedce9d40494f328f"
"603ba4564453e06cdcee6cbe728a4519bbe6f0d41e8a14b5b225174a566dbfa61b"
"56afb1e452dc08c804f8c3143c9e2cc4a31bb738bf8c1917b55830c6e657972117"
"01dc0b98daa1faeaa6ee9e56ab606ce03a1a881e8f14e87a4acf4646272cfd12",
"7671dde590c95d5ac9616651ff5aa0a27bee5913a348e053b8aa9108917fe07011"
"6c0acff3f0d1fa97ab38d813fd46506089118147d83393019b068a55d646251ecf"
"81105f798d76a10ae413f3d925787d6216a7eb444e510fd56916f1d753a5544ecf"
"0072134a146b2615b42f50c179f56b8fae0788008e3e27c67482349e249cb86a",
},
{
5,
"b40b44dfd97e7a84a996a91af8b85188c66c126940ba7aad2e7ae6b385402aa2eb"
"cfdac6c5d32c31209e1f81a454751280db64942ce395104e1e4eaca62607de1c2c"
"a748251754ea5bbe8c20150e7f47efd57012c63b3c6a6632dc1c7cd15f3e1c9999"
"04037d60fac2eb9397f2adbe458d7f264e64f1e73aa927b30988e2aed2f03620",
"73ac69eecf286894d8102018a6fc729f4b1f4247d3703f69bdc6a5fe3e0c84616a"
"b199d1f2f3e53bffb17f0a2209fe8b4f7d4c7bae59c2bc7d01f1ff94c67588cc6b"
"38fa6024886f2c078bfe09b5d9e6584cd6c521c3bb52f4de7687b37117a2dbbec0"
"d59e92fa9a8cc3240d4432f91757aabcae03e87431dac003e7d73574bfdd8218",
},
{
6,
"06c4e8ffb6872fad96f9aaca5eee1553eb62aed0ad7198cef42e87f6a616c84461"
"1a30c4e4f37fe2fe23c0883cde5cf7059d88b657c7ed2087e3d210925ede716435"
"d6d5d82597a1e52b9553919e804f5656278bd739880692c94bff2824d8e0b48cac"
"1d24682699e4883389dc4f2faa2eb3b4db6e39debd5061ff3609916f3e07529a",
"82d3199d0013035682cc7f2a399d4c212544376a839aa863a0f4c91220ca7a6dc2"
"ffb3aa05f2631f0fa9ac19b6e97eb7e6669e5ec254799350c8b8d189e880780084"
"2a5383c4d907c932f34490aaf00064de8cdb157357bde37c1504d2960034930887"
"603abc5ccb9f5247f79224baff6120a3c622a46d7b1bcaee02c5025460941256",
},
{
7,
"3f8770f387faad08faa9d8414e9f449ac68e6ff0417f673f602a646a891419fe66"
"036ef6e6d1a8f54baa9fed1fc11c77cfb9cff65bae915045027046ebe0c01bf5a9"
"41f3bb0f73791d3fc0b84370f9f30af0cd5b0fc334dd61f70feb60dad785f070fe"
"f1f343ed933b49a5ca0d16a503f599a365a4296739248b28d1a20b0e2cc8975c",
"af0a7ec382aedc0cfd626e49e7628bc7a353a4cb108855541a5651bf64fbb28a7c"
"5035ba0f48a9c73dabb2be0533d02e8fd5d0d5639a18b2803ba6bf527e1d145d5f"
"d6406c437b79bcaad6c7bdf1cf4bd56a893c3eb9510335a7a798548c6753f74617"
"bede88bef924ba4b334f8852476d90b26c5dc4c3668a2519266a562c6c8034a6",
},
{
8,
"2351207d04fc16ade43ccab08600939c7c1fa70a5c0aaca76063d04c3228eaeb72"
"5d6d46ceed8f785ab9f2f9b06acfe398c6699c6129da084cb531177445a682894f"
"9685eaf836999221d17c9a64a3a057000524cd2823986db378b074290a1a9b93a2"
"2e135ed2c14c7e20c6d045cd00b903400374126676ea78874d79f2dd7883cf5c",
"be2f5495c61cba1bb348a34948c004045e3bd4dae8f0fe82bf44d0da245a060048"
"eb5e68ce6dea1eb0229e144f578b3aa7e9f4f85febd135df8525e6fe40c6f0340d"
"13dd09b255ccd5112a94238f2be3c0b5b7ecde06580426a93e0708555a265305ab"
"f86d874e34b4995b788e37a823491f25127a502fe0704baa6bfdf04e76c13276",
},
{
63,
"e9bc37a594daad83be9470df7f7b3798297c3d834ce80ba85d6e207627b7db7b11"
"97012b1e7d9af4d7cb7bdd1f3bb49a90a9b5dec3ea2bbc6eaebce77f4e470cbf46"
"87093b5352f04e4a4570fba233164e6acc36900e35d185886a827f7ea9bdc1e5c3"
"ce88b095a200e62c10c043b3e9bc6cb9b6ac4dfa51794b02ace9f98779040755",
"bb1eb5d4afa793c1ebdd9fb08def6c36d10096986ae0cfe148cd101170ce37aea0"
"5a63d74a840aecd514f654f080e51ac50fd617d22610d91780fe6b07a26b0847ab"
"b38291058c97474ef6ddd190d30fc318185c09ca1589d2024f0a6f16d45f116783"
"77483fa5c005b2a107cb9943e5da634e7046855eaa888663de55d6471371d55d",
},
{
64,
"4eed7141ea4a5cd4b788606bd23f46e212af9cacebacdc7d1f4c6dc7f2511b98fc"
"9cc56cb831ffe33ea8e7e1d1df09b26efd2767670066aa82d023b1dfe8ab1b2b7f"
"bb5b97592d46ffe3e05a6a9b592e2949c74160e4674301bc3f97e04903f8c6cf95"
"b863174c33228924cdef7ae47559b10b294acd660666c4538833582b43f82d74",
"ba8ced36f327700d213f120b1a207a3b8c04330528586f414d09f2f7d9ccb7e682"
"44c26010afc3f762615bbac552a1ca909e67c83e2fd5478cf46b9e811efccc93f7"
"7a21b17a152ebaca1695733fdb086e23cd0eb48c41c034d52523fc21236e5d8c92"
"55306e48d52ba40b4dac24256460d56573d1312319afcf3ed39d72d0bfc69acb",
},
{
65,
"de1e5fa0be70df6d2be8fffd0e99ceaa8eb6e8c93a63f2d8d1c30ecb6b263dee0e"
"16e0a4749d6811dd1d6d1265c29729b1b75a9ac346cf93f0e1d7296dfcfd4313b3"
"a227faaaaf7757cc95b4e87a49be3b8a270a12020233509b1c3632b3485eef309d"
"0abc4a4a696c9decc6e90454b53b000f456a3f10079072baaf7a981653221f2c",
"c0a4edefa2d2accb9277c371ac12fcdbb52988a86edc54f0716e1591b4326e72d5"
"e795f46a596b02d3d4bfb43abad1e5d19211152722ec1f20fef2cd413e3c22f2fc"
"5da3d73041275be6ede3517b3b9f0fc67ade5956a672b8b75d96cb43294b904149"
"7de92637ed3f2439225e683910cb3ae923374449ca788fb0f9bea92731bc26ad",
},
{
127,
"d81293fda863f008c09e92fc382a81f5a0b4a1251cba1634016a0f86a6bd640de3"
"137d477156d1fde56b0cf36f8ef18b44b2d79897bece12227539ac9ae0a5119da4"
"7644d934d26e74dc316145dcb8bb69ac3f2e05c242dd6ee06484fcb0e956dc4435"
"5b452c5e2bbb5e2b66e99f5dd443d0cbcaaafd4beebaed24ae2f8bb672bcef78",
"c64200ae7dfaf35577ac5a9521c47863fb71514a3bcad18819218b818de85818ee"
"7a317aaccc1458f78d6f65f3427ec97d9c0adb0d6dacd4471374b621b7b5f35cd5"
"4663c64dbe0b9e2d95632f84c611313ea5bd90b71ce97b3cf645776f3adc11e27d"
"135cbadb9875c2bf8d3ae6b02f8a0206aba0c35bfe42574011931c9a255ce6dc",
},
{
128,
"f17e570564b26578c33bb7f44643f539624b05df1a76c81f30acd548c44b45efa6"
"9faba091427f9c5c4caa873aa07828651f19c55bad85c47d1368b11c6fd99e47ec"
"ba5820a0325984d74fe3e4058494ca12e3f1d3293d0010a9722f7dee64f71246f7"
"5e9361f44cc8e214a100650db1313ff76a9f93ec6e84edb7add1cb4a95019b0c",
"b04fe15577457267ff3b6f3c947d93be581e7e3a4b018679125eaf86f6a628ecd8"
"6bbe0001f10bda47e6077b735016fca8119da11348d93ca302bbd125bde0db2b50"
"edbe728a620bb9d3e6f706286aedea973425c0b9eedf8a38873544cf91badf49ad"
"92a635a93f71ddfcee1eae536c25d1b270956be16588ef1cfef2f1d15f650bd5",
},
{
129,
"683aaae9f3c5ba37eaaf072aed0f9e30bac0865137bae68b1fde4ca2aebdcb12f9"
"6ffa7b36dd78ba321be7e842d364a62a42e3746681c8bace18a4a8a79649285c71"
"27bf8febf125be9de39586d251f0d41da20980b70d35e3dac0eee59e468a894fa7"
"e6a07129aaad09855f6ad4801512a116ba2b7841e6cfc99ad77594a8f2d181a7",
"d4a64dae6cdccbac1e5287f54f17c5f985105457c1a2ec1878ebd4b57e20d38f1c"
"9db018541eec241b748f87725665b7b1ace3e0065b29c3bcb232c90e37897fa5aa"
"ee7e1e8a2ecfcd9b51463e42238cfdd7fee1aecb3267fa7f2128079176132a412c"
"d8aaf0791276f6b98ff67359bd8652ef3a203976d5ff1cd41885573487bcd683",
},
{
1023,
"10108970eeda3eb932baac1428c7a2163b0e924c9a9e25b35bba72b28f70bd11a1"
"82d27a591b05592b15607500e1e8dd56bc6c7fc063715b7a1d737df5bad3339c56"
"778957d870eb9717b57ea3d9fb68d1b55127bba6a906a4a24bbd5acb2d123a37b2"
"8f9e9a81bbaae360d58f85e5fc9d75f7c370a0cc09b6522d9c8d822f2f28f485",
"c951ecdf03288d0fcc96ee3413563d8a6d3589547f2c2fb36d9786470f1b9d6e89"
"0316d2e6d8b8c25b0a5b2180f94fb1a158ef508c3cde45e2966bd796a696d3e13e"
"fd86259d756387d9becf5c8bf1ce2192b87025152907b6d8cc33d17826d8b7b9bc"
"97e38c3c85108ef09f013e01c229c20a83d9e8efac5b37470da28575fd755a10",
},
{
1024,
"42214739f095a406f3fc83deb889744ac00df831c10daa55189b5d121c855af71c"
"f8107265ecdaf8505b95d8fcec83a98a6a96ea5109d2c179c47a387ffbb404756f"
"6eeae7883b446b70ebb144527c2075ab8ab204c0086bb22b7c93d465efc57f8d91"
"7f0b385c6df265e77003b85102967486ed57db5c5ca170ba441427ed9afa684e",
"75c46f6f3d9eb4f55ecaaee480db732e6c2105546f1e675003687c31719c7ba4a7"
"8bc838c72852d4f49c864acb7adafe2478e824afe51c8919d06168414c265f298a"
"8094b1ad813a9b8614acabac321f24ce61c5a5346eb519520d38ecc43e89b50002"
"36df0597243e4d2493fd626730e2ba17ac4d8824d09d1a4a8f57b8227778e2de",
},
{
1025,
"d00278ae47eb27b34faecf67b4fe263f82d5412916c1ffd97c8cb7fb814b8444f4"
"c4a22b4b399155358a994e52bf255de60035742ec71bd08ac275a1b51cc6bfe332"
"b0ef84b409108cda080e6269ed4b3e2c3f7d722aa4cdc98d16deb554e5627be8f9"
"55c98e1d5f9565a9194cad0c4285f93700062d9595adb992ae68ff12800ab67a",
"357dc55de0c7e382c900fd6e320acc04146be01db6a8ce7210b7189bd664ea6936"
"2396b77fdc0d2634a552970843722066c3c15902ae5097e00ff53f1e116f1cd535"
"2720113a837ab2452cafbde4d54085d9cf5d21ca613071551b25d52e69d6c81123"
"872b6f19cd3bc1333edf0c52b94de23ba772cf82636cff4542540a7738d5b930",
},
{
2048,
"e776b6028c7cd22a4d0ba182a8bf62205d2ef576467e838ed6f2529b85fba24a9a"
"60bf80001410ec9eea6698cd537939fad4749edd484cb541aced55cd9bf54764d0"
"63f23f6f1e32e12958ba5cfeb1bf618ad094266d4fc3c968c2088f677454c288c6"
"7ba0dba337b9d91c7e1ba586dc9a5bc2d5e90c14f53a8863ac75655461cea8f9",
"879cf1fa2ea0e79126cb1063617a05b6ad9d0b696d0d757cf053439f60a99dd101"
"73b961cd574288194b23ece278c330fbb8585485e74967f31352a8183aa782b2b2"
"2f26cdcadb61eed1a5bc144b8198fbb0c13abbf8e3192c145d0a5c21633b0ef860"
"54f42809df823389ee40811a5910dcbd1018af31c3b43aa55201ed4edaac74fe",
},
{
2049,
"5f4d72f40d7a5f82b15ca2b2e44b1de3c2ef86c426c95c1af0b687952256303096"
"de31d71d74103403822a2e0bc1eb193e7aecc9643a76b7bbc0c9f9c52e8783aae9"
"8764ca468962b5c2ec92f0c74eb5448d519713e09413719431c802f948dd5d9042"
"5a4ecdadece9eb178d80f26efccae630734dff63340285adec2aed3b51073ad3",
"9f29700902f7c86e514ddc4df1e3049f258b2472b6dd5267f61bf13983b78dd5f9"
"a88abfefdfa1e00b418971f2b39c64ca621e8eb37fceac57fd0c8fc8e117d43b81"
"447be22d5d8186f8f5919ba6bcc6846bd7d50726c06d245672c2ad4f61702c6464"
"99ee1173daa061ffe15bf45a631e2946d616a4c345822f1151284712f76b2b0e",
},
{
3072,
"b98cb0ff3623be03326b373de6b9095218513e64f1ee2edd2525c7ad1e5cffd29a"
"3f6b0b978d6608335c09dc94ccf682f9951cdfc501bfe47b9c9189a6fc7b404d12"
"0258506341a6d802857322fbd20d3e5dae05b95c88793fa83db1cb08e7d8008d15"
"99b6209d78336e24839724c191b2a52a80448306e0daa84a3fdb566661a37e11",
"044a0e7b172a312dc02a4c9a818c036ffa2776368d7f528268d2e6b5df19177022"
"f302d0529e4174cc507c463671217975e81dab02b8fdeb0d7ccc7568dd22574c78"
"3a76be215441b32e91b9a904be8ea81f7a0afd14bad8ee7c8efc305ace5d3dd61b"
"996febe8da4f56ca0919359a7533216e2999fc87ff7d8f176fbecb3d6f34278b",
},
{
3073,
"7124b49501012f81cc7f11ca069ec9226cecb8a2c850cfe644e327d22d3e1cd39a"
"27ae3b79d68d89da9bf25bc27139ae65a324918a5f9b7828181e52cf373c84f35b"
"639b7fccbb985b6f2fa56aea0c18f531203497b8bbd3a07ceb5926f1cab74d14bd"
"66486d9a91eba99059a98bd1cd25876b2af5a76c3e9eed554ed72ea952b603bf",
"68dede9bef00ba89e43f31a6825f4cf433389fedae75c04ee9f0cf16a427c95a96"
"d6da3fe985054d3478865be9a092250839a697bbda74e279e8a9e69f0025e4cfdd"
"d6cfb434b1cd9543aaf97c635d1b451a4386041e4bb100f5e45407cbbc24fa53ea"
"2de3536ccb329e4eb9466ec37093a42cf62b82903c696a93a50b702c80f3c3c5",
},
{
4096,
"015094013f57a5277b59d8475c0501042c0b642e531b0a1c8f58d2163229e96902"
"89e9409ddb1b99768eafe1623da896faf7e1114bebeadc1be30829b6f8af707d85"
"c298f4f0ff4d9438aef948335612ae921e76d411c3a9111df62d27eaf871959ae0"
"062b5492a0feb98ef3ed4af277f5395172dbe5c311918ea0074ce0036454f620",
"befc660aea2f1718884cd8deb9902811d332f4fc4a38cf7c7300d597a081bfc0bb"
"b64a36edb564e01e4b4aaf3b060092a6b838bea44afebd2deb8298fa562b7b597c"
"757b9df4c911c3ca462e2ac89e9a787357aaf74c3b56d5c07bc93ce899568a3eb1"
"7d9250c20f6c5f6c1e792ec9a2dcb715398d5a6ec6d5c54f586a00403a1af1de",
},
{
4097,
"9b4052b38f1c5fc8b1f9ff7ac7b27cd242487b3d890d15c96a1c25b8aa0fb99505"
"f91b0b5600a11251652eacfa9497b31cd3c409ce2e45cfe6c0a016967316c426bd"
"26f619eab5d70af9a418b845c608840390f361630bd497b1ab44019316357c61db"
"e091ce72fc16dc340ac3d6e009e050b3adac4b5b2c92e722cffdc46501531956",
"00df940cd36bb9fa7cbbc3556744e0dbc8191401afe70520ba292ee3ca80abbc60"
"6db4976cfdd266ae0abf667d9481831ff12e0caa268e7d3e57260c0824115a54ce"
"595ccc897786d9dcbf495599cfd90157186a46ec800a6763f1c59e36197e9939e9"
"00809f7077c102f888caaf864b253bc41eea812656d46742e4ea42769f89b83f",
},
{
5120,
"9cadc15fed8b5d854562b26a9536d9707cadeda9b143978f319ab34230535833ac"
"c61c8fdc114a2010ce8038c853e121e1544985133fccdd0a2d507e8e615e611e9a"
"0ba4f47915f49e53d721816a9198e8b30f12d20ec3689989175f1bf7a300eee0d9"
"321fad8da232ece6efb8e9fd81b42ad161f6b9550a069e66b11b40487a5f5059",
"2c493e48e9b9bf31e0553a22b23503c0a3388f035cece68eb438d22fa1943e209b"
"4dc9209cd80ce7c1f7c9a744658e7e288465717ae6e56d5463d4f80cdb2ef56495"
"f6a4f5487f69749af0c34c2cdfa857f3056bf8d807336a14d7b89bf62bef2fb54f"
"9af6a546f818dc1e98b9e07f8a5834da50fa28fb5874af91bf06020d1bf0120e",
},
{
5121,
"628bd2cb2004694adaab7bbd778a25df25c47b9d4155a55f8fbd79f2fe154cff96"
"adaab0613a6146cdaabe498c3a94e529d3fc1da2bd08edf54ed64d40dcd6777647"
"eac51d8277d70219a9694334a68bc8f0f23e20b0ff70ada6f844542dfa32cd4204"
"ca1846ef76d811cdb296f65e260227f477aa7aa008bac878f72257484f2b6c95",
"6ccf1c34753e7a044db80798ecd0782a8f76f33563accaddbfbb2e0ea4b2d0240d"
"07e63f13667a8d1490e5e04f13eb617aea16a8c8a5aaed1ef6fbde1b0515e3c810"
"50b361af6ead126032998290b563e3caddeaebfab592e155f2e161fb7cba939092"
"133f23f9e65245e58ec23457b78a2e8a125588aad6e07d7f11a85b88d375b72d",
},
{
6144,
"3e2e5b74e048f3add6d21faab3f83aa44d3b2278afb83b80b3c35164ebeca2054d"
"742022da6fdda444ebc384b04a54c3ac5839b49da7d39f6d8a9db03deab32aade1"
"56c1c0311e9b3435cde0ddba0dce7b26a376cad121294b689193508dd63151603c"
"6ddb866ad16c2ee41585d1633a2cea093bea714f4c5d6b903522045b20395c83",
"3d6b6d21281d0ade5b2b016ae4034c5dec10ca7e475f90f76eac7138e9bc8f1dc3"
"5754060091dc5caf3efabe0603c60f45e415bb3407db67e6beb3d11cf8e4f79075"
"61f05dace0c15807f4b5f389c841eb114d81a82c02a00b57206b1d11fa6e803486"
"b048a5ce87105a686dee041207e095323dfe172df73deb8c9532066d88f9da7e",
},
{
6145,
"f1323a8631446cc50536a9f705ee5cb619424d46887f3c376c695b70e0f0507f18"
"a2cfdd73c6e39dd75ce7c1c6e3ef238fd54465f053b25d21044ccb2093beb01501"
"5532b108313b5829c3621ce324b8e14229091b7c93f32db2e4e63126a377d2a63a"
"3597997d4f1cba59309cb4af240ba70cebff9a23d5e3ff0cdae2cfd54e070022",
"9ac301e9e39e45e3250a7e3b3df701aa0fb6889fbd80eeecf28dbc6300fbc539f3"
"c184ca2f59780e27a576c1d1fb9772e99fd17881d02ac7dfd39675aca918453283"
"ed8c3169085ef4a466b91c1649cc341dfdee60e32231fc34c9c4e0b9a2ba87ca8f"
"372589c744c15fd6f985eec15e98136f25beeb4b13c4e43dc84abcc79cd4646c",
},
{
7168,
"61da957ec2499a95d6b8023e2b0e604ec7f6b50e80a9678b89d2628e99ada77a57"
"07c321c83361793b9af62a40f43b523df1c8633cecb4cd14d00bdc79c78fca5165"
"b863893f6d38b02ff7236c5a9a8ad2dba87d24c547cab046c29fc5bc1ed142e1de"
"4763613bb162a5a538e6ef05ed05199d751f9eb58d332791b8d73fb74e4fce95",
"b42835e40e9d4a7f42ad8cc04f85a963a76e18198377ed84adddeaecacc6f3fca2"
"f01d5277d69bb681c70fa8d36094f73ec06e452c80d2ff2257ed82e7ba34840098"
"9a65ee8daa7094ae0933e3d2210ac6395c4af24f91c2b590ef87d7788d7066ea3e"
"aebca4c08a4f14b9a27644f99084c3543711b64a070b94f2c9d1d8a90d035d52",
},
{
7169,
"a003fc7a51754a9b3c7fae0367ab3d782dccf28855a03d435f8cfe74605e781798"
"a8b20534be1ca9eb2ae2df3fae2ea60e48c6fb0b850b1385b5de0fe460dbe9d9f9"
"b0d8db4435da75c601156df9d047f4ede008732eb17adc05d96180f8a735485228"
"40779e6062d643b79478a6e8dbce68927f36ebf676ffa7d72d5f68f050b119c8",
"ed9b1a922c046fdb3d423ae34e143b05ca1bf28b710432857bf738bcedbfa5113c"
"9e28d72fcbfc020814ce3f5d4fc867f01c8f5b6caf305b3ea8a8ba2da3ab69fabc"
"b438f19ff11f5378ad4484d75c478de425fb8e6ee809b54eec9bdb184315dc8566"
"17c09f5340451bf42fd3270a7b0b6566169f242e533777604c118a6358250f54",
},
{
8192,
"aae792484c8efe4f19e2ca7d371d8c467ffb10748d8a5a1ae579948f718a2a635f"
"e51a27db045a567c1ad51be5aa34c01c6651c4d9b5b5ac5d0fd58cf18dd61a4777"
"8566b797a8c67df7b1d60b97b19288d2d877bb2df417ace009dcb0241ca1257d62"
"712b6a4043b4ff33f690d849da91ea3bf711ed583cb7b7a7da2839ba71309bbf",
"dc9637c8845a770b4cbf76b8daec0eebf7dc2eac11498517f08d44c8fc00d58a48"
"34464159dcbc12a0ba0c6d6eb41bac0ed6585cabfe0aca36a375e6c5480c22afdc"
"40785c170f5a6b8a1107dbee282318d00d915ac9ed1143ad40765ec120042ee121"
"cd2baa36250c618adaf9e27260fda2f94dea8fb6f08c04f8f10c78292aa46102",
},
{
8193,
"bab6c09cb8ce8cf459261398d2e7aef35700bf488116ceb94a36d0f5f1b7bc3bb2"
"282aa69be089359ea1154b9a9286c4a56af4de975a9aa4a5c497654914d279bea6"
"0bb6d2cf7225a2fa0ff5ef56bbe4b149f3ed15860f78b4e2ad04e158e375c1e0c0"
"b551cd7dfc82f1b155c11b6b3ed51ec9edb30d133653bb5709d1dbd55f4e1ff6",
"954a2a75420c8d6547e3ba5b98d963e6fa6491addc8c023189cc519821b4a1f5f0"
"3228648fd983aef045c2fa8290934b0866b615f585149587dda229903996532883"
"5a2b18f1d63b7e300fc76ff260b571839fe44876a4eae66cbac8c67694411ed7e0"
"9df51068a22c6e67d6d3dd2cca8ff12e3275384006c80f4db68023f24eebba57",
},
{
16384,
"f875d6646de28985646f34ee13be9a576fd515f76b5b0a26bb324735041ddde49d"
"764c270176e53e97bdffa58d549073f2c660be0e81293767ed4e4929f9ad34bbb3"
"9a529334c57c4a381ffd2a6d4bfdbf1482651b172aa883cc13408fa67758a3e475"
"03f93f87720a3177325f7823251b85275f64636a8f1d599c2e49722f42e93893",
"9e9fc4eb7cf081ea7c47d1807790ed211bfec56aa25bb7037784c13c4b707b0df9"
"e601b101e4cf63a404dfe50f2e1865bb12edc8fca166579ce0c70dba5a5c0fc960"
"ad6f3772183416a00bd29d4c6e651ea7620bb100c9449858bf14e1ddc9ecd35725"
"581ca5b9160de04060045993d972571c3e8f71e9d0496bfa744656861b169d65",
},
{
31744,
"62b6960e1a44bcc1eb1a611a8d6235b6b4b78f32e7abc4fb4c6cdcce94895c4786"
"0cc51f2b0c28a7b77304bd55fe73af663c02d3f52ea053ba43431ca5bab7bfea2f"
"5e9d7121770d88f70ae9649ea713087d1914f7f312147e247f87eb2d4ffef0ac97"
"8bf7b6579d57d533355aa20b8b77b13fd09748728a5cc327a8ec470f4013226f",
"efa53b389ab67c593dba624d898d0f7353ab99e4ac9d42302ee64cbf9939a4193a"
"7258db2d9cd32a7a3ecfce46144114b15c2fcb68a618a976bd74515d47be08b628"
"be420b5e830fade7c080e351a076fbc38641ad80c736c8a18fe3c66ce12f95c61c"
"2462a9770d60d0f77115bbcd3782b593016a4e728d4c06cee4505cb0c08a42ec",
},
{
102400,
"bc3e3d41a1146b069abffad3c0d44860cf664390afce4d9661f7902e7943e085e0"
"1c59dab908c04c3342b816941a26d69c2605ebee5ec5291cc55e15b76146e6745f"
"0601156c3596cb75065a9c57f35585a52e1ac70f69131c23d611ce11ee4ab1ec2c"
"009012d236648e77be9295dd0426f29b764d65de58eb7d01dd42248204f45f8e",
"1c35d1a5811083fd7119f5d5d1ba027b4d01c0c6c49fb6ff2cf75393ea5db4a7f9"
"dbdd3e1d81dcbca3ba241bb18760f207710b751846faaeb9dff8262710999a59b2"
"aa1aca298a032d94eacfadf1aa192418eb54808db23b56e34213266aa08499a16b"
"354f018fc4967d05f8b9d2ad87a7278337be9693fc638a3bfdbe314574ee6fc4",
},
{
0, 0, 0
}
};
#ifdef BLAKE3_DEBUG
#define dprintf printf
#else
#define dprintf(...)
#endif
static char fmt_tohex(char c);
static size_t fmt_hexdump(char *dest, const char *src, size_t len);
static char fmt_tohex(char c) {
return ((char)(c >= 10 ? c-10+'a' : c+'0'));
}
static size_t fmt_hexdump(char *dest, const char *src, size_t len) {
register const unsigned char *s = (const unsigned char *) src;
size_t written = 0, i;
if (!dest)
return ((len > ((size_t)-1)/2) ? (size_t)-1 : len*2);
for (i = 0; i < len; ++i) {
dest[written] = fmt_tohex(s[i]>>4);
dest[written+1] = fmt_tohex(s[i]&15);
written += 2;
}
return (written);
}
int
main(int argc, char *argv[])
{
boolean_t failed = B_FALSE;
uint8_t buffer[102400];
uint64_t cpu_mhz = 0;
int id, i, j;
if (argc == 2)
cpu_mhz = atoi(argv[1]);
/* fill test message */
for (i = 0, j = 0; i < sizeof (buffer); i++, j++) {
if (j == 251)
j = 0;
buffer[i] = (uint8_t)j;
}
(void) printf("Running algorithm correctness tests:\n");
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
const char *name = blake3_get_impl_name();
dprintf("Result for BLAKE3-%s:\n", name);
for (i = 0; TestArray[i].hash; i++) {
blake3_test_t *cur = &TestArray[i];
BLAKE3_CTX ctx;
uint8_t digest[TEST_DIGEST_LEN];
char result[TEST_DIGEST_LEN];
/* default hashing */
Blake3_Init(&ctx);
Blake3_Update(&ctx, buffer, cur->input_len);
Blake3_FinalSeek(&ctx, 0, digest, TEST_DIGEST_LEN);
fmt_hexdump(result, (char *)digest, 131);
if (memcmp(result, cur->hash, 131) != 0)
failed = B_TRUE;
dprintf("HASH-res: %s\n", result);
dprintf("HASH-ref: %s\n", cur->hash);
/* salted hashing */
Blake3_InitKeyed(&ctx, (const uint8_t *)salt);
Blake3_Update(&ctx, buffer, cur->input_len);
Blake3_FinalSeek(&ctx, 0, digest, TEST_DIGEST_LEN);
fmt_hexdump(result, (char *)digest, 131);
if (memcmp(result, cur->shash, 131) != 0)
failed = B_TRUE;
dprintf("SHASH-res: %s\n", result);
dprintf("SHASH-ref: %s\n", cur->shash);
printf("BLAKE3-%s Message (inlen=%d)\tResult: %s\n",
name, cur->input_len, failed?"FAILED!":"OK");
}
}
if (failed)
return (1);
#define BLAKE3_PERF_TEST(impl, diglen) \
do { \
BLAKE3_CTX ctx; \
uint8_t digest[diglen / 8]; \
uint8_t block[131072]; \
uint64_t delta; \
double cpb = 0; \
int i; \
struct timeval start, end; \
memset(block, 0, sizeof (block)); \
(void) gettimeofday(&start, NULL); \
Blake3_Init(&ctx); \
for (i = 0; i < 8192; i++) \
Blake3_Update(&ctx, block, sizeof (block)); \
Blake3_Final(&ctx, digest); \
(void) gettimeofday(&end, NULL); \
delta = (end.tv_sec * 1000000llu + end.tv_usec) - \
(start.tv_sec * 1000000llu + start.tv_usec); \
if (cpu_mhz != 0) { \
cpb = (cpu_mhz * 1e6 * ((double)delta / \
1000000)) / (8192 * 128 * 1024); \
} \
(void) printf("BLAKE3-%s %llu us (%.02f CPB)\n", impl, \
(u_longlong_t)delta, cpb); \
} while (0)
printf("Running performance tests (hashing 1024 MiB of data):\n");
for (id = 0; id < blake3_get_impl_count(); id++) {
blake3_set_impl_id(id);
const char *name = blake3_get_impl_name();
BLAKE3_PERF_TEST(name, 256);
}
return (0);
}

3
tests/zfs-tests/cmd/checksum/edonr_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining Edon-R code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <sys/edonr.h>
#include <stdlib.h>

3
tests/zfs-tests/cmd/checksum/sha2_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining SHA2 code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <stdarg.h>
#include <stdlib.h>

3
tests/zfs-tests/cmd/checksum/skein_test.c
View File

@ -28,9 +28,6 @@
* gettimeofday due to -D_KERNEL (we can do this since we're actually
* running in userspace, but we need -D_KERNEL for the remaining Skein code).
*/
#ifdef _KERNEL
#undef _KERNEL
#endif
#include <sys/skein.h>
#include <stdlib.h>

1
tests/zfs-tests/include/commands.cfg
View File

@ -212,6 +212,7 @@ export ZFSTEST_FILES='badsend
zed_fd_spill-zedlet
suid_write_to_file
cp_files
blake3_test
edonr_test
skein_test
sha2_test

2
tests/zfs-tests/include/properties.shlib
View File

@ -17,7 +17,7 @@
typeset -a compress_prop_vals=('off' 'lzjb' 'lz4' 'gzip' 'zle' 'zstd')
typeset -a checksum_prop_vals=('on' 'off' 'fletcher2' 'fletcher4' 'sha256'
'noparity' 'sha512' 'skein')
'noparity' 'sha512' 'skein' 'blake3')
if ! is_freebsd; then
checksum_prop_vals+=('edonr')
fi

1
tests/zfs-tests/tests/Makefile.am
View File

@ -545,6 +545,7 @@ nobase_dist_datadir_zfs_tests_tests_SCRIPTS += \
functional/checksum/cleanup.ksh \
functional/checksum/filetest_001_pos.ksh \
functional/checksum/filetest_002_pos.ksh \
functional/checksum/run_blake3_test.ksh \
functional/checksum/run_edonr_test.ksh \
functional/checksum/run_sha2_test.ksh \
functional/checksum/run_skein_test.ksh \

2
tests/zfs-tests/tests/functional/checksum/default.cfg
View File

@ -30,4 +30,4 @@
. $STF_SUITE/include/libtest.shlib
set -A CHECKSUM_TYPES "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr"
set -A CHECKSUM_TYPES "fletcher2" "fletcher4" "blake3" "sha256" "sha512" "skein" "edonr"

30
tests/zfs-tests/tests/functional/checksum/run_blake3_test.ksh Executable file
View File

@ -0,0 +1,30 @@
#!/bin/ksh -p
#
# This file and its contents are supplied under the terms of the
# Common Development and Distribution License ("CDDL"), version 1.0.
# You may only use this file in accordance with the terms of version
# 1.0 of the CDDL.
#
# A full copy of the text of the CDDL should have accompanied this
# source. A copy of the CDDL is also available via the Internet at
# http://www.illumos.org/license/CDDL.
#
#
# Copyright (c) 2015, 2016 by Delphix. All rights reserved.
#
. $STF_SUITE/include/libtest.shlib
#
# Description:
# Run the tests for the BLAKE3 hash algorithm.
#
log_assert "Run the tests for the BLAKE3 hash algorithm."
freq=$(get_cpu_freq)
log_must blake3_test $freq
log_pass "BLAKE3 tests passed."

2
tests/zfs-tests/tests/functional/cli_root/zfs_set/checksum_001_pos.ksh
View File

@ -46,7 +46,7 @@
verify_runnable "both"
set -A dataset "$TESTPOOL" "$TESTPOOL/$TESTFS" "$TESTPOOL/$TESTVOL"
set -A values "on" "off" "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr" "noparity"
set -A values "on" "off" "fletcher2" "fletcher4" "sha256" "sha512" "skein" "edonr" "blake3" "noparity"
log_assert "Setting a valid checksum on a file system, volume," \
"it should be successful."

1
tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg
View File

@ -99,5 +99,6 @@ if is_linux || is_freebsd; then
"feature@zstd_compress"
"feature@zilsaxattr"
"feature@head_errlog"
"feature@blake3"
)
fi