log xattr=sa create/remove/update to ZIL

As such, there are no specific synchronous semantics defined for
the xattrs. But for xattr=on, it does log to ZIL and zil_commit() is
done, if sync=always is set on dataset. This provides sync semantics
for xattr=on with sync=always set on dataset.

For the xattr=sa implementation, it doesn't log to ZIL, so, even with
sync=always, xattrs are not guaranteed to be synced before xattr call
returns to caller. So, xattr can be lost if system crash happens, before
txg carrying xattr transaction is synced.

This change adds xattr=sa logging to ZIL on xattr create/remove/update
and xattrs are synced to ZIL (zil_commit() done) for sync=always.
This makes xattr=sa behavior similar to xattr=on.

Implementation notes:
The actual logging is fairly straight-forward and does not warrant
additional explanation.
However, it has been 14 years since we last added new TX types
to the ZIL [1], hence this is the first time we do it after the
introduction of zpool features. Therefore, here is an overview of the
feature activation and deactivation workflow:

1. The feature must be enabled. Otherwise, we don't log the new
    record type. This ensures compatibility with older software.
2. The feature is activated per-dataset, since the ZIL is per-dataset.
3. If the feature is enabled and dataset is not for zvol, any append to
    the ZIL chain will activate the feature for the dataset. Likewise
    for starting a new ZIL chain.
4. A dataset that doesn't have a ZIL chain has the feature deactivated.

We ensure (3) by activating on the first zil_commit() after the feature
was enabled. Since activating the features requires waiting for txg
sync, the first zil_commit() after enabling the feature will be slower
than usual. The downside is that this is really a conservative
approximation: even if we never append a 'TX_SETSAXATTR' to the ZIL
chain, we pay the penalty for feature activation. The upside is that the
user is in control of when we pay the penalty, i.e., upon enabling the
feature.

We ensure (4) by hooking into zil_sync(), where ZIL destroy actually
happens.

One more piece on feature activation, since it's spread across
multiple functions:

zil_commit()
  zil_process_commit_list()
    if lwb == NULL // first zil_commit since zil_open
      zil_create()
        if no log block pointer in ZIL header:
          if feature enabled and not active:
	    // CASE 1
            enable, COALESCE txg wait with dmu_tx that allocated the
	    log block
         else // log block was allocated earlier than this zil_open
          if feature enabled and not active:
	    // CASE 2
            enable, EXPLICIT txg wait
    else // already have an in-DRAM LWB
      if feature enabled and not active:
        // this happens when we enable the feature after zil_create
	// CASE 3
        enable, EXPLICIT txg wait

[1] https://github.com/illumos/illumos-gate/commit/da6c28aaf62fa55f0fdb8004aa40f88f23bf53f0

Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Christian Schwarz <christian.schwarz@nutanix.com>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Ryan Moeller <freqlabs@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jitendra Patidar <jitendra.patidar@nutanix.com>
Closes #8768 
Closes #9078

tests/zfs-tests/include/tunables.cfg

@@ -91,6 +91,7 @@ XATTR_COMPAT			xattr_compat			zfs_xattr_compat
 ZEVENT_LEN_MAX			zevent.len_max			zfs_zevent_len_max
 ZEVENT_RETAIN_MAX		zevent.retain_max		zfs_zevent_retain_max
 ZIO_SLOW_IO_MS			zio.slow_io_ms			zio_slow_io_ms
+ZIL_SAXATTR			zil_saxattr			zfs_zil_saxattr
 %%%%
 while read name FreeBSD Linux; do
 	eval "export ${name}=\$${UNAME}"

tests/zfs-tests/tests/functional/cli_root/zpool_get/zpool_get.cfg

@@ -97,5 +97,6 @@ if is_linux || is_freebsd; then
 	    "feature@bookmark_v2"
 	    "feature@livelist"
 	    "feature@zstd_compress"
+	    "feature@zilsaxattr"
 	)
 fi

tests/zfs-tests/tests/functional/slog/Makefile.am

@@ -19,7 +19,8 @@ dist_pkgdata_SCRIPTS = \
 	slog_015_neg.ksh \
 	slog_replay_fs_001.ksh \
 	slog_replay_fs_002.ksh \
-	slog_replay_volume.ksh
+	slog_replay_volume.ksh \
+	slog_016_pos.ksh
 
 dist_pkgdata_DATA = \
 	slog.cfg \

tests/zfs-tests/tests/functional/slog/slog_016_pos.ksh

@@ -0,0 +1,157 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+
+#
+# Copyright (c) 2021 by Nutanix. All rights reserved.
+#
+
+. $STF_SUITE/tests/functional/slog/slog.kshlib
+
+#
+# DESCRIPTION:
+#	Verify saxattr logging in to ZIL works
+#
+# STRATEGY:
+#	1. Create an empty file system (TESTFS)
+#	2. Freeze TESTFS
+#	3. Create Xattrs.
+#	4. Unmount filesystem
+#	   <at this stage TESTFS is empty again and unfrozen, and the
+#	   intent log contains a complete set of deltas to replay it>
+#	5. Remount TESTFS <which replays the intent log>
+#	6. Check xattrs.
+#
+
+verify_runnable "global"
+
+function cleanup_testenv
+{
+	cleanup
+	log_must set_tunable32 ZIL_SAXATTR $orig_zil_saxattr
+}
+
+log_assert "Verify saxattr logging in to ZIL works"
+
+orig_zil_saxattr=$(get_tunable ZIL_SAXATTR)
+
+log_onexit cleanup_testenv
+log_must setup
+
+NFILES=10
+function validate_zil_saxattr
+{
+	saxattrzil=$1
+	if [ "$2" == "disabled" ]; then
+		zilsaxattr_feature_disabled=1
+		zpoolcreateflags="-ofeature@zilsaxattr=disabled"
+	else
+		zilsaxattr_feature_disabled=0
+		zpoolcreateflags=""
+	fi
+
+	log_must set_tunable32 ZIL_SAXATTR $saxattrzil
+
+	#
+	# 1. Create an empty file system (TESTFS)
+	#
+	log_must zpool create $zpoolcreateflags $TESTPOOL $VDEV log mirror $LDEV
+	log_must zfs set compression=on $TESTPOOL
+	log_must zfs create -o xattr=sa $TESTPOOL/$TESTFS
+	log_must mkdir -p $TESTDIR
+
+	#
+	# This dd command works around an issue where ZIL records aren't created
+	# after freezing the pool unless a ZIL header already exists. Create a
+	# file synchronously to force ZFS to write one out.
+	#
+	log_must dd if=/dev/zero of=/$TESTPOOL/$TESTFS/sync \
+	    conv=fdatasync,fsync bs=1 count=1
+
+	#
+	# 2. Freeze TESTFS
+	#
+	log_must zpool freeze $TESTPOOL
+
+	rm /$TESTPOOL/$TESTFS/sync
+	#
+	# 3. Create xattrs
+	#
+	for i in $(seq $NFILES); do
+		log_must mkdir /$TESTPOOL/$TESTFS/xattr.d.$i
+		log_must set_xattr test test /$TESTPOOL/$TESTFS/xattr.d.$i
+
+		log_must touch /$TESTPOOL/$TESTFS/xattr.f.$i
+		log_must set_xattr test test /$TESTPOOL/$TESTFS/xattr.f.$i
+	done
+
+	#
+	# 4. Unmount filesystem and export the pool
+	#
+	# At this stage TESTFS is empty again and unfrozen, and the
+	# intent log contains a complete set of deltas to replay it.
+	#
+	log_must zfs unmount /$TESTPOOL/$TESTFS
+
+	log_note "Verify transactions to replay:"
+	log_must zdb -iv $TESTPOOL/$TESTFS
+
+	log_must zpool export $TESTPOOL
+
+	#
+	# 5. Remount TESTFS <which replays the intent log>
+	#
+	# Import the pool to unfreeze it and claim log blocks.  It has to be
+	# `zpool import -f` because we can't write a frozen pool's labels!
+	#
+	log_must zpool import -f -d $VDIR $TESTPOOL
+
+	#
+	# 6. Verify Xattr
+	# If zilsaxattr_feature_disabled=1 or saxattrzil=0, then xattr=sa
+	# logging in ZIL is not enabled, So, xattrs would be lost.
+	# If zilsaxattr_feature_disabled=0 and saxattrzil=1, then xattr=sa
+	# logging in ZIL is enabled, So, xattrs shouldn't be lost.
+	#
+	for i in $(seq $NFILES); do
+		if [ $zilsaxattr_feature_disabled -eq 1 -o \
+		    $saxattrzil -eq 0 ]; then
+			log_mustnot get_xattr test /$TESTPOOL/$TESTFS/xattr.d.$i
+			log_mustnot get_xattr test /$TESTPOOL/$TESTFS/xattr.f.$i
+		else
+			log_must get_xattr test /$TESTPOOL/$TESTFS/xattr.d.$i
+			log_must get_xattr test /$TESTPOOL/$TESTFS/xattr.f.$i
+		fi
+	done
+
+	cleanup
+	log_must setup
+}
+
+
+#Validate zilsaxattr feature enabled.
+validate_zil_saxattr 0
+validate_zil_saxattr 1
+#Validate zilsaxattr feature disabled.
+validate_zil_saxattr 0 disabled
+validate_zil_saxattr 1 disabled
+
+log_pass "Verify saxattr logging in to ZIL works"