Cross-platform xattr user namespace compatibility

ZFS on Linux originally implemented xattr namespaces in a way that is
incompatible with other operating systems.  On illumos, xattrs do not
have namespaces.  Every xattr name is visible.  FreeBSD has two
universally defined namespaces: EXTATTR_NAMESPACE_USER and
EXTATTR_NAMESPACE_SYSTEM.  The system namespace is used for protected
FreeBSD-specific attributes such as MAC labels and pnfs state.  These
attributes have the namespace string "freebsd:system:" prefixed to the
name in the encoding scheme used by ZFS.  The user namespace is used
for general purpose user attributes and obeys normal access control
mechanisms.  These attributes have no namespace string prefixed, so
xattrs written on illumos are accessible in the user namespace on
FreeBSD, and xattrs written to the user namespace on FreeBSD are
accessible by the same name on illumos.

Linux has several xattr namespaces.  On Linux, ZFS encodes the
namespace in the xattr name for every namespace, including the user
namespace.  As a consequence, an xattr in the user namespace with the
name "foo" is stored by ZFS with the name "user.foo" and therefore
appears on FreeBSD and illumos to have the name "user.foo" rather than
"foo".  Conversely, none of the xattrs written on FreeBSD or illumos
are accessible on Linux unless the name happens to be prefixed with one
of the Linux xattr namespaces, in which case the namespace is stripped
from the name.  This makes xattrs entirely incompatible between Linux
and other platforms.

We want to make the encoding of user namespace xattrs compatible across
platforms.  A critical requirement of this compatibility is for xattrs
from existing pools from FreeBSD and illumos to be accessible by the
same names in the user namespace on Linux.  It is also necessary that
existing pools with xattrs written by Linux retain access to those
xattrs by the same names on Linux.  Making user namespace xattrs from
Linux accessible by the correct names on other platforms is important.
The handling of other namespaces is not required to be consistent.

Add a fallback mechanism for listing and getting xattrs to treat xattrs
as being in the user namespace if they do not match a known prefix.

Do not allow setting or getting xattrs with a name that is prefixed
with one of the namespace names used by ZFS on supported platforms.

Allow choosing between legacy illumos and FreeBSD compatibility and
legacy Linux compatibility with a new tunable.  This facilitates
replication and migration of pools between hosts with different
compatibility needs.

The tunable controls whether or not to prefix the namespace to the
name.  If the xattr is already present with the alternate prefix,
remove it so only the new version persists.  By default the platform's
existing convention is used.

Reviewed-by: Christian Schwarz <christian.schwarz@nutanix.com>
Reviewed-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@iXsystems.com>
Closes #11919

tests/zfs-tests/include/tunables.cfg

@@ -87,6 +87,7 @@ VDEV_VALIDATE_SKIP		vdev.validate_skip		vdev_validate_skip
 VOL_INHIBIT_DEV			UNSUPPORTED			zvol_inhibit_dev
 VOL_MODE			vol.mode			zvol_volmode
 VOL_RECURSIVE			vol.recursive			UNSUPPORTED
+XATTR_COMPAT			xattr_compat			zfs_xattr_compat
 ZEVENT_LEN_MAX			zevent.len_max			zfs_zevent_len_max
 ZEVENT_RETAIN_MAX		zevent.retain_max		zfs_zevent_retain_max
 ZIO_SLOW_IO_MS			zio.slow_io_ms			zio_slow_io_ms

tests/zfs-tests/tests/functional/xattr/Makefile.am

@@ -14,7 +14,8 @@ dist_pkgdata_SCRIPTS = \
 	xattr_010_neg.ksh \
 	xattr_011_pos.ksh \
 	xattr_012_pos.ksh \
-	xattr_013_pos.ksh
+	xattr_013_pos.ksh \
+	xattr_compat.ksh
 
 dist_pkgdata_DATA = \
 	xattr_common.kshlib \

tests/zfs-tests/tests/functional/xattr/xattr_compat.ksh

@@ -0,0 +1,92 @@
+#!/bin/ksh -p
+#
+# CDDL HEADER START
+#
+# The contents of this file are subject to the terms of the
+# Common Development and Distribution License (the "License").
+# You may not use this file except in compliance with the License.
+#
+# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
+# or http://www.opensolaris.org/os/licensing.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# When distributing Covered Code, include this CDDL HEADER in each
+# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
+# If applicable, add the following below this CDDL HEADER, with the
+# fields enclosed by brackets "[]" replaced with your own identifying
+# information: Portions Copyright [yyyy] [name of copyright owner]
+#
+# CDDL HEADER END
+#
+# Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
+# Use is subject to license terms.
+#
+
+#
+# Copyright 2022 iXsystems, Inc.
+#
+
+. $STF_SUITE/include/libtest.shlib
+
+#
+# DESCRIPTION:
+# The zfs_xattr_compat tunable and fallback works as expected.
+#
+# STRATEGY:
+#	For both of xattr=sa and xattr=dir:
+#	1. Create a filesystem with the native zfs_xattr_compat
+#	2. Create a file on the filesystem and add some xattrs to it
+#	3. Change the zfs_xattr_compat to the alternative setting
+#	4. Verify that the xattrs can still be accessed and modified
+#	5. Change zfs_xattr_compat back to the native setting
+#	6. Verify that the xattrs can still be accessed and modified
+#
+
+function cleanup {
+	rm -f $TESTFILE $TMPFILE
+	zfs set xattr=sa $TESTPOOL/$TESTFS
+	set_tunable32 XATTR_COMPAT $NATIVE_XATTR_COMPAT
+}
+
+log_assert "The zfs_xattr_compat tunable and fallback works as expected"
+log_onexit cleanup
+
+TESTFILE=$TESTDIR/testfile.$$
+TMPFILE=$TEST_BASE_DIR/tmpfile.$$
+NATIVE_XATTR_COMPAT=$(get_tunable XATTR_COMPAT)
+ALTERNATIVE_XATTR_COMPAT=$((1 - NATIVE_XATTR_COMPAT))
+
+for x in sa dir; do
+	log_must zfs set xattr=$x $TESTPOOL/$TESTFS
+	log_must touch $TESTFILE
+	log_must set_xattr testattr1 value1 $TESTFILE
+	log_must set_xattr testattr2 value2 $TESTFILE
+	log_must set_xattr testattr3 value3 $TESTFILE
+	log_must ls_xattr $TESTFILE
+
+	log_must set_tunable32 XATTR_COMPAT $ALTERNATIVE_XATTR_COMPAT
+	log_must ls_xattr $TESTFILE
+	log_must eval "get_xattr testattr1 $TESTFILE > $TMPFILE"
+	log_must test $(<$TMPFILE) = value1
+	log_must set_xattr testattr2 newvalue2 $TESTFILE
+	log_must rm_xattr testattr3 $TESTFILE
+	log_must set_xattr testattr4 value4 $TESTFILE
+	log_must ls_xattr $TESTFILE
+
+	log_must set_tunable32 XATTR_COMPAT $NATIVE_XATTR_COMPAT
+	log_must ls_xattr $TESTFILE
+	log_must eval "get_xattr testattr1 $TESTFILE > $TMPFILE"
+	log_must test $(<$TMPFILE) = value1
+	log_must eval "get_xattr testattr2 $TESTFILE > $TMPFILE"
+	log_must test $(<$TMPFILE) = newvalue2
+	log_mustnot get_xattr testattr3 $TESTFILE
+	log_must set_xattr testattr3 value3 $TESTFILE
+	log_must eval "get_xattr testattr4 $TESTFILE > $TMPFILE"
+	log_must test $(<$TMPFILE) = value4
+	log_must ls_xattr $TESTFILE
+
+	log_must rm $TESTFILE
+done
+
+log_pass "The zfs_xattr_compat tunable and fallback works as expected"