Add "compatibility" property for zpool feature sets

Property to allow sets of features to be specified; for compatibility
with specific versions / releases / external systems. Influences
the behavior of 'zpool upgrade' and 'zpool create'. Initial man
page changes and test cases included.

Brief synopsis:

zpool create -o compatibility=off|legacy|file[,file...] pool vdev...

compatibility = off : disable compatibility mode (enable all features)
compatibility = legacy : request that no features be enabled
compatibility = file[,file...] : read features from specified files.
Only features present in *all* files will be enabled on the
resulting pool. Filenames may be absolute, or relative to
/etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d (/etc
checked first).

Only affects zpool create, zpool upgrade and zpool status.

ABI changes in libzfs:

* New function "zpool_load_compat" to load and parse compat sets.
* Add "zpool_compat_status_t" typedef for compatibility parse status.
* Add ZPOOL_PROP_COMPATIBILITY to the pool properties enum
* Add ZPOOL_STATUS_COMPATIBILITY_ERR to the pool status enum

An initial set of base compatibility sets are included in
cmd/zpool/compatibility.d, and the Makefile for cmd/zpool is
modified to install these in $pkgdatadir/compatibility.d and to
create symbolic links to a reasonable set of aliases.

Reviewed-by: ericloewe
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Colm Buckley <colm@tuatha.org>
Closes #11468

man/man8/zpool-create.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOL-CREATE 8
@@ -40,6 +41,7 @@
 .Op Fl m Ar mountpoint
 .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ...
 .Oo Fl o Ar feature@feature Ns = Ns Ar value Oc
+.Op Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
 .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ...
 .Op Fl R Ar root
 .Ar pool vdev Ns ...
@@ -52,6 +54,7 @@
 .Op Fl m Ar mountpoint
 .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ...
 .Oo Fl o Ar feature@feature Ns = Ns Ar value Oc Ns ...
+.Op Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
 .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ...
 .Op Fl R Ar root
 .Op Fl t Ar tname
@@ -135,9 +138,14 @@ This can be overridden with the
 .Fl m
 option.
 .Pp
-By default all supported features are enabled on the new pool unless the
+By default all supported features are enabled on the new pool. The
 .Fl d
-option is specified.
+option or the
+.Fl o Ar compatibility
+property (eg:
+.Fl o Ar compatibility=2020
+) can be used to restrict the features that are enabled, so that the
+pool can be imported on other releases of the ZFS software.
 .Bl -tag -width Ds
 .It Fl d
 Do not enable any features on the new pool.
@@ -179,6 +187,10 @@ Sets the given pool properties.
 See the
 .Xr zpoolprops
 manual page for a list of valid properties that can be set.
+.It Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
+Specifies compatibility feature sets. See
+.Xr zpool-features 5
+for more information about compatibility feature sets.
 .It Fl o Ar feature@feature Ns = Ns Ar value
 Sets the given pool feature. See the
 .Xr zpool-features 5

man/man8/zpool-upgrade.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOL-UPGRADE 8
@@ -54,7 +55,11 @@ formatted using a legacy ZFS version number.
 These pools can continue to be used, but some features may not be available.
 Use
 .Nm zpool Cm upgrade Fl a
-to enable all features on all pools.
+to enable all features on all pools. (If a pool has specified compatibility
+feature sets using the
+.Fl o Ar compatibility
+property, only the features present in all requested compatibility sets will
+be enabled on that pool.)
 .It Xo
 .Nm zpool
 .Cm upgrade
@@ -70,7 +75,11 @@ for a description of feature flags features supported by the current software.
 .Op Fl V Ar version
 .Fl a Ns | Ns Ar pool Ns ...
 .Xc
-Enables all supported features on the given pool.
+Enables all supported features on the given pool. (If the pool has specified
+compatibility feature sets using the
+.Fl o Ar compatibility
+property, only the features present in all requested compatibility sets will be
+enabled.)
 Once this is done, the pool will no longer be accessible on systems that do not
 support feature flags.
 See
@@ -79,7 +88,8 @@ for details on compatibility with systems that support feature flags, but do not
 support all features enabled on the pool.
 .Bl -tag -width Ds
 .It Fl a
-Enables all supported features on all pools.
+Enables all supported features (from specified compatibility sets, if any) on all
+pools.
 .It Fl V Ar version
 Upgrade to the specified legacy version.
 If the

man/man8/zpoolprops.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOLPROPS 8
@@ -285,6 +286,24 @@ A text string consisting of printable ASCII characters that will be stored
 such that it is available even if the pool becomes faulted.
 An administrator can provide additional information about a pool using this
 property.
+.It Sy compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
+Specifies that the pool maintain compatibility with specific feature sets.
+When set to
+.Sy off
+(or unset); compatibility is disabled (all features are enabled); when set to
+.Sy legacy Ns ;
+no features are enabled. When set to a comma-separated list of
+filenames (each filename may either be an absolute path, or relative to
+.Pa /etc/zfs/compatibility.d or Pa /usr/share/zfs/compatibility.d Ns )
+the lists of requested features are read from those files, separated by
+whitespace and/or commas. Only features present in all files are enabled.
+
+See
+.Xr zpool-features 5 Ns ,
+.Xr zpool-create 8
+and
+.Xr zpool-upgrade 8
+for more information on the operation of compatibility feature sets.
 .It Sy dedupditto Ns = Ns Ar number
 This property is deprecated and no longer has any effect.
 .It Sy delegation Ns = Ns Sy on Ns | Ns Sy off