Add "compatibility" property for zpool feature sets

Property to allow sets of features to be specified; for compatibility
with specific versions / releases / external systems. Influences
the behavior of 'zpool upgrade' and 'zpool create'. Initial man
page changes and test cases included.

Brief synopsis:

zpool create -o compatibility=off|legacy|file[,file...] pool vdev...

compatibility = off : disable compatibility mode (enable all features)
compatibility = legacy : request that no features be enabled
compatibility = file[,file...] : read features from specified files.
Only features present in *all* files will be enabled on the
resulting pool. Filenames may be absolute, or relative to
/etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d (/etc
checked first).

Only affects zpool create, zpool upgrade and zpool status.

ABI changes in libzfs:

* New function "zpool_load_compat" to load and parse compat sets.
* Add "zpool_compat_status_t" typedef for compatibility parse status.
* Add ZPOOL_PROP_COMPATIBILITY to the pool properties enum
* Add ZPOOL_STATUS_COMPATIBILITY_ERR to the pool status enum

An initial set of base compatibility sets are included in
cmd/zpool/compatibility.d, and the Makefile for cmd/zpool is
modified to install these in $pkgdatadir/compatibility.d and to
create symbolic links to a reasonable set of aliases.

Reviewed-by: ericloewe
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Richard Laager <rlaager@wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Colm Buckley <colm@tuatha.org>
Closes #11468

man/man5/zpool-features.5

@@ -16,6 +16,7 @@
 .\" Portions Copyright [yyyy] [name of copyright owner]
 .\" Copyright (c) 2019, Klara Inc.
 .\" Copyright (c) 2019, Allan Jude
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .TH ZPOOL-FEATURES 5 "Aug 24, 2020" OpenZFS
 .SH NAME
 zpool\-features \- ZFS pool feature descriptions
@@ -26,7 +27,8 @@ ZFS pool on\-disk format versions are specified via "features" which replace
 the old on\-disk format numbers (the last supported on\-disk format number is
 28). To enable a feature on a pool use the \fBupgrade\fR subcommand of the
 zpool(8) command, or set the \fBfeature@\fR\fIfeature_name\fR property
-to \fBenabled\fR.
+to \fBenabled\fR. Please also see the \fB"Compatibility feature sets"\fR
+section for information on how sets of features may be enabled together.
 .sp
 .LP
 The pool format does not affect file system version compatibility or the ability
@@ -140,6 +142,61 @@ read\-only mode.
 Some features depend on other features being enabled in order to function
 properly. Enabling a feature will automatically enable any features it
 depends on.
+
+.SS "Compatibility feature sets"
+.sp
+.LP
+It is sometimes necessary for a pool to maintain compatibility with a
+specific on\-disk format, by enabling and disabling particular features. The
+\fBcompatibility\fR feature facilitates this by allowing feature sets to
+be read from text files. When set to \fBoff\fR (the default); compatibility
+feature sets are disabled (ie: all features are enabled); when set to
+\fBlegacy\fR; no features are enabled. When set to a comma\-separated list
+of filenames (each filename may either be an absolute path, or relative to
+\fB/etc/zfs/compatibility.d\fR or \fB/usr/share/zfs/compatibility.d\fR)
+the lists of requested features are read from those files, separated by
+whitespace and/or commas. Only features present in all files are enabled.
+.LP
+Simple sanity checks are applied to the files; they must be between 1 and
+16,384 bytes in size, and must end with a newline character.
+.LP
+The requested features are applied when a pool is created using
+\fBzpool create \-o compatibility=...\fR and controls which features are
+enabled when using \fBzpool upgrade\fR. \fBzpool status\fR
+will not show a warning about disabled features which are not part
+of the requested feature set.
+.LP
+By convention, compatibility files in \fB/usr/share/zfs/compatibility.d\fR
+are provided by the distribution package, and include feature sets
+supported by important versions of popular distribtions, and feature
+sets commonly supported at the start of each year.  Compatibility files
+in \fB/etc/zfs/compatibility.d\fR, if present, will take precedence over
+files with the same name in \fB/usr/share/zfs/compatibility.d\fR.
+.LP
+Compatibility files may include comments; any text from \fB#\fR to the end
+of the line is ignored.
+.LP
+\fBExample:\fR
+.EX
+# \fBcat /usr/share/zfs/compatibility.d/grub2\fR
+# Features which are supported by GRUB2
+async_destroy
+bookmarks
+embedded_data
+empty_bpobj
+enabled_txg
+extensible_dataset
+filesystem_limits
+hole_birth
+large_blocks
+lz4_compress
+spacemap_histogram
+
+# \fBzpool create \-o compatibility=grub2 bootpool vdev\fR
+.EE
+.LP
+See \fBzpool\-create(8)\fR and \fBzpool\-upgrade(8)\fR for more information
+on how these commands are affected by feature sets.
 .SH FEATURES
 .sp
 .LP

man/man8/zpool-create.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOL-CREATE 8
@@ -40,6 +41,7 @@
 .Op Fl m Ar mountpoint
 .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ...
 .Oo Fl o Ar feature@feature Ns = Ns Ar value Oc
+.Op Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
 .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ...
 .Op Fl R Ar root
 .Ar pool vdev Ns ...
@@ -52,6 +54,7 @@
 .Op Fl m Ar mountpoint
 .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ...
 .Oo Fl o Ar feature@feature Ns = Ns Ar value Oc Ns ...
+.Op Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
 .Oo Fl O Ar file-system-property Ns = Ns Ar value Oc Ns ...
 .Op Fl R Ar root
 .Op Fl t Ar tname
@@ -135,9 +138,14 @@ This can be overridden with the
 .Fl m
 option.
 .Pp
-By default all supported features are enabled on the new pool unless the
+By default all supported features are enabled on the new pool. The
 .Fl d
-option is specified.
+option or the
+.Fl o Ar compatibility
+property (eg:
+.Fl o Ar compatibility=2020
+) can be used to restrict the features that are enabled, so that the
+pool can be imported on other releases of the ZFS software.
 .Bl -tag -width Ds
 .It Fl d
 Do not enable any features on the new pool.
@@ -179,6 +187,10 @@ Sets the given pool properties.
 See the
 .Xr zpoolprops
 manual page for a list of valid properties that can be set.
+.It Fl o Ar compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
+Specifies compatibility feature sets. See
+.Xr zpool-features 5
+for more information about compatibility feature sets.
 .It Fl o Ar feature@feature Ns = Ns Ar value
 Sets the given pool feature. See the
 .Xr zpool-features 5

man/man8/zpool-upgrade.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOL-UPGRADE 8
@@ -54,7 +55,11 @@ formatted using a legacy ZFS version number.
 These pools can continue to be used, but some features may not be available.
 Use
 .Nm zpool Cm upgrade Fl a
-to enable all features on all pools.
+to enable all features on all pools. (If a pool has specified compatibility
+feature sets using the
+.Fl o Ar compatibility
+property, only the features present in all requested compatibility sets will
+be enabled on that pool.)
 .It Xo
 .Nm zpool
 .Cm upgrade
@@ -70,7 +75,11 @@ for a description of feature flags features supported by the current software.
 .Op Fl V Ar version
 .Fl a Ns | Ns Ar pool Ns ...
 .Xc
-Enables all supported features on the given pool.
+Enables all supported features on the given pool. (If the pool has specified
+compatibility feature sets using the
+.Fl o Ar compatibility
+property, only the features present in all requested compatibility sets will be
+enabled.)
 Once this is done, the pool will no longer be accessible on systems that do not
 support feature flags.
 See
@@ -79,7 +88,8 @@ for details on compatibility with systems that support feature flags, but do not
 support all features enabled on the pool.
 .Bl -tag -width Ds
 .It Fl a
-Enables all supported features on all pools.
+Enables all supported features (from specified compatibility sets, if any) on all
+pools.
 .It Fl V Ar version
 Upgrade to the specified legacy version.
 If the

man/man8/zpoolprops.8

@@ -26,6 +26,7 @@
 .\" Copyright (c) 2018 George Melikov. All Rights Reserved.
 .\" Copyright 2017 Nexenta Systems, Inc.
 .\" Copyright (c) 2017 Open-E, Inc. All Rights Reserved.
+.\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
 .\"
 .Dd August 9, 2019
 .Dt ZPOOLPROPS 8
@@ -285,6 +286,24 @@ A text string consisting of printable ASCII characters that will be stored
 such that it is available even if the pool becomes faulted.
 An administrator can provide additional information about a pool using this
 property.
+.It Sy compatibility Ns = Ns Ar off | legacy | file Bq , Ns Ar file Ns ...
+Specifies that the pool maintain compatibility with specific feature sets.
+When set to
+.Sy off
+(or unset); compatibility is disabled (all features are enabled); when set to
+.Sy legacy Ns ;
+no features are enabled. When set to a comma-separated list of
+filenames (each filename may either be an absolute path, or relative to
+.Pa /etc/zfs/compatibility.d or Pa /usr/share/zfs/compatibility.d Ns )
+the lists of requested features are read from those files, separated by
+whitespace and/or commas. Only features present in all files are enabled.
+
+See
+.Xr zpool-features 5 Ns ,
+.Xr zpool-create 8
+and
+.Xr zpool-upgrade 8
+for more information on the operation of compatibility feature sets.
 .It Sy dedupditto Ns = Ns Ar number
 This property is deprecated and no longer has any effect.
 .It Sy delegation Ns = Ns Sy on Ns | Ns Sy off