Improvements to the 'compatibility' property

Several improvements to the operation of the 'compatibility' property:

1) Improved handling of unrecognized features:
Change the way unrecognized features in compatibility files are handled.

 * invalid features in files under /usr/share/zfs/compatibility.d
   only get a warning (as these may refer to future features not yet in
   the library),
 * invalid features in files under /etc/zfs/compatibility.d
   get an error (as these are presumed to refer to the current system).

2) Improved error reporting from zpool_load_compat.
Note: slight ABI change to zpool_load_compat for better error reporting.

3) compatibility=legacy inhibits all 'zpool upgrade' operations.

4) Detect when features are enabled outside current compatibility set
   * zpool set compatibility=foo <-- print a warning
   * zpool set feature@xxx=enabled <-- error
   * zpool status <-- indicate this state

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Colm Buckley <colm@tuatha.org>
Closes #11861

man/man5/zpool-features.5

@@ -166,6 +166,12 @@ enabled when using \fBzpool upgrade\fR. \fBzpool status\fR
 will not show a warning about disabled features which are not part
 of the requested feature set.
 .LP
+The special value \fBlegacy\fR prevents any features from being enabled,
+either via \fBzpool upgrade\fR or via \fBzpool set feature@XX=enabled\fR.
+This setting also prevents pools from being upgraded to newer on-disk
+versions. This is a safety measure to prevent new features from being
+accidentally enabled, breaking compatibility.
+.LP
 By convention, compatibility files in \fB/usr/share/zfs/compatibility.d\fR
 are provided by the distribution package, and include feature sets
 supported by important versions of popular distributions, and feature
@@ -173,6 +179,15 @@ sets commonly supported at the start of each year.  Compatibility files
 in \fB/etc/zfs/compatibility.d\fR, if present, will take precedence over
 files with the same name in \fB/usr/share/zfs/compatibility.d\fR.
 .LP
+If an unrecognized feature is found in these files, an error message will
+be shown. If the unrecognized feature is in a file in
+\fB/etc/zfs/compatibility.d\fR, this is treated as an error and processing
+will stop. If the unrecognized feature is under
+\fB/usr/share/zfs/compatibility.d\fR, this is treated as a warning and
+processing will continue. This difference is to allow distributions to
+include features which might not be recognized by the currently-installed
+binaries.
+.LP
 Compatibility files may include comments; any text from \fB#\fR to the end
 of the line is ignored.
 .LP

man/man8/zpool-upgrade.8

@@ -55,11 +55,9 @@ formatted using a legacy ZFS version number.
 These pools can continue to be used, but some features may not be available.
 Use
 .Nm zpool Cm upgrade Fl a
-to enable all features on all pools. (If a pool has specified compatibility
-feature sets using the
+to enable all features on all pools (subject to the
 .Fl o Ar compatibility
-property, only the features present in all requested compatibility sets will
-be enabled on that pool.)
+property).
 .It Xo
 .Nm zpool
 .Cm upgrade
@@ -75,11 +73,15 @@ for a description of feature flags features supported by the current software.
 .Op Fl V Ar version
 .Fl a Ns | Ns Ar pool Ns ...
 .Xc
-Enables all supported features on the given pool. (If the pool has specified
-compatibility feature sets using the
+Enables all supported features on the given pool.
+.Pp
+If the pool has specified compatibility feature sets using the
 .Fl o Ar compatibility
 property, only the features present in all requested compatibility sets will be
-enabled.)
+enabled. If this property is set to 
+.Ar legacy
+then no upgrade will take place.
+.Pp
 Once this is done, the pool will no longer be accessible on systems that do not
 support feature flags.
 See