Improvements to the 'compatibility' property

Several improvements to the operation of the 'compatibility' property:

1) Improved handling of unrecognized features:
Change the way unrecognized features in compatibility files are handled.

 * invalid features in files under /usr/share/zfs/compatibility.d
   only get a warning (as these may refer to future features not yet in
   the library),
 * invalid features in files under /etc/zfs/compatibility.d
   get an error (as these are presumed to refer to the current system).

2) Improved error reporting from zpool_load_compat.
Note: slight ABI change to zpool_load_compat for better error reporting.

3) compatibility=legacy inhibits all 'zpool upgrade' operations.

4) Detect when features are enabled outside current compatibility set
   * zpool set compatibility=foo <-- print a warning
   * zpool set feature@xxx=enabled <-- error
   * zpool status <-- indicate this state

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Colm Buckley <colm@tuatha.org>
Closes #11861

man/man8/zpool-upgrade.8

@@ -55,11 +55,9 @@ formatted using a legacy ZFS version number.
 These pools can continue to be used, but some features may not be available.
 Use
 .Nm zpool Cm upgrade Fl a
-to enable all features on all pools. (If a pool has specified compatibility
-feature sets using the
+to enable all features on all pools (subject to the
 .Fl o Ar compatibility
-property, only the features present in all requested compatibility sets will
-be enabled on that pool.)
+property).
 .It Xo
 .Nm zpool
 .Cm upgrade
@@ -75,11 +73,15 @@ for a description of feature flags features supported by the current software.
 .Op Fl V Ar version
 .Fl a Ns | Ns Ar pool Ns ...
 .Xc
-Enables all supported features on the given pool. (If the pool has specified
-compatibility feature sets using the
+Enables all supported features on the given pool.
+.Pp
+If the pool has specified compatibility feature sets using the
 .Fl o Ar compatibility
 property, only the features present in all requested compatibility sets will be
-enabled.)
+enabled. If this property is set to 
+.Ar legacy
+then no upgrade will take place.
+.Pp
 Once this is done, the pool will no longer be accessible on systems that do not
 support feature flags.
 See