Allow to control failfast

Linux defaults to setting "failfast" on BIOs, so that the OS will not
retry IOs that fail, and instead report the error to ZFS.

In some cases, such as errors reported by the HBA driver, not
the device itself, we would wish to retry rather than generating
vdev errors in ZFS. This new property allows that.

This introduces a per vdev option to disable the failfast option.
This also introduces a global module parameter to define the failfast
mask value.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Allan Jude <allan@klarasystems.com>
Signed-off-by: Mariusz Zaborski <mariusz.zaborski@klarasystems.com>
Sponsored-by: Seagate Technology LLC
Submitted-by: Klara, Inc.
Closes #14056

man/man7/vdevprops.7

@@ -20,7 +20,7 @@
 .\"
 .\" Copyright (c) 2021 Klara, Inc.
 .\"
-.Dd November 27, 2021
+.Dd October 30, 2022
 .Dt VDEVPROPS 7
 .Os
 .
@@ -121,6 +121,9 @@ dataset.
 A text comment up to 8192 characters long
 .It Sy bootsize
 The amount of space to reserve for the EFI system partition
+.It Sy failfast
+If this device should propage BIO errors back to ZFS, used to disable
+failfast.
 .It Sy path
 The path to the device for this vdev
 .It Sy allocating